Downsides of Virtualisation?

This post possibly falls into the "stating the bleeding obvious " category! However,...

Reducing server count isn't just about reducing the amount of hardware. But, if you have virtualised a bunch of servers will you be so keen to look into combining their functions?
There still can be value in combining the functions of two virtual servers. There are the management benefits of fewer servers to manage and if you have software that is licensed on a per server basis, then there will be financial benefits too.

Eventually, you will develop an architectural dependence on servers that are virtualised. If your mail relay is virtualised and your payroll system isn't, then should your payroll system need to send email, a lot of the infrastructure needed for your virtualisation environment will need to be up and running too.
In most cases the virtual environment will be up and running. However, there are periods when you may be recovering from a disaster when that isn't the case.

Are these reasons not to virtualise?

Of course, not!
But they are factors to consider and factor in to your decision making.