The experimental outline pursued herein is aligned with the needs of our lab and the various customers we serve. It is often the case that the lab serves as an “incubator” for departmental projects, and those that prove themselves are promoted to clinical applications that move to the official hospital data center. Because the data center has standardized on VMWare ESXi, we have found it most efficacious to perform our base development in that arena. However, one can also see that VMWare is not often the performance winner. Fortunately, free tools from VMWare (i.e., Convertor Standalone Client) make it trivial to convert VMWare machines to Open Virtual Format which can be read by Virtual Box and Xen.
It is also a frequent requirement of our work to share our results with outside labs which are vey cost sensitive. For this reason, we chose to perform this analysis with products that may not be Free Open Source Software (FOSS), but are at least available without cost. Since we share the resulting VMs with third parties, it is also axiomatic that we must create them on platforms that are based on FOSS licenses; hence, the benchmark VM used here was based on Linux. Others could obviously replicate the current work on using a Windows VM benchmark platform; indeed, it would be interesting to see if the noted trends are reproduced.
One would expect, and indeed we certainly did, that the thin hypervisor group would be closest to bare metal results. However, the results are more complex than that, and as one can see from the preceding data, one can see that selecting the “best” VM environment depends on the target application’s behavior; is it compute limited, R/W limited, or a combination of both? It was also somewhat puzzling that sometimes the Write performance (be it on RAM, local disk, or network disk) was sometimes faster on a VM then on bare metal (note the performance of Virtual Box in this regard). In retrospect, however, this should not have been so surprising. In an OS on bare metal, the Write performance is totally gated by the input/output (I/O) performance of the real OS, whereas in a VM the VM memory manager may employ newer and more efficient buffering algorithms then the real OS can when writing to a slower physical I/O system. However, this cannot be done in the case of reads; the entire path to the physical layer has to be traversed and one notes in no case does VM read performance beat that of bare metal.
Another surprising result is the Integer and Floating point performance of the Virtual Box VM verses bare metal. One may expect that a virtual environment could largely expose the CPU directly to the VM client (without the overhead of virtual device drivers inherent in disk and other I/O operations), and thus that client could approach bare metal speeds. But it is difficult to comprehend how the VM could actually best the bare metal Dhrystone 2 results—clearly there is some very clever engineering in play in the Virtual Box.
One final observation is the relatively poor across the board performance of the VMWare ESXi server compared to the other thin platforms (Xen and KVM). This may be due to ignorance of tuning on our part; but as all platforms were used “out of the box,” we believe this experience would be observed by others. Another possible explanation is the difference in VM architecture. Both Xen and KVM rely on and use dedicated features in both the physical CPU and the guest OS being virtualized. This is called “para-virtualization” meaning that the VM environment performs some, but not all of the work, some of it is relegated to the physical CPU [8
]. Obviously hardware runs faster than software, but the downside is that only newer hardware and modified OS’ can be used. On the other hand, the full virtualization approach used by ESXi can run older hardware and support an unmodified OS (i.e., Windows NT and 2000), but apparently at a performance cost.
Based on the preceding one can deduce the following recommendations:
- For applications that are highly integer compute sensitive, the best choice is Virtual Box on Windows7, 32 bit (unless longer 64 bit math is required in which case Fedora 64 bit is the winner).
- For floating point sensitive applications, Virtual Box on either Solaris or Redhat 64 bit OS offer similar performance at about 80% of bare metal speed. The 20% penalty may be considered worthwhile, however, given the maintenance advantages that virtual machines have. Given the type of operations most often encountered in medical imaging processing (image registration, segmentation, etc.) this is the most common scenario .
- For high speed network file or web serving needs, no VM result is better than about 25% of bare metal performance. Hence, VM methods cannot be recommended as a competitive replacement for physical network file servers at this time.