Virtualization cost savings
I have several new articles in the hopper with various degrees of progress but work has interfered with their completion in the past two weeks. One project that came up was the replacement of hardware. One of our older servers died and its brethen bought at the same time (more than 5 years ago) are sure to go soon. But instead of replacing these 4 servers with 4 new ones, we decided to take virtualization to it's next step and replace them with just 2 machines. (Our IT manager was not yet comfortable with a single machine replacing all 4 of them.)
After using Linux KVM for about 6 months, I am fairy comfortable with it and quickly replace all the old servers. Instead installing and customizing the same software packages 4 times on 4 different machines, I only had to make a single VM running all the necessary software. Then I cloned the image 3 times and modified the UUID plus MAC address to be slightly different. Voila, 4 virtual machines up in a flash. Unexpected high volume? I can keep a few more cloned images on other machines configured for KVM and launch them whenever needed.
It's pretty clear virtualization decreases IT administration costs. But what about performance? After all, having 4 virtual servers that run dog slow does not help. So while browsing the Linux KVM news, I came across news reporting Redhat recently released their Windows BlockIO drivers (Windows 2003/2008, Windows XP) under GPL. Ordinarily, a VM must emulate the network and disk controller devices -- however, using VirtIO and BlockIO, a custom driver is installed under the guest OS. I/O is then handed to the Linux KVM hypervisor to execute at the host OS level -- which greatly improves performance -- instead of being emulated. I was excited to test it out of course and installed both the network and disk driver on a single production VM. (We have many production servers so no big deal if 1 crashes due to untested software.) I was shocked after a day of production transactions showing the the Windows VM running 30%+ faster than native WIndows execution on the same hardware!
I don't know the exact reasons why a VM is so much faster. It could be because Linux's I/O and process scheduler is much better than Windows' so with Linux backstopping everything, the normally inefficient Windows transactions get rescheduled by Linux. Whatever the reason though, this changes the equation for the way we should manage our data centers. Our data centers have been built-up over time. We started with a few machines and kept adding and adding as we signed up more customers. With all this accumulation of hardware, we now have a full cage (2 racks) in a San Francisco data center and a full cabinet in Sterling, VA. We pay $2700/mo for the cage+extra power and $2300/mo for the cabinet+extra power. In total, it costs $5000/mo -- $60K/year -- to host our production servers.
Now let's say we threw out all our existing hardware and replace them with 2 monstrous servers. Something like a 4 x Opteron 6136 system (32 cores total) w/ 64GB of memory and 20 hard drives could replace all our existing hardware in both data centers. The cost of this server would probably be about $12K+ fully decked out so we would be looking at an initial outlay of $25K. In return though, we could downgrade our hosting storage to 1/3rd cabinets at $1200/mo. 2 data centers at $2400/mo comes out to $29K/year -- cost savings of $31K! This means a single year of operation would pay for this change not including the savings on IT administration.
The only real hesitation is now we have only a single machine that can fail at each location. It feels riskier so we might use instead 2 "lesser" machines per data center or adding a 3rd data center within driving distance (say Sacramento). In the past, the enterprise software from VMware was very expensive but with KVM supported under Centos Linux 5.4 and the VirtIO/BlockIO drivers available for Windows, it's getting to the no brainer territory to embrace virtualization.
After using Linux KVM for about 6 months, I am fairy comfortable with it and quickly replace all the old servers. Instead installing and customizing the same software packages 4 times on 4 different machines, I only had to make a single VM running all the necessary software. Then I cloned the image 3 times and modified the UUID plus MAC address to be slightly different. Voila, 4 virtual machines up in a flash. Unexpected high volume? I can keep a few more cloned images on other machines configured for KVM and launch them whenever needed.
It's pretty clear virtualization decreases IT administration costs. But what about performance? After all, having 4 virtual servers that run dog slow does not help. So while browsing the Linux KVM news, I came across news reporting Redhat recently released their Windows BlockIO drivers (Windows 2003/2008, Windows XP) under GPL. Ordinarily, a VM must emulate the network and disk controller devices -- however, using VirtIO and BlockIO, a custom driver is installed under the guest OS. I/O is then handed to the Linux KVM hypervisor to execute at the host OS level -- which greatly improves performance -- instead of being emulated. I was excited to test it out of course and installed both the network and disk driver on a single production VM. (We have many production servers so no big deal if 1 crashes due to untested software.) I was shocked after a day of production transactions showing the the Windows VM running 30%+ faster than native WIndows execution on the same hardware!
I don't know the exact reasons why a VM is so much faster. It could be because Linux's I/O and process scheduler is much better than Windows' so with Linux backstopping everything, the normally inefficient Windows transactions get rescheduled by Linux. Whatever the reason though, this changes the equation for the way we should manage our data centers. Our data centers have been built-up over time. We started with a few machines and kept adding and adding as we signed up more customers. With all this accumulation of hardware, we now have a full cage (2 racks) in a San Francisco data center and a full cabinet in Sterling, VA. We pay $2700/mo for the cage+extra power and $2300/mo for the cabinet+extra power. In total, it costs $5000/mo -- $60K/year -- to host our production servers.
Now let's say we threw out all our existing hardware and replace them with 2 monstrous servers. Something like a 4 x Opteron 6136 system (32 cores total) w/ 64GB of memory and 20 hard drives could replace all our existing hardware in both data centers. The cost of this server would probably be about $12K+ fully decked out so we would be looking at an initial outlay of $25K. In return though, we could downgrade our hosting storage to 1/3rd cabinets at $1200/mo. 2 data centers at $2400/mo comes out to $29K/year -- cost savings of $31K! This means a single year of operation would pay for this change not including the savings on IT administration.
The only real hesitation is now we have only a single machine that can fail at each location. It feels riskier so we might use instead 2 "lesser" machines per data center or adding a 3rd data center within driving distance (say Sacramento). In the past, the enterprise software from VMware was very expensive but with KVM supported under Centos Linux 5.4 and the VirtIO/BlockIO drivers available for Windows, it's getting to the no brainer territory to embrace virtualization.
(Filed in technology)
More useful KVM links:
Kernel Same-Page Merging -- combines similar memory pages from multiple VMs together. Gave this a test on a machine running 2 Windows VMs and free memory went from 900MB to 3.2GB -- a whopping memory savings. This would greatly increase VM density. If you are running Centos 5.4, it's a kernel module instead of built-in to the kernel -- hence, none of the statistics commands in the below link will work. But you can tell it's working from looking at top.
* http://www.linux-kvm.com/content/using-ksm-kernel-samepage-merging-kvm
Huge TLB FS can boost CPU performance by 10% but will not work with KSM. So if you have a lot of memory, go with Huge TLB. Otherwise, KSM to free up more memory for disk caching.
* http://www.linux-kvm.com/content/get-performance-boost-backing-your-kvm-guest-hugetlbfs
Para-virtualization guest network driver (VirtIO):
* http://www.linux-kvm.com/content/tip-how-setup-windows-guest-paravirtual-network-drivers
VMware VGA driver:
* http://www.linux-kvm.com/content/using-vmware-vga-kvm-windows-guests