How Xen works
Xen not only didn't make it to the main tree of the Linux kernel. It doesn't even run on Linux, although it looks like it. It's a bare metal hypervisor (or: type 1 hypervisor)- a piece of software that runs directly on hardware. If you install a Xen package on your normal Linux distribution, after rebooting you will see Xen messages first. It will then boot your existing system into a first, specially privileged virtual machine called dom0.
This makes the process quite complex. If you start experimenting with Xen and at first attempt make your machine unbootable, don't worry - it happened to many people, including Yours Truly. You can also download Xen Server - commercial, but free distribution of Xen which comes with a simple to use installer, a specially tailored, minimal Linux system in dom0 and enterprise-class management tools. I'll write some more about diffences between XenServer and "community" Xen in a few days.
It also means you won't be able to manipulate VMs using ordinary Linux tools, e.g. stop them with kill and monitor with top. However, Xen comes with some great management software and even greater 3rd-party apps are available (be careful, some of them don't work with Xen Server). They can fully utilize interesting features of Xen, like storing snapshots of VMs and live-migration between physical servers.
Xen is also special for its use of technology called paravirtualization. In short, it means that the guest operating systems knows it runs on a virtualized system. There is an obvious downside: it needs to be specially modified, although with open source OSes that's not much of an issue. But there's also one very important advantage: speed. Xen delivers almost native performance. Other virtualization platforms use this approach in a very limited way, usually in form of a driver package that you install on a guest systems. This improves the speed compaired to completely non-paravirtualized system, but is still far from what can be achieved with Xen.
How KVM works
KVM runs inside a Linux system, not above it - it's called type 2, or hosted hypervisor. This has several significant implications. From technical point of view, it makes it easier to deploy and manage, no need for special boot-time support; but it also makes it harder to deliver good performance. From political point of view, Linux developers view it as superior to Xen because it's a part of the system, not an outside piece of software.
KVM requires CPU with hardware virtualization support. Most new server, desktop and laptop processors from Intel and AMD work with KVM. Older CPUs or low-power units for netbooks, PDAs and the like lack this feature. Hardware-assisted virtualization makes it possible to run an unmodified operating system with an adequate speed. Xen can do it too, although this feature is mostly used to run Windows or other proprietary guests. Even with hardware support, pure virtualization is still much slower than paravirtualization.
Rest of the world
Some VMware server platforms and Microsoft Hyper-V are bare-metal hypervisors, like Xen. VMware's desktop solutions (Player, Workstation) are hosted, as well as QEMU, VirtualBox, Microsoft Virtual PC and pretty much everything else. None of them employ a full paravirtualization, although they sometimes offer drivers improving the performance of guest systems.
KVM only runs on machines with hardware virtualization support. Some enterprise platforms have this requirement too. VirtualBox and desktop versions of VMware work on CPUs lacking virtualization support, but the performance is greatly reduced.
What shoud you choose?
For the server, grid or cloud
If you want to run Linux, BSD or Solaris guests, nothing beats the paravirtualized performance of Xen. For Windows and other proprietary operating systems, there's not much difference
between the platforms. Performance and features are similar.In the beginning KVM lacked live migration and good tools. Nowadays most open source VM management applications (like virt-manager on the screenshot) support both Xen and KVM. Live migration was added in 2007. The whole system is considered stable, although some people still have reservations and think it's not mature enough. Out of the box support in leading Linux distributions is definitely a good point.
VMware is the most widespread solutions - as they proudly admit, it's used by all companies from Fortune 100. Main disadvantage is poor support from open source community. If free management software from VMware is not enough for you, you usually have no choice but to buy a commercial solution - and they don't come cheap. Expect to pay several thousand $ per server or even per CPU.
My subjective choice would be: 1 - Xen, 2 - KVM, 3 - VMware ESXi.
For the personal computer
While Xen is my first choice for the server, it would be very far on the list of "best desktop virtualization platforms". One reason is poor support for power management. It slowly improves, but still I wouldn't install Xen on my laptop. Also the installation method is more suitable for server platforms, but inconvenient for the desktop.
KVM falls somewhere in the middle. As a hosted hypervisor, it's easier to run. Your Linux distribution probably already supports it. Yet, it lacks some user-friendliness of true desktop solutions and if your CPU doesn't have virtualization extensions, you're out of luck.
VMware Player (free of charge, but not open source) is extremaly easy to use, when you want to run VMs prepared by somebody else (hence the name Player - nothing to do with games). Creating a new machine requires editing configuration file or external software (eg. this web-based VM creator). What I really like is convenient hardwa
re management (see screenshot) - just one click to decide if your USB drive belongs to host or guest operating system, another to mount ISO image as guest's DVD-ROM. Another feature is easy file sharing between guest and host. Player's bigger brother is VMware Workstation (about $180). It comes with the ability to create new VMs as well as some other additions. Due to the number of features it slightly harder to use, but still very user-friendly.VMware offers special drivers for guest operating systems. They are bundled with Workstation, for Player they have to be downloaded separately (or you can borrow them from Workstation, even demo download - license allows it). They are especially useful if you want to run Windows guest, even on older CPUs without hardware assist it's quite responsive.
VirtualBox comes close to VMware. It also has the desktop look&feel and runs on non-hardware-assisted platforms. Bundled guest additions improve performance of virtualized systems. Sharing files and hardware is easy - but not that easy. Overall, in both speed and features, it comes second.
My subjective choice: 1 - VMware Player or Workstation, 2 - VirtualBox, 3 - KVM
EDIT: I later found out that new version of VirtualBox is superior to VMware Player.
What other people think?
Read some more comparisons:
- Xen vs. KVM at gridvm.org (very interesting site, by the way)
- Xen vs. KVM
- Xen vs. KVM from LinuxWorld 2007
Posts
Thank you. That was very informative.
ReplyDeleteComing from someone who works with XEN and KVM on a daily basis, I can tell your article is probably the most intelligent and unbiased one on the internet.
ReplyDeleteGood job!
Good article. One extra category could be "the Home server". Power management does matter, bus so does speed. Ease of use is important, yet not as important as on the desktop as it's probably a set up once and hardly look at it again thing. What would your choice be in this case? Xen or KVM?
ReplyDeletemartdj: if your CPU doesn't have virtualization extensions, KVM won't work. VirtualBox and VMware would, but with reduced speed. Xen is the only solution that would give you a good performance.
ReplyDeleteSo I assume a CPU with AMD-V or Intel VT-x. If I had no previous experience, I'd probably use KVM. It's way easier. Since I wrote that post it got even better support in most distributions, while Xen dom0 support is now quite poor. Xen means spending few days learning, getting (often compiling) the software, configuring the system. If you want to put Xen experience in your resume, go ahead. If not, probably not worth the effort.
Thanks for your response. From what I understood the past days there are two versions of hardware support. The first version is the basic AMD-V / Intel VT support in the cpu. The 2nd version, the full IOMMU support, has to come from the chipset as can be found in the Intel chipsets for the Core iX platform. My cpu has AMD-V support, so KVM does work, but PCI passthrough with KVM doesn't as it needs the 2nd version of support.
ReplyDeleteThat said your response made me help decide that's probably best for me to use KVM and run the stuff for which I wanted PCI passthrough support on the host, instead of spending days on learning and configuring Xen.
Thanks!
If you go to support forum for any hypervisor, you'll notice that half of the questions is about PCI passthrough. I try to avoid it too.
ReplyDeleteSadly, running Xen on recent distributions is getting harder. It's now a serious investment of your time. Unless you use a distribution that has good support for dom0 - Debian Etch works like a charm, I think that RedHat and SuSE have it too. Mind you, it'd still be harder than KVM.
You can also have a very easy setup with XenServer, commercial but free of charge version of Xen. With it, you create VMs with a few clicks. But it requires a clean install and a Windows machine as an admin's workstation.
Great article!
ReplyDeleteI'm playing with Xen last few days, and its really time consumint for setup, especialy on newer kernel, because it does not have support, and you must download patched kernel from git. But after your review i give up on xen and start with KVM setup.
BTW i have problem with install XEN on Arch linux on my laptop. With Centos 5.4 installing xen was simple as that. Because it has all packages and don't require any compilation of xen and kernel.
Once more great article!
Well, if you wanted an unbias comparison between KVM and XEN, don't bother coming back here.
ReplyDelete