Virtually A Machine

Xen vs. KVM yet again

2011-11-30T15:17:00.000+01:00

Interesting read on Baremetal vs. Xen vs. KVM benchmark

In short: Xen with HVM guests + PV drivers (that is, full virtualization mode with paravirtualised drivers, as opposed to paravirtualized mode) and KVM are equally effective. In some tests one was marginally faster, in some tests it was the other.

If you read my old posts, I was in favor of Xen for better performance of its PV (paravirtualized) guests and ability to run on older CPUs. Today all my boxes have hardware support for virtualization and I don't even use PV mode. While it's still slightly faster, it's not worth the additional complexity. If you've ever had a VM that failed to boot when PyGrub couldn't find the right kernel, you know what I mean. HVM + drivers/tools, that's the way to do it on any platform - be it Xen server or Virtualbox on a desktop.

Even shorter: virtualization platforms are so mature there's no real difference in performance or stability. For large installations, advanced features and management tools are the important factors. For one machine - familiarity and ease of use.

How to run a network monitoring station on a XenServer VM

2011-11-25T11:21:00.000+01:00

I spent way too much time on this seemingly simple task, I hope someone will find this guide useful.

Network monitoring applications require a promiscous mode on a network interface. Easy on a physical machine, didn't work on XenServer. Interface inside a VM switched to promiscous mode, but the traffic was filtered before it reached VM. Only for a brief period I could see all the packets, then only broadcasts got through. Just like on the switch that's learning - that should have pointed me to the right answer.

Open VSwitch

I googled for a solution and found some (e.g. this one from Citrix), but nothing worked. Until I remembered that XenServer 6 uses Open VSwitch by default and all the answers I found were for earlier versions using bridging backend. Now it should be easy. VSwitch works like a high-end, configurable switch so I'll just put the right port into traffic mirroring mode, right? Wrong.

To configure Open VSwitch in XenServer you use a VSwitch Controller virtual appliance. Installation is simple - just download the file, open it with XenCenter and select a VM. Then configure an IP address if you don't use DHCP, point the web browser at it and enter a pool to use. At this point it complained it can't work with free XenServer license. Which Citrix failed to mention on their website.

The other solution

It's always good to have a backup plan. Mine was simple: set XenServer to use the old bridging code. I rejected it at first for two reasons:
- it requires a reboot,
- VSwitch has more features (which I didn't need at the moment, but I might one day) and should perform better.

After scheduling a downtime for VMs, I typed one simple command into a CLI:

xe-switch-network-backend bridge

Reboot and it's done. You can check with 'cat /etc/xensource/network.conf' to be sure.

Forcing promiscous mode - the old way

Now all you have to do is put the right interfaces into promiscous mode. Interfaces - because you need to do it on both PIF and VIF (physical and virtual interface, one that connects to the real network and one that connects to VM). First, find the UUIDs of the interfaces with:

xe vif-list
xe pif-list

If you've got many VMs and NICs, you'll get a long list. You can try filtering, e.g. xe vif-list vm-name-label="My monitoring VM", see xe help vif-list for details. However, some filters didn't work for me. Once you have UUIDs, do:

xe pif-param-set uuid= other-config:promiscuous="true"
xe vif-param-set uuid= other-config:promiscuous="true"

Check if it worked:

xe pif-param-list uuid=
xe vif-param-list uuid=

Disconnect VIF from VM and connect it back:

xe vif-unplug uuid=
xe vif-plug uuid=

Note: unplug/plug only works if you have XenServer Tools installed. Otherwise reboot the VM.

Other possible ways

Some howtos for older XenServers used: brctl setageing xenbr0 0 (or whatever Xen bridging interface you use). I didn't need it, but you can try it if xe ...param-set fails.

I could probably configure VSwitch without Citrix controller appliance. I might one day.

Important new releases

2011-07-08T15:11:00.001+02:00

~~I don't usually write about the releases~~ : Who am I kidding? I only write about releases, it's been months since I wrote a useful Howto. I promise one as soon as I find some mythical free time.

Oracle has released an early version of Java 7. It promises some important features, most importantly native support for other languages on JVM. Speaking of which, O'Reilly has a short summary. Also from Oracle, new version of VirtualBox is mostly a bugfix release.

Samsung, as a part of Khronos group, is working to make GPUs useful (yep, I'm not a gamer, for me they're useless power drains most of the time). You've probably heard of OpenCL, a platform for running all-purpose software on GPUs or special-purpose accelerators (with a nice feature of falling back to CPU if nothing else is available). WebCL is a JavaScript binding for OpenCL that allows to use GPU computing in client-side web apps. Samsung not only released the WebCL implementation, it also did it under BSD license, meaning it's free to use for any purpose.

On more personal side, I got a couple of machines for an experimental XenServer pool, meaning I can test new features without jeopardizing production VMs. Shiny!

Xen support in mainline Linux kernel

2011-06-02T13:29:00.001+02:00

For the past two years, Xen infrastructure has been getting included in the Linux kernel piece by piece. It's finally done. A nice coincidence is that new version we'll be called 3.0 instead of ~~2.6.30~~ 2.6.40 - just like Xen was the feature so important it justified the change (in reality, there was no single large addition, just the sum of small changes since 2.6.0 made today kernel something completely different).

Soon an ordinary Linux system will be able to run as Xen dom0 (host) without any changes in the kernel, just like it is with KVM, VirtualBox and some other virtualization solutions. I hope it will stop the decline of Xen: when it's no harder to setup then its competitors and offers better performance, it's becoming and interesting choice again.

Password storage, part 2 - my choice

2011-03-31T10:46:00.000+02:00

Note: read Part 1 for general considerations.

After some tries I settled on Keepass family. The original app works on Windows and has two lines: Keepass Classic (1.x) and Keepass Professional (2.x). Before you jump to the latter (it has a bigger version number and it's called professional, so it must be better, right?), bear in mind that Keepass 2 is written in .NET. What's wrong with that?

- It severely limits the number of systems that can run it. While you can use it on most versions of Windows and (with Mono) on Linux, BSD and MacOS X, .NET runtime needs to be installed. What if you need to run it on someone else's machine? No problem in carrying a few MB portable Keepass binary with you, but .NET Framework is large and invasive.
- It's way slower then Keepass Classic.

Supported systems

Keepass 1.x only runs on Windows, but there are ports and clones for almost any system and device: Linux, MacOS X, PDAs/smartphones (iPhone, Blackberry, Android, Windows Mobile) and even dumb phones (using Java Mobile, almost any phone can run it). All of them use the same file format. I mostly use KeepassX on Linux, supplemented by Windows and phone versions.

Windows version can be integrated with Portable Apps Suite and Bart Preinstalled Environment, two great tools that many IT professionals carry around. No problem in using my passwords on a friend's computer, got Keepass Portable and the database is backed on my USB drive [*].

Organizing passwords

Image via Wikipedia Keepass can organize your passwords in the tree structure (eg. you can start by dividing into Personal and Work, than in Personal into Shops, Forums, Websites...). Useful if you have dozens of them. Even better, you can search your database. When adding a new entry, you have a choice of writing a password or generating a new one with a given length and set of characters. There's even an option to generate pronouncable passwords. I usually go with a long random string of mixed case letters, numbers and several special characters.

Keepass database is a single file. It means in case of corruption, you lose it all, so Keepass automatically writes backups. It's protected with a strong encryption, so it's safe to send it over the network. Just remember to choose good password: containing numbers, special signs and REALLY long - like, 20 characters or more.

Noticed that Keepass gives you an option to see the password you're typing instead of asterisks and to correct it in case of a mistake? All apps written by security-savvy people do it. Asterisks give little protection against someone looking over your shoulder (they can see the keyboard anyway!), but discourage from using strong passwords.

Integrating with other applications

Keepass can import passwords from several applications, including Firefox, and export to XML (both human and machine readable) or TXT. There are plugins to integrate it with web browsers and other apps. I don't use them though, there's a simpler and more secure option: press Ctrl-Shift-A while visiting a page, Keepass will auto-type your username and password. It matches application titlebar to the entry's Title and URL field. To make sure the match is correct, install Hostname in Titlebar extension in Firefox. For other apps, use Ctrl-C to copy password to clipboard (Ctrl-B for username). Keepass will clear your clipboard after a specified time (20s by default).

-------------
Footnote:

[*] Some would consider it insecure: if you don't control the machine, it might steal the password you use - or ALL passwords once Keepass database is unlocked. In my opinion it's secure enough provided that you:

1) trust your friends not to intentionally steal your password,
2) trust your friends to secure their machines.

While condition 1 is generally always met, condition 2 rules out many machines. When I need to work from an untrusted system, I use Keepass on my phone. It's damn inconvenient to type a Keepass password on the phone's keyboard (obviously I've chosen a strong one) and even worse to read on the phone and type on the computer the password from Keepass, considering that most of them look like F25c6D-SGe#r5vK;DVb5, but I do it maybe twice a year.

Password storage, part 1 - considerations

2011-03-30T13:25:00.000+02:00

Typical user needs passwords for dozens if not hundreds of different systems: computers, mail and IM accounts, forums, shops, newsletters etc. It is clearly impossible to remember more than a handful of good passwords, which leaves two choices:
- write down the passwords, or
- use the same password for different systems.

Both choices can be insecure, but experts agree: reusing password is the worst choice.

Some kind of password storage system is then essential. As usual, we have many options, but neither of them really stands out. I'd like my password storage to have the following six characteristics, ordered from the most important:
#1 Security
#2 Reliability
#3 Ease of use
#4 Universal access
#5 Portability

Security
Obviously, you don't want anybody to see your passwords. It's easy to say you want maximum security, but this requirement t interferes with all other characteristics.

Reliability
In the digital age, if you loose all your passwords, you might as well restart your life. OK, maybe it's not that dramatic, but you'd waste days to regain access to your data and some will be irrepplacably lost. You might even consider reliability more important than security, I won't argue. Questions to ask when assessing the system:

- Is the storage easy to backup?
In case of the software, that rules out everything that stores passwords in locations like Windows registry. A hidden file somewhere deep in your home directory is marginally better. A known location is preferred.

- Is it likely that the storage can be corrupted? What then - do I loose all information or a part of it?
That again rules out registry. Also, be careful with the software that stores all your passwords in one encrypted file. A one byte error can make it inaccessible. On the other hand, one file is trivially easy to backup, so there's no excuse not to have dozens of copies scattered everywhere.

Ease of use
If the system is too complicated, people won't use it. Simple. But many so-called security professionals get it wrong. You can see it in every other office. If you implement authentication system that requires complicated, often changed passwords and limits access to data, you'll probably find out your users share accounts and post the password on the bulletin board.

How does it apply to a password storage system? Ideally, it should be completely transparent: you need to login somewhere, magic happens, you're in. Browser password manager get's close to the ideal, especially when not protected with master password. Convenient, insecure, unreliable and limited to one app.

On the other end of the spectrum is a password written down on a piece of paper, sealed in an envelope and stored in a safe, protected by walls, locks, alarm systems and armed guards. When you need it, go to the safe, authenticate to the guard, disarm alarm system, use the key and code to open the deposit box, break the seal, take out the paper. Codes to nuclear warheads where stored that way. Guess what? After the cold war had ended, it turned out some of the codes were set to a string of zeros. Lesson learned: even in maximum security environment you can't make your system too hard to use or it WILL be circumvented.

Universal access and portability
Ideally, it should be equally easy to use your system with every website and every application. Taking your passwords from one computer to another (e.g. on a USB drive or over the network) is very convenient, but might clash with #1. If you use different operating systems, you want your system to work with all of them. Maybe you need a command line and a GUI version.

What are the choices?

1) Paper and pen
Don't laugh, the low-tech solution actually have some strong points. Paper is quite reliable (unless you have a terrible handwriting). It can be backed up with a copy machine. It's portable. It works with every opearing system, website and application.

On the other hand, paper storage scores terrible on #1 and #3. It's inconvenient. Typing the password you read from the paper is slow and prone to errors. The stronger the password, the worse it gets. It's also hard to keep the system secure, especially with dozens of passwords: that means either a large sheet of paper or many small cards, both unwieldy and devastating when lost.

However, it's easy to relatively securely store a few passwords that way - security guru Bruce Schneier recommends keeping them in the wallet. You might consider some form of simple encryption or obscufation. It won't stop a commited attacker - anything simple enough to do in your head is also too simple to be secure - but might be enough to stop a kid who accidentaly found your password card.

2) Text file
Only marginally more advanced then paper and pen, text file with all logins and passwords also has some merits. It's can't break. It's easy to copy. It works with every system with one obvious exception: the password to the computer you use to read the passwords has to be stored elsewhere (in organic memory, maybe). You can use it with Notepad on Windows and vi on Linux, copy-paste or retype passwords to every application.

To be secure you should never keep it unencrypted. And you absolutely shouldn't send it unencrypted. This reduces the portability. You can, however, choose an encryption software that doesn't require installation and keep on the USB drive together with your password file. You can find a program compatible with OpenPGP standard, they work on every operating system. However, it reduces ease of use which wasn't particularly great to begin with. Retrieving password requires decrypting, finding an entry, copy-pasting and securely deleting an unencrypted file. Adding new password is even worse: decrypt, enter your login, create a password (either with external tool or just by creating a random string, whatever you do, never reuse an old password), encrypt, securely delete a clear-text copy. Even security-savvy users don't want to bother doing that many times a day. Still, it's a good backup system - write your passwords once, enrypt with a strong password, store it everywhere you can.

3) Stored in an application
Mail client, instant messenger, web browser - all can store the passwords for you. It's handy and everyone but the most paranoid use that option to some extent. If you keep some considerations in mind, it's quite secure. Your computer should be protected anyway. If it's not, whichever option you choose, you're not safe: even keeping your passwords on the paper won't protect you from someone recording keys pressed on your keyboard. Never use it as your only password storage. Otherwise, one day you'd want to access your account with another application or from a different machine and you'll find there's no way to recover your password.

4) Password storage software
Generally, it's a graphical user interface to option number 2. Password store encrypts your secret data, you use a master password to unlock it. It's easier to add new entries, most apps have an integrated password generator. However, they usually work on one operating system only, be it Linux, Windows or MacOS. Some require installation, I recommend against them - choose one that works from the USB drive. Also note where it stores the passwords, so you can backup the file regularly. Passwords stores sometimes integrate with applications that require passwords, others have an auto-type feature, worst case - use clipboard.

It's quite clear that password store is the most secure option while being almost as convenient as storing password in the application. Which software I've chosen? Read part 2.

Xen 4.1 released

2011-03-29T11:49:00.000+02:00

Image via WikipediaAfter almost a year, a new major version of Xen hypervisor was released. Xen 4.1 adds several performance enhancements and support for very large systems.

There's also a new API: Xen 4.1 includes support for old xend/xm tools, but they're deprecated. You should migrate to xl command and libxl toolstack. There are some significant differences, so upgrade path is not easy. A terrible idea in my opinion: Xen was always harder to run then its competitors, adding another obstacle definitely won't help. That is, unless you run a high-level tool, in which case you don't care about the gory details.

Now if only I can get my hands on a physical machine to run some tests...

EC2 can now import VMware images

2010-12-17T11:58:00.000+01:00

Amazon recently announced a new feature: importing VMDK images into EC2. Right now the support is limited only Windows Server 2008 SP2 (now that is specific!), support for Linux and other versions of Windows is coming soon.

There are some other limitations that can make a perfectly good VMware image unusable in EC2. Remote access needs to be enabled for obvious reasons. Also anti-virus and intrusion detection software cannot run during conversion process, but you can enable them again later.

Gartner: 6 myths about server virtualization

2010-11-22T15:28:00.000+01:00

A short but interesting read. Thomas Bittman, analyst at Gartner, wrote summarized some common misconceptions about server virtualization:

Server virtualisation is not as hot as it once was
Server virtualisation is mainly for large organisations
Server virtualisation saves money
Server virtualisation is a commodity decision
Virtualisation is an IT thing
VMware Has little competition in the server virtualisation market

What's new in Fedora?

2010-11-17T13:10:00.000+01:00

I'm more into Debian-based distros, but a freshly released Fedora 14 and upcoming Fedora 15 pack some very interesting features. It almost made me switch right away and I might still do that when 15 comes. Keep up a good job, people of Fedora!

Fedora 14

Virtualization
Virtual2Virtual migration scripts helps with transition from Xen to KVM. I don't like it, but I expected it and if I have to do such transition in future, at least I hope it'll be easy. Xen Dom0 support is missing from a stock Fedora kernel.

SPICE
A new rendering protocol for remote desktops, much improved compared to RDP and VNC. An interesting feature is placing the graphic processing either on the client (which reduces network and server load) or on the server (for thin clients with insufficient CPU power). According to Redhat, it's even possible to watch full HD video on remote desktop. You can also use USB devices plugged to a terminal - a feature long available in commercial, Windows-based solutions, but missing in a Linux world. For more:
http://www.redhat.com/virtualization/rhev/desktop/spice/

Libjpeg-turbo
It might seem like a small change, but if you tried it and if you work with digital images, you know better. I've used it for a few monhts and it gives a massive speedup for Gimp and other photo-related software.

D compiler
In the last few years we've seen a wave of new programming languages. Most of them run on JVM, CLR (.NET) or other such platform. D is one the very few that can be compiled to machine code and yet has all the features that programmers learned to love in new languages. I've never tried D, not being much of a software developer myself, but I like the idea of having a better language for system programming than a 40 year old C. Fedora 13 features a GNU D compiler as a part of a GCC collection, Debian Squeeze has it too.

Perl 6
Perl is in transition to a new, greatly redesigned version. Again, I'm not going to use it myself (I used to write my sysadmin scripts in Perl, than moved to Python) but the developers will get a better, cleaner tool which means more great free software for me.

Fedora 15

Libre Office
A branch of OpenOffice that appeared after Oracle's acquisition of Sun. Probably all distro will switch to Libre Office in 2011.

Systemd
An init replacement, alternative to Upstart used in Ubuntu and other distros (including Fedora 14). You can find more information and a comparison of these two systems in this insightful, but very long post by a Systemd developer. I also have some experience with Solaris SMF and I found it superior to Upstart, although confusing and overly complicated (due to XML configuration and reinventing the names for pretty much everything). Systemd started later than any of these and learned on their mistakes, so it'll likely be the Best Init System Ever. Granted, most people don't care much about that, they just want there daemons started. But believe me, it's the big thing if you need to dive into the guts of your system.

No SUID
No binaries will be setuid root on Fedora 15. This will make the system more resillient against yet unknown security holes.

Amazon introduces EC2 GPU instance

2010-11-15T14:23:00.000+01:00

Image by Philippe Lin via FlickrGPU programming is an emerging trend in High-Performance Computing. We tried it here and while it doesn't work for most types of computations, it's enormously useful for the others. Now you can try working with nVidia Tesla without buying the expensive hardware. A new EC2 instance called Cluster GPU or cg1.4xlarge provides some serious processing power:
- dual Tesla M2050 board (with 448 cores and 3 GB RAM each)
- 2x quad-core Intel X5570
- 22 GB RAM,
- 10 Gbps Ethernet.

Today there's enough software utilizing the stuff to start working right away: CUDA bindings for C, C++, Python, Java and other languages, OpenCL libraries, GPU-enabled math software. Also Amazon's Elastic MapReduce can utilize both GPU and ordinary Cluster Instances, as well as regular EC2 machines (note to self: we just lived to the day when a virtual machine is regular and a physical machine is a special feature).

Ubuntu 10.10 - what's under the hood

2010-10-22T13:28:00.000+02:00

Most of my machines run a current LTS, previous LTS or have been moved to Debian. But one test server just got upgraded to Ubuntu 10.10. There are loads of post about changes in the latest Ubuntu, but they all focus on what's visible. Which is - not much. Let's dig a little bit deeper.

This time, there are no fundamental changes like a transition to Upstart. The biggest difference is in the kernel - you get 2.6.35 instead of 2.6.32. If you keep an eye on kernel development (eg. on Linux Weekly News), you know there were much code cleanup and new drivers. No major redesign though.

Ubuntu now supports LUSTRE - a cluster file system that's used in the places like my datacenter. It promises high performance and reliability. It also usualy fails to deliver them, especially if not configured by experienced admins. I'm not going to run a LUSTRE server in a predictable future, but I might need a client.

Development tools were also updated. The GCC suite is now 4.5 - this version contains two major new features:
- Link-time opimization that produces faster binaries,
- A plugin architecture, something that GCC developers avoided for years.

There are also LLVM 2.7 and 2.8. Both versions were significant steps. LLVM 2.7 is a first edition that can compile itself (a sure sign of compiler's maturity). LLVM 2.8, in addition to other improvements, comes with DragonEgg - a GCC plugins that let's you use GCC front-end with LLVM optimizers and code generators. If you don't know what's all the fuss about LLVM, read their website or at least Wikipedia page. It's way too complicated and important to summarize. Suffice to say it allowed for real progress in research on compilation techniques, it's integral part of Apple's operating systems and it inspired notable improvements in other compilers.

Use Amazon Web Services for free

2010-10-22T13:23:00.000+02:00

Starting from November 1, Amazon will give some free services for new customers. The offer goes well beyond the usual "try before you buy". For a year, each month you can get:
- 750 hours of Amazon EC2 Linux Micro Instance usage
- 750 hours of an Elastic Load Balancer
(Mind you, a 31-day month is just 744 hours - you can run one machine continuously for free!)
- 10 GB of EBS storage and 5 GB S3 storage
- 30 GB of internet transfer (15 in, 15 out)

Also all customers, new and existing will get another free usage tier that won't expire in 12 months.
- 25 hours and 1 GB of storage for Amazon SimpleDB
- 100 thousands requests for Simple Queue Servisc and Simple Notification Service.

See AWS website for details. It made me wish I didn't sign up a couple of years ago.

Creator of ZFS leaves Oracle

2010-09-29T14:55:00.000+02:00

Jeff Bonwick, creator of ZFS and head of the ZFS team announced he's leaving Oracle and starting an own business. He states he enjoyed working for Sun and later for Oracle. Somehow I only believe in the first part. There were more high profile engineers leaving, including Java creator James Gosling, as well as several thousands anonymous employees laid off. But this time it's personal. For me, ZFS is the single reason to choose Solaris over other operating systems.

The future of ZFS is probably safe, it's mature enough, it's open source, thousands of companies want to see it supported. Still, can't say I like this message.

EC2 quick news - September 2010

2010-09-21T10:24:00.000+02:00

Oracle Certifies Enterprise Software on Amazon EC2
Several Oracle products, including the most known database, have been certified to run on EC2. Oracle also prepared VM images. It's not clear yet, but looks like you can use the same image on EC2 and on your Oracle VM Server, but without live migration between the two.

RSA key import
Now you can import your own RSA public key instead of having it generated by Amazon. For the truly paranoid or for those that need to use the same key for many applications. A variety of formats is supported, including the OpenSSH format.

Idempotent Instance Creation
This strange name refers to a new mode supported by EC2 tools: you can issue the same command many times and it will only run once. Useful if you often experience errors and you're not sure if you should wait a bit longer or try again.

Resource tagging and filtering
You can now attach up to 10 tags to EC2 instances, AMIs, EBS volumes and snapshots, and Amazon VPC resources such as VPCs, subnets, connections, and gateways. Web front-end and command line tools now support filters with the usual '?' for a single character and '*' for any string. Multiple filters can be combined with AND and OR. Filters work on various fields, such as names, instance types and availibility zones. All of these make your infrastructure easier to manage of you use hundreds or thousands of resources.

Most new features are described in a new EC2 Developer Guide.

Official EC2 Linux instances

2010-09-16T16:01:00.000+02:00

EC2 users can choose from a wide range of predefined Linux virtual machines. Now there's one more, but this time it's official, created by Amazon. Looks like it's based on Fedora. Standard installation is quite minimal, but you can add packages.

Nice thing for beginners. You don't have to choose from thousands of images, you're sure it's created by professionals who know the insides of EC2 infrastructure. Of course, I'd rather use Debian or Ubuntu...

EC2 micro instances

2010-09-13T14:42:00.000+02:00

Few days ago Amazon announced a new type of EC2 machine - micro instance. It has 613 MB of RAM, supports 64-bit systems and costs only 2 cents per hour (3 cents for Windows version). It also has less CPU power and I/O capabilities, but it's well suited for many applications that don't require much resources. I can thing of many possible uses: monitoring tools, revision control systems, low-traffic web servers, training, testing. Micro instances come without ephemeral storage. You boot them right from EBS and keep your data there, which is fine unless you create GBs of temporary files.

If you're not sure if it has enough power, consider two things:
- The sustained CPU power is low, but if you need a short burst, you might get it if the cycles are available. In fact, you can get twice as much as a small instance! No guarantee though, still useful if your app generates variable load.
- As with all EC2 instances, you can convert it to a bigger type.

See also this post on running Ubuntu on EC2 micro.

Live migrating Ubuntu to Debian - howto

2010-09-02T15:19:00.000+02:00

0. Prerequisites
You'll need:
- unlocked root acount,
- about 4 GB of disk space,
- pen and paper to take notes about things that failed and need to be repaired later,
- ability to ignore warnings and error messages, some of them pretty serious
- about 6 hours, excluding download time (I have a Debian mirror next room, so dowloads are almost instantenous)
1. Logout from your desktop environment
Switch to console (Alt-Ctrl-F1). Login as root.
The idea is to run as few apps as possible, but don't switch to single user mode - you might need virtual terminals.
2. Remove some software
This kind of migration is tough on your package manager. Some packages are newer in Debian, some are older. Some apps are divided between several packages in a different way. Files belong to another package. There's no way Apt will get everything right. To make its life easier I removed several apps. The point is to remove only a few apps (you'll have to reinstall them later) with lots of dependencies.
My choice was:
- KDE apps - I use Gnome and only a handful of KDE apps - by removing them I got rid of dozens of KDE libs
- build-essential and dkms - this removed all compilers and development libraries
- texmaker - it removed all the LaTeX stuff that I rarely use anyway
- chromium, firefox, openoffice.org, gimp
- gdm - depends on Ubuntu-specific packages
- Ubuntu-specific software: apport, software-center, indicator, usb-creator-gtk, plymouth, everything with ubuntu in its name.
Note that I couldn't remove all these at one point - some of the apps would remove too much. If that's your case, skip it and try again later.
3. Remove dependencies
Get rid of unnecesary libraries with:

apt-get autoremove
orphaner

If the latter gives "command not found", apt-get install deborphan.
4. Change /etc/apt/sources.list
Add Debian Squeeze repositories (choose a mirror close to you) and whatever you may need, I added unnoficial Multimedia repo:

deb http://www.debian-multimedia.org/ squeeze main non-free

5. Install Debian package signing keys
Apt will complain about untrusted packages. Start by:

apt-get update
apt-get install debian-archive-keyring debian-multimedia-keyring

For some reasons it wasn't enough on my box, I had to install the key manually:

gpg --recv-keys 9AA38DCD55BE302B --keyserver subkeys.pgp.net
gpg --export 9AA38DCD55BE302B | apt-key add -

6. First Big Cycle - upgrade packages that are newer in Debian
There's no way you'll get all the packages reinstalled in one try. My way was to reinstall some packages, fix errors, remove unneeded dependencies, try again. The cycle looks like this:
- [upgrade command]
- apt-get -f install
- [manual fix]
- apt-get autoremove
- orphaner
What's an upgrade command? There's a subtle difference in the way apt-get and aptitude resolve dependencies. I did apt-get upgrade first. It only deals with easy cases, so it'll upgrade a few packages in one cycle. When it can't find anything to upgrade, switch to apt-get dist-upgrade. Try it for a few cycles, than do aptitude dist-upgrade.
After installing a few packages, upgrade command will fail. First try automatic repair with apt-get -f install. If it still doesn't work, see what the problem is. Usually a package tries to overwrite a file that's also present in another package. Install it manually, using dpkg and full path to the package file:

dpkg -i --force-all /var/cache/apt/archives/package-name.deb

You can also remove Ubuntu version and install the Debian package next. Again, use dpkg instead of apt-get, --remove packagename will remove the software, but leave the config files intact, --purge will also remove configuration and often works better. See also fixes for specific packages at the end of this howto. Write down any problems, maybe the best way is to remove the package and reinstall later.
If there are no partially installed packages, remove crud with apt and orhpaner and upgrade some more. You might run apt-get clean at the end of the cycle if you're low on disk space.
7. Prepare for downgrade
Edit /etc/apt/preferences, create the file if it doesn't exist.

Package: *
Pin: release o=Debian
Pin-Priority: 1001
Package: *
Pin: release o=Ubuntu
Pin-Priority: 600
Package: *
Pin: release o=Unknown
Pin-Priority: 600

Exact numbers don't matter, just keep Debian priority higher than 1000, it means that matching packages will be installed even if it means downgrading.
7. Second Big Cycle - downgrade packages
This works the same as the first cycle, only now the upgrade command will do a downgrade. Don't forget apt-get autoremove and orphaner.
8. Check if all packages have been replaced

apt-get install apt-show versions
apt-show-versions | grep -v squeeze

This will show all packages that don't come from Debian Squeeze. There might be some claiming to belong to Lenny, that's OK.
9. Make sure your system will boot.
See if grub and init scripts upgraded correctly. Only then reboot to Debian.
10. Fix your system.
Check config files, install back packages that you removed. Remember that Firefox was renamed to Iceweasel in Debian.
Appendix - problems with specific packages
Locales
Got loads of messages about unsupported locale. Solution:

locale-gen
dpkg-reconfigure locales

mysql-client, sane-utils, python-gnomecanvas, grub-pc
Remove and install back.
Printing system

apt-get remove cups ghostscript
dpkg --purge cups-pdf

Install back later.
Unusuable X
At first X wouldn't start at all, than I got locked keyboard. I purged and installed back many packages before I got it to work. Only some of them were needed, also I attempted to remove some multiple times (won't hurt to remove a package that's not installed), that's the essence I got from my .bash_history:

dpkg --purge --force-all hal dbus udev xserver-xorg
aptitude --purge remove x11-apps x11-session-utils x11-server-utils xauth xbase-clients xinit xorg
dpkg --purge --force-all dbus dbus-x11 hal udev desktop-base hardinfo libx11-data x11-common x11-utils x11-xfs-utils x11-xkb-utils x11-xserver-utils xfonts-utils xinput xkb
-data xscreensaver-data xserver-common xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-input-evdev xserver-xorg-input-synaptics xserver-xorg-video-all xs
erver-xorg-video-intel xserver-xorg-video-nouveau xserver-xorg-video-nv gconf2 gconf2-common libice6 libx11-6 libxklavier16 libxrandr2 libxrender1 libxres1 libxt6 libxv1
aptitude install hal dbus udev xserver-xorg xorg xserver-xorg-video-intel xserver-xorg-input-all

Syslog
Didn't work.

dpkg --purge rsyslog
aptitude install rsyslog

Init scripts and networking system
Get rid of upstart first. I had problems with sysv-rc and needed to force reconfiguration. Some init scripts were installed with "dpkg-new" in the name, I had to rename them. As with X, I tried few solutions, this is distilled but stil redundant version:

dpkg-reconfigure sysv-rc
cd /etc/init.d
mv acpid.dpkg-new acpid
mv anacron.dpkg-new anacron
mv atd.dpkg-new atd
mv cron.dpkg-new cron
mv dbus.dpkg-new dbus
mv module-init-tools.dpkg-new module-init-tools
mv procps.dpkg-new procps
mv rsyslog.dpkg-new rsyslog
mv udev-mtab udev
dpkg --purge --force-all base-files ifupdown initscripts netbase sysv-rc sysvinit sysvinit-utils
apt-get --reinstall install base-files ifupdown initscripts netbase sysv-rc sysvinit sysvinit-utils

At one point, dpkg started complaining about missing '/sbin/update-rc.d'. I couldn't install anything until I did:

touch /sbin/update-rc.d
chmod +x /sbin/update-rc.d

Live migrating Ubuntu to Debian - introduction

2010-08-11T11:53:00.001+02:00

I was a long time Debian user before I switched to Ubuntu in 2005. I always liked Debian for its stability, wide choice of software and superior package management. However, at this point Debian release cycle, always long and unpredictable, got even longer than usual. A stable or even testing version of Debian meant seriously outdated software, not to mention poor hardware support. With Ubuntu, my laptop finally got a decent power management out of the box. There were other improvements, like NetworkManager and a general focus on ease of use. Plus, being a Debian derivative, Ubuntu inherited many good traits of its ancestor. It was a good choice than and for a few following years.
Then something began to break. There were some controversial design choices, but even worse was declining quality. For the last few releases I had to deal with numerous bugs, some serious, most minor but annoying. 9.10 was pretty bad, but a last normal release before LTS is always a playground for new technology. But 10.4 is even worse and it's supposed to be a Long Term Support release! Everyday I install a few updates (yes, I'm happy the bugs are getting fixed, but there shouldn't be that many in the first place) and yet my machine is still unstable. My box would sometimes lock up or get a load average of 15 for no appearent reason.
In the meantime, Debian release cycle got shorter and predictable. Many Ubuntu usability improvements were merged back upstream. Some widely reported features are not useful at all for me: I don't need Ubuntu One, I use unison on my own server; I don't need Software Center cause I prefer Aptitude etc. I decided it's time to move back. Clean install would be an obvious choice, but I decided to migrate a live system instead. There were some issues, but it worked. Did I mention that Debian has a great package manager?
If you want to know how to migrate from Ubuntu do Debian, stay tuned for the next part (update: it's here). In the meantime, you can read my post on downgrading Ubuntu. If it seems complicated or dangerous, be warned that migration to Debian is quite harder.

Howto: disable eCryptFS

2010-08-11T11:49:00.002+02:00

Encrypting your disk is usually a good idea and eCryptFS (aka Ubuntu Encrypted Home) clearly has some advantages over the competitors. It's also easy to implement on an existing system. But, if for some reason you want to decrypt your home directory, you won't get a one-click solution. The process is quite simple if you feel comfortable working with command line.

I'll use /home/igor as an example, replace it with your home directory.

1. Prepare another user account.

Either unlock your root account (sudo passwd root) or create another user account and let it use sudo. I've chosen the first option because a root account is useful for recovery if anything goes wrong.

2. Logout from your desktop environment, switch to console (Alt-Ctrl-F1)

3. Login as root or as another user.

This is critical - you can't be logged in on your regular account. Type 'mount' and see if it shows a line similar to this one:

/home/igor/.Private on /home/igor type ecryptfs (ecryptfs_sig=...)

It shouldn't be present. If it is, try 'umount /home/igor'. If it fails, you probably have a session still loged on or one of your applications is still running. If you can't find it, try rebooting and switching to console without ever logging in.

4. Edit file /home/.ecryptfs/igor/.ecryptfs/Private.mnt

Change the line /home/igor to /home/igor/Private

5. Create Private directory and make it belong to your regular account

mkdir /home/igor/Private
chown igor.igor /home/igor/Private

6. Reboot the computer

After the reboot, everything that was in /igor/home directory, is now in /igor/home/Private. There's only one step left.

7. Move the files

Switch to console, login with your regular account. Move the files:

shopt -s dotglob
mv Private/* .

First command is very important - it'll make sure that files beginning with dot will also be moved. Second one ends with dot - don't miss it!

8. Uninstall eCryptFS or leave it as it is

At this point, eCryptFS encrypts your ~/Private directory instead of your home. You can leave it for confidential files, or if you want to get rid of eCryptFS, do:

sudo apt-get remove ecryptfs-utils

Virtualization news - May 2010

2010-05-26T09:07:00.002+02:00

Open vSwitch 1.0 released

I've already written about virtual switches. See my old post if you don't know why they're so damn useful. Core functionality is already there:

VLANs
Monitoring of inter-VM communication via
Per VM policing
NIC bonding with source-MAC load balancing
Support for OpenFlow
Ethernet over GRE
Compatibility layer for the Linux bridging code
Kernel and user-space forwarding engine options
Support for Xen, XenServer, KVM and VirtualBox

XenServer 5.6 annouced

In a couple of days, Citrix will release new version of XenServer. It will be available in four versions: Free, Advanced, Enterprise and Platinum. Essentials for XenServer will not be continued, instead its functionality is integrated into paid versions of XenServer.

Most notable new feature is Dynamic Memory - ability to automatically adjust RAM provided for the VMs within specified minimal and maximal value.

VirtualBox 3.2.0 released

New version of VirtualBox - and the first one with Oracle branding - brings some interesting features:

hardware support: fully utilizes hardware virtualization support on newest Intel CPUs, multi-monitor setup (physical screen can be mapped to a specific VM)
redesigned I/O and networking stack, promising increased performance
Remote Video Acceleration
hot-plugging virtual CPUs
deleting and merging snapshots while VM is running

Plus other enhancements. As usual, there's a free version for personal use.

New Apache projects, Xen news and my learning project

2010-05-06T12:29:00.001+02:00

Six Apache projects have gained a top level status. In my opinion, a status of Apache project and especially a top-level one is a sure indicator of high quality and innovative software. The projects are: Apache Traffic Server (web proxy), Mahout (machine learning library), Tika (content analysis toolkit), Nutch (web searching engine), Avro (data serialization system) and HBase (a NoSQL database).

Which reminds me of my to-learn list. I started with Skipfish. It quickly discovered one potential XSS flaw in the software I checked. But I was dissapointed with the number of false positives and repeated messages. Still, it was much better than other tools of the same kind I had tried before.

Nginx came next, but I only had time for a very simple test with one box and httperf . Short answer: if you're searching for a fast server for static content, search no more. Nginx without any tweaking is about 6 times faster than fine-tuned Apache 2 with prefork model. Threaded Apache 2 came closer, but still it took 30% longer to complete the test. Impressive. And that's without any caching and load balancing, where Nginx really excells.

In other news, 3 Xen projects have been accepted to Google Summer of Code: migration from memory ballooning to memory hotplug, new features for XCP web interface and XCP integration with DRBD. The last one sounds interesting.

I'm not upgrading to Ubuntu Lucid

2010-04-30T12:39:00.000+02:00

At least not yet. I'll probably do it in two or three weeks. Why?

Because official release date doesn't magically make the distro stable. It does, however, make thousands of people install it and report bugs. I've been using Lucid beta for weeks and I've seen a few bugs. Granted, not as much as with Karmic. LTS releases take their packages from Debian Testing instead of Debian Unstable, which greatly improves quality. Still, I don't think Lucid is ready for production use.

Now, don't get me wrong, I'm all for installing early version to report bugs and help improve the software. I do a lot of work in a Lucid VM. I've reported some bugs on Lucid and commented on the others. But my main machine needs to be stable and I'm not going to take a risk. Not after going through the painful process of downgrading a few months ago.

Xen 4 released

2010-04-14T11:18:00.000+02:00

Version 4 of opensource Xen hypervisor has been available for a few days now. This release brings up some interesting features:

New kernel for dom0 - instead of ancient 2.6.18, we get a new subsystem built on PVOps interface of new Linux kernels. Currently 2.6.31.13 is supported, but porting it to a new version is now trivial compared to the old approach. Finally Xen gets a decent support for new hardware and running a modern distribution in dom0 is way easier.

Blktap2 - new disk I/O system. This almost deserves a separate post. Virtual block devices (hard disks) can be backed by files, partitions, LVM volumes or even network devices. VHD, the native format of Microsoft virtualization systems and a de-facto industry standard (VirtualBox and VMware can read it too) is now supported. Advanced disk formats such as copy-on-write (run multiple VMs from the same image initially and only copy the blocks that changed), sparse images, on-the-fly encryption and compression are easier and faster then before. Disk images can be accessed from outside the VM. But the real killer feature is live snapshot.

Copy-on-write RAM. VMs can now share memory. The scalability is also improved: by default Xen supports 1 TB RAM and 128 CPUs, limits can be increased.

Improved PCI passthrough - PV guests still use old mechanism by default cause it doesn't require hardware support. If your CPU and chipset combination support IOMMU, you have to enable the new way manually, see http://wiki.xensource.com/xenwiki/XenPCIpassthrough

Many other improvements. Updated Pygrub, certified PV drivers for Windows guests, disk resize without reboot, libxl - new userspace Xen management API, improved networking, physical CPU and RAM hotplug, support for Smart NICs with multi-queue and SR-IOV (one physical Ethernet card appearing as multiple PCI devices)

Qubes: secure Xen-based desktop OS

2010-04-08T15:47:00.000+02:00

Security researcher Joanna Rutkowska of Blue Pill fame is back with a new project. Qubes OS is a system that proposes an interesting approach to security:
- Applications HAVE and WILL HAVE security flaws.
- The holes eventually WILL be exploited.
- The best we can do is isolate, so an intruder won't get access to all the data.

This is already done in the server world. Nobody runs an all-purpose server if one can avoid it, dividing between multiple machines (physical or virtual) makes life easier. It doesn't work that well on the desktop.

Let's say you use Firefox for your casual browsing. Even if you are cautious and use a non-Windows OS, you can't completely rule out a possibility to get it infected with malware. Then you use the same browser for online shopping or banking. Next day you notice your money was transfered to a Nigerian bank. How about using another browser for banking? Better, but not 100% proof. Another process can still read or alter other app's data. OK than, another browser from a different user account? Much better, now you're safe... unless an intruder already got a root access.

VMs add another layer of security at the same time adding complexity. I admit I considered using this approach on my personal machine. To have my data and important apps on one VM, use another for potentialy dangerous activities (that includes web browsing, email and pretty much anything that has to do with network or removable media) and another for high-security stuff. But it'd be too much hassle. Say I found a PDF document I'd like to keep. I'm supposed to open it on a "dangerous" VM, see if it's OK (how can I be sure anyway?), then move it to a "work" VM. How long before I do something in a wrong VM, either by mistake or out of laziness? And how long it'd take me to setup such system anyway?

Here's how Qube OS does it (or aims to do - it's an early development release and some of the features are only planned). Lightweight Xen-based VMs are spawned transparently. They all run an X server but without any desktop environment. Only one privileged VM runs a desktop environment. All VMs share the same screen, start menu, system tray etc. With one click, you start a Firefox in a "work", "banking" or "personal" VM, their windows are distinguished by a coloured border. Networking stack runs in another, unprivileged VM. Yet another one, not connected to a network, is a secure storage. You can copy/paste text and copy files between VMs, but it requires confirmation and can only be initiated from a priviliged VM. The VMs share system files, so you upgrade all of them at once.

That takes away about 90% of the hassle. The remaining 10% is still too much for the average user. But if your security requirements are higher than average, Qube OS is worth a look. A modern CPU with VT-d and IOMMU is required to use all features and you'll need plenty of RAM.

If you think it's too complicated and decide to stay with one machine for everything, just use a separate browser for banking. Please.

See also: article on virtualization.info

Ubuntu Lucid window controls

2010-04-08T10:31:00.000+02:00

I've been playing around with Ubuntu 10.04 Lucid Lynx inside a VM for a few weeks. Like many people I was annoyed with the decision to move window controls. Fortunately, it's easy to move them back where they belong. Fire up a terminal and paste the following lines:

gconftool-2 --set /apps/metacity/general/button_layout \
--type string "menu:minimize,maximize,close"

Xen 4 on Ubuntu Lucid

2010-03-29T15:27:00.001+02:00

Excellent post on Xen Blog - Steps to try Xen 4.0.0 Release Candidate 8 on Ubuntu Lucid 10.04 64 Bits.

Be careful, that means running experimental software on top of other experimental software. Ubuntu Beta can be quite unstable even a few days before release, as I found out in the past.

Software to learn

2010-03-23T17:02:00.001+01:00

Here's software that I'm going to try in a near future.

Distributed processing

Hadoop
Hadoop is a distributed computing environment, best known for its implementation of MapReduce. I've already played around with Hadoop. It's easy to start, cause you begin with MapReduce only, no need to run all the modules on the fist day. But now it's time to learn some more.

Mahout
Apache Mahout is a suite of machine learning libraries. It works on top of Hadoop, which means it should be easily scalable to thousands of machines. Machine learning is not my cup of tea, but I'm going to have a quick look.

CouchDB
You can't work in IT and not hear about NoSQL movement. I've never used a non-relational DB in my projects before, time to change it. CouchDB goes first, since it's widely deployed in both small-scale (Ubuntu's DesktopCouch) and large-scale projects.

Web

Skipfish
A fully automated, active web application security reconnaissance tool. Not much to learn really, I'm just run it against my servers right now and I'll see the results in a few hours.

nginx
An extremely high performance HTTP server supporting static files, caching and proxy, even dynamic content with FastCGI. It's often orders of magnitude faster then Apache.

Memcached
Memcached stores object in memory. Technically, it's a database, but the name suggests the most common use: to cache objects (eg. database queries) in web application. See wikipedia article for good introduction. Another way to drastically increase performance of a web app. I've used memcached with MediaWiki (where it's just a matter of a few lines in a config file), but never used it for my project.

You probably noticed that half of these are Apache projects. I didn't intentionally select them that way, but now I'm going to have a closer look at other Apache software - besides those listed and those I already know. It's no coincidence that one organization supports innovative distributed programming frameworks, the most successful web server, widely used J2EE app server, popular version control tool and build automation system, Java libraries and a load of other software. Apache Foundation clearly have both high standards and a knack for choosing the right projects.

IE 9 will use GPU to render pages

2010-03-17T15:25:00.000+01:00

News about the IE9 platform preview are everywhere today. One important information is at the bottom of Microsoft's announcement. IE 9 will render pages using GPU if the right device is available. It's a great idea (in a wrong browser) - video card is extremaly underutilized during web browsing. I hope other browsers will catch up soon.

Eucalyptus howto

2010-03-17T15:24:00.000+01:00

Canonical, Eucalyptus and Intel prepared a whitepaper on cloud design and deployment. It's clearly the best howto on Eucalyptus I've ever seen. And I admit it even though I've written one too (and I'll use this opportunity to link to it). Their whitepaper is much better.

Nexenta: Power of Solaris with usability of Linux? Not yet

2010-03-16T10:26:00.000+01:00

Being a newbie Solaris user, I discovered what seasoned admins know: this system has its strengths, but daily administration tasks are a pain in the ass. Really neat and modern solutions like SMF coexist with software that looks like it comes straight from 1970s. Simple process of software patch could take hours.

Then I discovered Nexenta. It's the OS distribution that combines OpenSolaris kernel and some system software with userspace from Debian and Ubuntu - including APT! Just imagine - a system with ZFS, SMF, zones and at the same time - proper package manager, sane directory tree and thousands of apps. I downloaded Nexenta 3.0 Beta and installed it into a VM right away.

First impression was positive. The installation was simple, after booting I got a bash shell instead of lousy sh. But it didn't take me a minute to run into a first bug. I tried using aptitude to see available packages and immediately discovered it doesn't react to keystrokes (appareantly, it's a long known bug). Say goodbye to my dream of proper package manager. Some other text-mode software had issues with keyboard and fonts as well. Now, after a few days of testing (read: playing around), I think this system is promising, but it requires a lot of work before it's usable. Just installing the packages you can see that some of them were taken from Linux without much modification - their pre/post install scripts try to mess with Linux-specific files in /proc or modify Upstart settings.

I can't recommend Nexenta to neither server nor desktop. I doubt it'll ever be able to become an alternative to Solaris with the small number of developers the project has. I hope it'll serve as a testbed for ideas to be implemented in (Open)Solaris and in a few years we'll see more convergence between Linux and Solaris.

Introduction to desktop virtualization

2010-03-03T14:11:00.002+01:00

I'm getting familiar with Xendestop - I've attended a workshop recently and read a bit. Desktop virtualization is less popular then server virtualization, but it's slowly getting more attention in large corporations, government organizations, universities etc. - everywhere you have dozens or hundreds of desktops sharing the same configuration.

How?

Desktop virtualization comes in many forms. You can use thin client or PC as an endpoint. Usually in addition to standard protocols like RDP or VNC, they use a proprietary one with more features - for example, handling remote hardware (thin clients these days come with soundcards, cameras, DVD drives, USB ports), better network utilization. On a PC client, you can use a specialized software or a browser-based client - useful for one-time access from a remote machine. Some PC clients can also work offline

On the server end, you can run multiple desktop OSes on one server. This form, called VDI, is the most popular for obvious reasons - lower costs, better hardware utilization, less power, cooling and rack space. Another solution is to use one physical machine per client. It's useful when you need a specialized hardware on the client but still want to benefit from other advantages of desktop virtualization. You can also virtualize a single app - it'll run in it's own isolated environment, but from a user's perspective it'll look like it's running on his or her system. Xendesktop can combine all three forms.

Why?

Decoupling software and data from hardware is one advantage. If your terminal fails, replace it and continue working. Log on with your credentials to any box in the company and you get your own desktop.

But the real benefit is centralized administration. All desktops using the same operating system can be run from the single image. Ever had to install patch on 10 machines? How about 1000 and a full operating system upgrade? Plus, you don't have to experiment on a live system - prepare a new image, test it and then change the users' configuration: at the next login, they'll get a new system. Users can't do any damage to their OS, the machine image is read-only and user data is isolated from it.

Why not?

There are downsides, too. You need a reliable, redundant hardware and skilled operators, otherwise a single failure will make your whole IT infrastructure unusuable. The system is best suited for office employees. IT engineers will want to meddle with their setup, run different OSes, use strange hardware. People who work with graphics-intensive apps, eg. for video editing, would notice a degraded performance. Current systems handle low-quality video (eg. flash animations) quite well, but full resolution movies or games would quickly bring your network to its knees (actually, the system that can't run games and movies can be considered superior in a workplace).

Parallel processing in shell scripts

2010-03-02T15:02:00.001+01:00

Shell scripts on multiple CPU cores

Most of the computing tasks at home are interactive, but there are some common batch jobs like file compression, audio/video encoding, graphic manipulation. Many Linux users prefer command line tools for such tasks. It's easy to run them on multiple files with for or find. But if you have multi-core CPU or multiple CPUs, running parallel jobs would vastly reduce the time it takes to process all files.

PPSS is a shell script that executes commands in parallel. There are many tools like that, what's different about PPSS is that it's dead simple to use. Let's start with an example. I put a bunch of JPG files into ~/test-ppss and I want to convert them to tiff. All I need is:

ppss -d ~/test-ppss -c 'jpg2tiff '

As you can see, it only needs two paremeters:

-d /directory/with/files/to/be/processed
-c 'command '

Note the trailing space in the command. It's the simplest form - filename will be appended at the end.

OK, I cheated a little bit. You probably won't find jpg2tiff command on your system. It's a shell script I created that contains:

convert -compress lzw "$1" "`basename "$1" jpg`tiff"

It uses "convert" from the ImageMagick package. I created it to have an easy to grasp command line for my first PPSS example. Now let's try without the script. How do we put a filename somewhere in the middle of PPSS command line? PPSS sets the $ITEM environment variable, so all you need use:
-c 'command -i "$ITEM" -s some -o other -a arguments'.
In our example:

ppss -d ~/test-ppss -c 'BASENAME=$(basename "$ITEM" .jpg); convert -compress lzw "$ITEM" "$BASENAME.tiff"'

As you can imagine, it took me more than one try to write it, so a helper script might be a good idea.

There are other arguments for PPSS. By default, it run one parallel job for each CPU core it detects, but doesn't take HyperThreading into account. You can change it with -j. It probably won't make it run faster, as HT is not as efficient as multi-core or SMP. You can also set the number of tasks manually with -p. It's useful if you want to keep one core for interactive tasks.

Shell scripts on multiple machines

PPSS can distribute tasks to other machines using SSH. The setup is slightly harder, but still way simpler than with typical distributed programming frameworks. All you need is:
- Passwordless (key-based) SSH authentication. You should create accounts and keyfiles for this sole purpose. Don't use your login account.
- Config file for PPSS with node list and parameters.
- Optionally, SMB or NFS share with files to be processed. PPSS can also send input and get the results with scp, but unless the files are very small and the task very CPU intensive, it won't have much sense. All the gains from parallel processing would be eaten up by the time it takes to distribute the files.

I haven't tested it yet, you can see the explanation and read the manual on PPS website.

Virtualization news: KVM, Xen, AlacricityVM, VirtualBox

2010-02-15T09:56:00.002+01:00

Numerous changes to KVM got into a mainline Linux kernel. Some are - as usual - performance improvements. Also, other software can now use CPU virtualization extensions when KVM module is loaded, but not in use. This should greatly decrease a number of people complaining about their VirtualBox or VMware not using VT-d, who didn't notice their distribution auto-loaded kvm.ko module.

Novell, the company behind OpenSuSE, is in process of switching from Xen to AlacricityVM, a version of KVM designed for lower I/O latency, as their default hypervisor. Recently they tried to get it included in the kernel. Unfortunately, the result was flame war between AlacrityVM developer Gregory Haskins and some KVM developers and Linus saying "no" due to various reasons. Main reason being not enough interest outside of AlacricityVM developers group, kind of obvious considering an early, generally unusuable stage of AlacricityVM.

VirtualBox released version 3.1.4. It's only a bugfix release, see changelog for details. Seeing a long list of bugs, I'm wondering why I've never encountered any of them.

XenServer goes open source - again

2010-02-10T10:06:00.000+01:00

In November 2009 Citrix released source code for XAPI. This time they made available almost full source for XenServer. Almost - High Availibility component and XenCenter management console remain closed. Citrix is clearly following a proven developement model: give your product for free, then use the power of the community to enhance it and make profits from commercial add-ons and services. Both the company and the community will benefit.

Amazon S3 gets versioning support

2010-02-09T11:13:00.002+01:00

You can now enable versioning feature for a bucket. Each time you replace (or delete) a file, an old version is stored along a new one. When you retrieve a file, you get the newest version by default, but you can specifically request another one.

If you've ever used a version control system like CVS or Subversion, the concept will be familiar to you. If not, think of it as an undo operation for the filesystem - how easy it will be to recover from software or user error? Just remember to implement a scheme to prune old versions - normal S3 storage charges apply.

See this entry on Amazon Web Services Blog for details.

Xen.org pamphlet in Polish

2010-02-09T08:41:00.002+01:00

My another tiny contribution to Xen Project - a translated pamphlet.

Solaris for Linux admins - part 3: Blastwave

2010-02-08T15:33:00.000+01:00

I've already written about Solaris package management. Today, more details about Blastwave - software repository for Solaris which comes with a tool that works a lot like Apt or Yum. Enabling Blastwave takes a few minutes and several simple steps. Note that half of them deals with package signing. If you don't care about security, you can skip them (but then, get off my Internet before you become a zombie PC).

1. Install pkgutil

pkgadd -d http://blastwave.network.com/csw/pkgutil_`/sbin/uname -p`.pkg

2. Fetch the catalog

/opt/csw/bin/pkgutil --catalog

Note: you should probably add /opt/csw/bin to your $PATH. From now on I'll assume you did that and omit the /opt/csw/bin part from the commands.

3. Install required software to check package signatures

pkgutil --install gnupg textutils

You'll notice it'll also fetch dependencies.

4. Get Blastwave package signing key

gpg --keyserver pgp.mit.edu --recv-keys A1999E90

Note: don't trust me, check the key ID on Blastwave website. It might have changed. Or I might be tricking you into installing malware.

5. Set the key as trusted

gpg --edit-key A1999E90

Enter trust, than 5 and quit.

6. Enable signature verification
Edit /opt/csw/etc/pkgutil.conf. Uncomment the lines:

use_gpg=true
use_md5=true

7. Download catalog again

8. That's it
You can now install the software the way you'd do with apt-get or yum, for example:

pkgutil --install mc

(personally, I think any Unix is much more user friendly with Midnight Commander)

pkgutil --remove [package name]

will remove a package with all dependencies.

Solaris for Linux admins - part 2: package management

2010-02-05T09:35:00.000+01:00

Unlike most Linux distributions, Solaris doesn't provide an all-purpose package manager. Instead, you use separate tools:
- to add/remove Sun-provided software
- to update the system,
- to add/remove software from external repositories

They come in two flavours, text and graphical. Let's start with the latter.

I want my G-U-I

If your system runs X, you can use graphical package managers. In Utilities menu, there's an Update Manager. It works automatically unless the package requires manual intervention during update process, in which case it will be marked as "download only" (sorry, no screenshot, I already updated my system). If it happens, see patch description - sometimes it says it should only be installed in single-user mode or in some circumstances it should not be installed at all. When you're ready, run "patchadd" with the path to patch spool and the patch ID, for example:

patchadd /var/spool/patch/121431-44

There's also a GUI for package management, although you can't find it in the menu. Type "prodreg" in the terminal to run it. The tool will generally just run pkgrm/pkginst (see below) in the terminal, but might be useful for navigating a list of installed software.

Update from the command line

The update process is simple:

smpatch analyze
smpatch download
smpatch update

Example:

bash-3.00# smpatch analyze
119281-22 CDE 1.6_x86: Runtime library patch for Solaris 10
bash-3.00# smpatch download
119281-22 has been validated.
bash-3.00# smpatch update
Installing patches from /var/sadm/spool...
119281-22 has been applied.
/var/sadm/spool/patchpro_dnld_2010.01.29@15:41:11:CET.txt has been moved to /var/sadm/spool/patchproSequester/patchpro_dnld_2010.01.29@15:41:11:CET.txt
/var/sadm/spool/patchpro_dnld_2010.01.29@15:42:03:CET.txt has been moved to /var/sadm/spool/patchproSequester/patchpro_dnld_2010.01.29@15:42:03:CET.txt

Managing system software

There's a number of commands starting with "pkg" that deal with packages. The most important are:
pkgadd - installs a package from a specified file
pkgchk - checks installation of a package
pkginfo - list of known packages
pkgrm - removes a package

These tools behave like dpkg or rpm, not the high-level Linux package managers APT or Yum. Which means: they will print out dependency info, but won't resolve anything automatically, also pkgadd can install a package from a file or URL, but won't find it for you. Package names always start with a vendor's prefix, Sun uses SUNW.

Let's try removing a package. I have a default install on my testbed VM with lots of software I'll never use. Let's get rid of RealPlayer.

bash-3.00# pkginfo | grep realpl
MOZ17       SUNWrealplayer                   RealPlayer media application
MOZ17       SUNWrealplayer-l10n-de           RealPlayer media application - l10n-de files
MOZ17       SUNWrealplayer-l10n-fr           RealPlayer media application - l10n-fr files
MOZ17       SUNWrealplayer-l10n-pl           RealPlayer media application - l10n-pl files

There's a main package and three translations.

bash-3.00# pkgrm SUNWrealplayer

The following package is currently installed:
   SUNWrealplayer RealPlayer media application
                   (i386) 10.0.6.1358,REV=10.2.3.2006.03.21.05.47

Do you want to remove this package? [y,n,?,q] y

## Removing installed package instance
## Verifying package dependencies in global zone
WARNING:
    The package depends on the
    package currently being removed.
WARNING:
    The package depends on the
    package currently being removed.
WARNING:
    The package depends on the
    package currently being removed.
Dependency checking failed.

Do you want to continue with the removal of this package [y,n,?,q] y
## Processing package information.
## Removing pathnames in class
/usr/share/pixmaps/realplay.png
/usr/share/pixmaps
/usr/share/mimelnk/video/x-3gpp2.desktop
/usr/share/mimelnk/video/vnd.rn-realvideo.desktop
[...]
/usr/bin/realplay
/usr/bin
## Updating system information.

Removal of was successful.

We can clearly see how these tools (don't) deal with dependencies. They have to be uninstalled manually:

bash-3.00# pkgrm SUNWrealplayer-l10n-de SUNWrealplayer-l10n-fr SUNWrealplayer-l10n-pl

High level package manager

The prodreg software I mentioned before also works as a CLI tool. With a proper magic you can for example uninstall a package with all the dependent software. Using it is a pain in a certain body part. Getting the right package name or UUID requires a long process of "prodreg browse". I'm not going to cover it here, see Solaris Basic Administration Guide if you need it.

Other software

There are several external repositories with tons of software, generally the same as for Linux. The most popular is Blastwave. It even comes with a proper package manager (one that behaves like Apt or Yum). It downloads the software, deals with dependencies and packages are protected by MD5 checksum and GPG signature. The tool is called pkgutil. Setting it up properly requires a few simple steps, so I'll post it as a separate entry.

New Xen Documentation

2010-01-29T10:14:00.000+01:00

Xen documentation team is pretty busy lately. They have just created or updated several useful Wiki pages: overview, Xen kernels (that one is great!) common problems, best practices and others.

See the full list on Xen blog.

Solaris for Linux admins - part 1: introduction

2010-01-27T15:44:00.001+01:00

I'm working on an experimental virtualized cluster. It's a complex setup involving - in addition to a number of Linux machines - one Solaris box. This system was always popular in academia and we have several full-time Solaris administrators, but I don't want to bother them every time I need to make a small change. It's time to refresh what I know about Solaris and learn more.

I've used Linux as my main OS for over ten years. I also have some experience with other Unix systems - Solaris included - but I'm far from the level of familiarity I have with Linux. There are some notable, sometimes annoying differences in location of files, command parameters and underlying concepts, I need to use man or other docs way more often than with Linux. I decided to write down what I learned to piece together my knowledge and to create a handy reference for myself. If you'll find it useful too, that's an added bonus.

Filesystem layout

At a first glance, filesystem in Solaris looks like a mess. After a full install there are about 50 directories or symlinks in /usr, some of which clearly shouldn't be there (e.g. /usr/spool), /var and / are not much better. The reason is Solaris users really value backwards compatibility. Directories in Linux conform to Filesystem Hierarchy Standard. Whenever it is updated, there are backward compability symlinks for a year or two, then they get dropped. In Solaris you can use the same paths as in 1985 BSD od 1993 SunOS - or 2010 Linux, for that matter.

One other difference is that software is often divided by vendor, technology or usage. On Linux, almost all executables land in /usr/bin, their libraries in /usr/lib and other files in /usr/share. In Solaris, you have:

/usr/ for the usual stuff
/usr/css for Sun's development tools
/usr/dt, /usr/gnome, /usr/openwin for desktop environments (/usr/dt is for CDE - no idea why)
/usr/jdk, /usr/java, /usr/perl5 for Java and Perl
/usr/ucb* for BSD utils (ucb stands for University of California Berkeley)
/usr/sfw for GNU apps (sfw stands for Sun Freeware, I think there's a company policy for using misleading directory names)

And a dozen or two others. Oh, and software installed from Blastwave (which is often the most used) goes into /opt/csw.

Some claim it's more tidy. Others think it's a pain in the ass and I'm one of them:
- You have to put dozen directories in your $PATH.
- Often you end up with 2 or 3 versions of the same tool (eg. one from Sun, other from BSD or GNU and another from Blastwave). Sun's remedy? Most GNU tools have names starting with "g" (e.g. grep is from Sun, ggrep is GNU).

There's also no /root directory. Superuser's home is /. That's because you're not supposed to login as root. If you really want to, you can create it and modify /etc/passwd.

See more details in this entry. The whole Ben Rockwood's blog is worth a read.

Init system - SMF

System startup, on the other hand, is an area that Sun's got right. Some Linux distros, lead by Ubuntu, are migrating to a new init system called upstart. Others still use SystemV scripts. And don't even get me started on Slackware. Solaris has already converted to a new model called SMF - Service Management Facility. It has all the features planned for upstart and much more:

parallel startup,
dependencies - e.g. service A will always start after service B
failed daemons are automatically restarted
ability to control a specific service can be delegated to a non-root user
snapshots of service configurations
debugging and logging features

Quick intro:
Enabling/disabling the service: svcadm [enable|disable] [service name]
Service status: svcs [service name]
List of services (both running and disabled): svcs -a

More details at Sun Bigadmin website

Shell

Another annoying weakness for me and praised historical compatibilty for others. Default shell on Solaris is the old-style Bourne shell with no command history, no tab completion, no nothing. First thing I do on Solaris is

usermod -s /usr/bin/bash root

You may find warnings on the web not to do it. Some people say it'd break scripts, others think /bin/sh is safer, because it's statically linked. Feel free to mock them:

scripts have a #!/bin/sh line and don't care about login shell,
Solaris 10 is smart enough to fall back to /bin/sh if bash doesn't work. And sh is not even static anymore:

Fedora goes towards KVM, away from Xen

2010-01-26T11:45:00.002+01:00

Fedora 13 (to be released in May 2010) will add some interesting features:

Kernel acceleration for KVM networking (off by default, considered unstable)
Stable PCI addreses, shared network interfaces
Enhanced VNC access to guests - tunnelling audio otput and serial console
Better authentication and encryption support in libvirt

Most of these only work with KVM. Another new tool will enable automatic migration from Xen to KVM. Xen dom0 support is planned "if it's supported by the mainstream kernel". Read: it won't be available.

While I welcome these additions to KVM, I think it's a pity that Xen support is gradually phased out from the main Linux distributions. What can you do if you have servers without hardware virtualization support, but otherwise completely fine?

Buy new servers and send the old ones to a growing heap of electronic junk in some 3rd world country.
Spend hours compiling and configuring the software yourself.
Use on a few distros that still supports Xen - if you can live with 2.6.18 kernel. I think that now we're only left with Scientific Linux and Debian.

References:
www.virtualization.info/2010/01/fedora-13-to-simplify-migration-from.html

Howto: two Firebirds on one Linux box

2010-01-25T13:46:00.001+01:00

Don't know if anyone ever needs such a crazy setup, but here it goes. I needed to run a software that requires Firebird 2.x. Nothing special, it's available in my distribution (Ubuntu). I also had to run another app that needs Firebird 1.5. That's where it started to get complicated:

- Firebird 1.5 is not available in Ubuntu - I can use vendor-provided binaries, but...

- They're only 32-bit, my box is amd-64

- There's no official support for running two versions of Firebird on one machine.

One solution would be to use a virtual machine. That way I would be compliant with a title of this blog. But I only needed these RDBMSes occasionally, maybe once a month, never both versions simultaneously. Also this machine is short on disk space. I didn't want to waste few GB on another copy of an operating system with all the necessary tools and libraries. After half an hour of meddling, I managed to get both versions on the same system.

I. Prerequisites

1. aptitude install ia32-libs
Required for Firebird 1.5, if you're on a 64-bit system and you haven't already installed 32-bit libraries

2. Install old version of libstdc++
Required for Firebird 2. Karmic doesn't have it anymore, get one from Jaunty and install with

sudo dpkg -i libstdc++5*deb

3. Create a file /etc/ld.so.conf.d/firebird.conf with the following contents:

/opt/firebird-1.5/lib
/opt/firebird-2/lib
/opt/firebird/lib

These directories don't exist yet, but they will be created soon.

II. Firebird 2

1. Get Firebird 2.1 from firebirdsql.org
You'll need the right version for your architecture (64 or 32 bit), superserver, compressed tarball.

2. Unpack it somewhere and run install.sh
If at any point you get an error starting like this:

ld: Unable to load shared library

stop the installation with Ctrl-C, run 'sudo ldconfig' and try installing again.

3. Firebird asks for a password for sysdba (admin) user.
Old versions used a default password 'masterkey' and some poorly written software relies on it. Don't use it if your application is exposed to the internet.

4. The software is now installed in /opt/firebird. Subsequent installation of Firebird 1.5 would overwrite it. We need to rename it:

sudo mv /opt/firebird /opt/firebird-2
sudo mv /etc/init.d/firebird /etc/init.d/firebird-2

Sometimes Firebird installer doesn't put the init.d script in the right place. You can find the script in /opt/firebird-2/misc/firebird.init.d.debian. Copy it to /etc/init.d and rename to firebird-2.

5. Use your favourite text editor on /etc/init.d/firebird-2 and replace the line

FIREBIRD=/opt/firebird

with

FIREBIRD=/opt/firebird-2

6. Run 'sudo ldconfig' again.

7. Test the init script.

/etc/init.d/firebird-2 stop

should stop the database, 'ps aux | grep firebird' would return nothing (or just the grep itself).

/etc/init.d/firebird-2 start

should start it, the command would return two processes, fbguard and fbserver

III. Firebird 1.5

1. Stop Firebird 2 if it's running at the moment.

2. Get Firebird 1.5 from firebirdsql.org
You'll need the version called Superserver for Linux (NPTL threading model), compressed tarball.

3. Unpack it somewhere and run install.sh
If at any point you get an error starting like this:

ld: Unable to load shared library

stop the installation with Ctrl-C, run 'sudo ldconfig' and try installing again.

4. Like with version 2, Firebird asks for a password for sysdba (admin) user.
An application bad enough to require such and old RDBMS probably also uses a default password. Mine does.

5. The software is now installed in /opt/firebird. Since we're not going to install any more versions, we could leave it here, but then an upgrade to Firebird 2 would overwrite it. Better rename it too. Steps are the same.

sudo mv /opt/firebird /opt/firebird-1.5
sudo mv /etc/init.d/firebird /etc/init.d/firebird-1.5

Replace /opt/firebird with /opt/firebird-1.5 in /etc/init.d/firebird-1.5

6. Run 'sudo ldconfig' and test the init script.

Xen.org overview slides

2010-01-21T11:51:00.002+01:00

I've joined the effort to provide Xen docs in other languages. Here's my tiny contribution - xen.org overview slides in Polish.

Xen.org Overview Polish

View more documents from Stephen Spector.

Free academic journals on HPC

2010-01-19T16:08:00.001+01:00

I've just found out these Springer journals offering free access to most or all of the content:

There's more, but these 3 were useful to me.

Experimental Xen and Ubuntu. Part 2: hypervisor

2009-12-04T11:52:00.001+01:00

This is second part of a two-part HOWTO. See Experimental Xen and Ubuntu. Part 1: kernel.

In this part we're going to compile the hypervisor and tools. You can install Ubuntu-provided binaries if you're happy with an older version. In that case, jump to Grub2 configuration. I'll again assume your compilation directory is ~/src.

Warning: You're about to install an experimental software. It might not work. The instructions might be outdated. In fact, I wrote this Howto over the course of 3 days and I had to update them. Use at your own risk.

0. Prerequisites

You'll need:

An hour or two

Some more development tools and libraries in addition to what you've already installed in part 1

aptitude install mercurial xorg-dev python-dev zlib1g-dev python-openssl python-pam libssl-dev uuid-dev bcc bison flex checkinstall

1. Get ACPICA

You'll need one more tool: Intel's ACPI Component Architecture compiler. It's not in the distribution. Download it from http://acpica.org/downloads/ to ~/src, then:

cd ~/src
tar -xvzf acpica-unix*.tar.gz
cd acpica*/compiler
make
sudo chown root.root iasl
sudo mv iasl /usr/local/bin

There's no "make install", so you need to use last two lines to put the binary in place.

2. Get the sources

Download latest Xen from repository into your compilation directory.

hg clone http://xenbits.xensource.com/xen-unstable.hg

To update the sources later, use:

cd ~/src/xen-unstable.hg
hg pull
hg update

3. Compile hypervisor and tools

cd xen-unstable.hg
make xen
make tools

Do not use "make all" or you'll also get another kernel and docs (which require TeX and take ages to compile).

4. Create packages

Now you're supposed to to type "make install". That would put the compiled files into right places, but without any traces in your package system and without easy way to uninstall. There's a "make uninstall" target, but not guaranteed to work. So I decided to make a primitive .deb with checkinstall. It won't have all the features of a real deb package: proper dependency information, pre/post install scripts etc. But it works.

sudo checkinstall make install-xen
sudo checkinstall make install-tools

Checkinstall will ask you a few questions. Set version number to 3.5 (anything numeric will do, but autodetected unstable.hg won't). Set the package name and "provides" to "xen-hypervisor" for the first package, "xen-utils" (not xen-tools!) for the second.

5. Install packages

cd ~/src/xen-unstable.hg
sudo dpkg -i xen-* system_1-2_all.deb

6. Update your Grub2 configuration

You'll need to boot Xen instead of Linux. Our primitive package won't modify Grub2 configuration. Create file /etc/grub.d/08_xen, make it executable and insert the following contents:

#!/bin/sh 

cat << EOF 
menuentry "Xen 3.5 / Ubuntu, 2.6.31.6-xen1" { 
    insmod ext2 
    set root=(hd0,1) 
    multiboot /boot/xen-3.5.gz 
    module /boot/vmlinuz-2.6.31.6-xen1 dummy=dummy dom0_mem=1024M root=/dev/sda1 ro 
    module /boot/initrd.img-2.6.31.6-xen1 
} 

EOF

Substitute root=(hd0,1) and root=/dev/sda1 with the right information (get one from /boot/grub/grub.cfg). Provide the right path for the Xen kernel you compiled with the previous Howto. Note the argument "dummy=dummy" for the Linux kernel - it's needed because some versions of Grub2 swallow the first argument. Command "insmod ext2" is also correct for ext3 and ext4.
You should also change /etc/default/grub and comment the lines starting with GRUB_HIDDEN_TIMEOUT. It'll make Grub2 display menu and wait for a few seconds. This way, it'll be easier to select another entry if the default won't boot. When you're ready, run:

update-grub

7. Run Xen daemon at boot

update-rc.d xend defaults
update-rc.d xendomains defaults

Troubleshooting

Unfortunately, chances of getting an unbootable system are high. My advice is to narrow down the problem. Try installing the Ubuntu-provided hypervisor or another kernel. Try updating and compiling again the kernel or Xen, new changes are commited every day. Search Xen mailing lists.

Extra Apt repositories in Ubuntu

2009-12-02T11:28:00.004+01:00

Ubuntu contains thousands of packages in standard repositories, but some useful software is not included, usually for licensing reason. It's convenient to install other apps the same way you use for Ubuntu - with Synaptic, aptitude or whatever your favourite package manager is. That's when extra repositories are useful.

Using extra repository requires two things:

- adding a repository URL to apt's configuration - by convention, each repository gets it's own file in /etc/apt/sources.list.d

- adding a package signing key, so APT won't complain about untrusted source.

You can use GUI or command line for that, I prefer the latter. See https://help.ubuntu.com/community/Repositories/Ubuntu for instructions on using GUI.

Be warned that unofficial repositories are not supported by Ubuntu and may contain lower quality software. Although the ones below usually don't. Here are the repos I add on my machines.

MediBuntu

The single most useful repo. It contains multimedia codecs that didn't make it to official system due to licensing. Other software includes Skype and Google Earth.


echo "deb http://packages.medibuntu.org/ karmic free non-free" | sudo tee -a /etc/apt/sources.list.d/medibuntu.list
sudo apt-get update && sudo apt-get install medibuntu-keyring

The last line will warn you unverified package.

GetDeb

Large collection of various packages. Highlights: Flock browser, Cinepaint movie editor, Ubuntu Tweak.


wget -q -O- http://archive.getdeb.net/getdeb-archive.key | sudo apt-key add -
echo "deb http://archive.getdeb.net/ubuntu karmic-getdeb apps" | sudo tee -a /etc/apt/sources.list.d/getdeb.list

Tip: if you installed Flock, you might want to use Firefox plugins (not extensions - plugins are Flash Player, Java, Adobe Reader etc.) in it. Just use this two lines:


cd /usr/share/flock/plugins
sudo ln -s /usr/lib/firefox-addons/plugins/* .

VirtualBox

My choice for desktop virtual machine.


wget -q -O- http://download.virtualbox.org/virtualbox/debian/sun_vbox.asc  | sudo apt-key add -
echo "deb http://download.virtualbox.org/virtualbox/debian karmic non-free" | sudo tee -a /etc/apt/sources.list.d/virtualbox.list

What's next?

After adding repositories, update your package list:


sudo apt-get update

Then you can view the new packages in Synaptic or aptitude. I always start with:


sudo aptitude install ubuntu-restricted-extras

It gives you codecs, Java plugin, Flash player and MS fonts, all with one command.

Experimental Xen and Ubuntu. Part 1: kernel

2009-12-01T09:44:00.006+01:00

Xen status in Ubuntu is strange. There are packages for hypervisor and utils, there are even some addon tools. Ubuntu runs as a Xen guest flawlessly. However, there's no kernel supporting dom0 (privileged domain, you can think of it as a host operating system) in Ubuntu, which makes hypervisor useless.

Many users resolved the problem by borrowing a Xen kernel from Debian. It's also easy to download binary Xen distribution with the kernel included. But official Xen kernel is still 2.6.18. It's as old as Windows Vista! I wouldn't want to use something that ancient. Luckily, Xen maintainers provide a patch for every new kernel release - see Xen Wiki on PVOPS kernel for details. Installation is relatively easy and you can even make a deb package.

While you're at it, you can also compile the newest, shiniest Xen hypervisor and get all the exciting features. It's not as straightforward as kernel installation, but it's not a rocket science either.

This two part HOWTO guides you through the installation of the latest Xen on Ubuntu. Part 1 deals with the kernel. Part 2 is about the hypervisor and tools. You can install Ubuntu-provided binaries if you're happy with an older version and skip to Grub2 configuration.

0. Prerequisites

You'll need:

an hour or two
some experience with kernel compilation
a way to restore your system if it won't boot
compiler, development libraries, git client:

sudo aptitude install kernel-package git-core libncurses5-dev

1. Create source directory

Prepare a directory for compilation, e.g. $HOME/src. DO NOT use /usr/src/linux, use non-system directory and don't compile as root. Note it has nothing to do with Xen, it's just the right way to compile kernel, although many people get it wrong. See Greg Kroah-Hartman's great book "Linux Kernel in a Nutshell" for details.

I'll assume you use ~/src. Substitute whatever you wish. You'll need about 1.5 GB of disk space.

2. Get the sources

cd ~/src
git clone git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git linux-2.6-xen

3. Configure the kernel - the important stuff

For convenience, I copied the standard Ubuntu kernel config to use as a base, it's in /boot/config-2.6.31-15-general. Put it in ~/src/linux-2.6-xen and rename to .config.

Go to your new kernel source directory and use "make menuconfig" or your preferred method of kernel configuration. You only really need one change:
Processor type and features ---> Paravirtualized guest support ---> Enable Xen privileged domain support

Another modification you might want is "Local version" from "General setup". Add something like "-xen1". This way if you need another try, you'll get a package with a different name.

4. Configure the kernel - the unimportant tweaks

While you're at it, you can also tweak some features for better performance. Some quick modifications:

Set the processor type to exact type you have instead of a generic one.
Disable all setting with debugging, profiling or probe in their name (especially from "Processor type and features" and "Kernel hacking").
Disable unusual hardware from "Processor type and features". That goes directly into the kernel, so reducing clutter here might help.
I don't bother with the drivers. They go into modules anyway and have no impact on your runtime performance, they onlymake your kernel compilation last longer. I'd rather have my computer spend extra 30 minutes compiling then have myself spend 30 minutes turning off drivers. You can however quickly disable whole groups, like Packet Radio or Video4Linux if you're sure you don't need them.
From "Security options", disable SELinux and Tomoyo if you don't use it.
Disable all KVM options. You can't use two hypervisors at the same time anyway.

5. Compile your kernel the Debian/Ubuntu way

make-kpkg --rootcmd fakeroot --initrd kernel_image

This might be a good time to go for lunch.

6) Install your package

dpkg -i [name of your kernel package]

Try rebooting. The kernel supports Xen dom0 and domU, but it runs on real hardware as well.

Blogged with the Flock Browser

Papers on CUDA

2009-11-26T13:09:00.001+01:00

Some new scientific papers by the GPU computing community. By the way, I highly recommend adding GPGPU feed to your RSS reader if you're interested in this subject. That's where I get most of my news.

PyCuda is an open-source framework written in Python for generating CUDA code. It isolates you from tedious details of CUDA. Most programmers would find Python much more comfortable to work with. There is a scientific paper on PyCuda: GPU Run-Time Code Generation for High-Performance Computing. Lots of tutorials for beginners in the net, just ask your search engine. I even found one in Polish.

CheCUDA is a checkpoint/restart tool for CUDA applications. Checkpointing is a known method in long-running scientific computations: periodically write the state to the disk and if your software crashes, system needs to be restarted or the job migrated to another machine, you don't have to start from the beginning. Until now, there was no checkpointing tool for GPU computing. See this paper by Hiroyuki Takizawa, Katuto Sato, Kazuhiko Komatsu, and Hiroaki Kobayashi for details.

Blogged with the Flock Browser

...flock together

2009-11-24T10:06:00.002+01:00

First experimental post from Flock.

Time to get more Web 2.0. I'm not the one who falls for the latest buzzword. But this one is 5 years old, so I might give it a chance.

Blogged with the Flock Browser

Data visualization 101

2009-11-20T12:16:00.010+01:00

For the last few weeks I've been reading on data visualization in my spare time. The concept is ubiquitous: it's used in business (reports and presentations), science and technology, newspapers, magazines, even daily life: look at your cellphone, it has a battery and signal indicator. I bet you can see at least ten examples of chart, map or timeline from the place where you seat (and that's without going to another website). The reason is obvious: we humans are much better in processing graphical then numerical data. There's high chance that someday you'll have to prepare a chart of some sort.

Yet visualization is often misunderstood and applied poorly. Take this spreadsheet chart:

The standard template has too much clutter. Notice the grey background reducing contrast. That makes heavy grid lines necessary, but they're yet another form of visual clutter. It also looks dull. Most people who want to make their graphs more interesting does it by applying 3D effects and gradients making it even less readable.

What's the right solution? If you need precision (e.g. in scientific paper), follow this simple guidelines:
- Reduce clutter. Remove backrounds, borders, as much grid as you can - imagine you're trying to save on ink.
- In bar graph, try spacing the bars and keeping them close, see what works best. There's no rule.
- If you only have one series of data, remove legend. If you have more, consider labeling directly on the graph.
- Colors should mean something. Use it to distinguish one series from the other or the outllier from the rest of samples. Or use color intensity to show value. Don't ever make every sample in a series a different color for no reason.
Surprisingly, it'll look better and definitely it'll make easier to compare values.

The example here is only about 10 seconds of clicking away from the default. Although it's far from perfect, it's already much better.

If you value aesthetics more than precision (e.g. for a newspaper or advertising), use anything but the standard template. Consider infographics: want to chart the real estate market? Use the drawing of the house for your graph. Or use maps, possibly with satellite imaging or in 3D isometric view. The further you go from the old, boring chart, the better it'll look. But don't overdo it or the readers won't grasp your idea. There's a similar problem in science - some forms of visualization work great for people with the right background (e.g. in statistics), but others won't understand them at all.

Want to know more?
www.juiceanalytics.com/writing/ - Great practical advice on designing graphs, presentation and dashboards.
www.visualcomplexity.com/vc/ - Lots of stunning examples.
www.edwardtufte.com/tufte/ - A renowned expert on data vizualition, inventor of sparkline, author of many articles and books.
Handbook of Data Visualization - More on the scientific side, but first chapters offer loads of general knowledge.
Visualizing data - Next one on my reading list (or more likely glancing through list). Judging by the table of contents, that should have been my starting point.

VirtualBox 3.1.0 adds live migration

2009-11-16T15:18:00.002+01:00

New VirtualBox, now in beta, adds live migration (and calls it teleportation for no apparent reason). This is standard feature for server hypervisors like Xen and KVM, but something new for desktop products. Correct me if I'm wrong, but I think VirtualBox is the first. This might not be that useful for workstation as it is for servers, but I can see some uses: collaboration on developing a system or moving the system from development to production.

Other important feature in this release is redesigned snapshot system. It is now possible to restore to any snapshot instead of only the last one. This, on the other hand, is really useful. I have a Windows guest and instead of reinstalling once every few months, I just revert to a last good state. Saves me hours on installing software, updating and configuring.

There's also improved hardware support, including 2D acceleration for Windows guests. Note it's a beta release with some known problems (ironically, in the area of snapshots). I've just upgraded my setup (Linux 64-bit hosts without VT, Windows XP guest) and it works fine, your mileage may vary.

Eucalyptus 1.6.1 available

2009-11-10T10:51:00.000+01:00

New release of Eucayptus got some interesting features. At last it's possible to manage a cloud consisting of multiple clusters. Support for dynamic DNS and monitoring tools Nagios and Ganglia was added. It also fixes numerous bugs and promises better stability. See changelog for more details. I didn't have time to upgrade my setup yet (I'd have to upgrade Debian first), but it sure looks interesting.

XenServer opensourced

2009-11-05T21:17:00.003+01:00

I've already written about Citrix opensourcing XenServer. The source code for XAPI is now released. Xen developers are working on providing better interoperability between both versions of Xen. Join if you can! See Developer's Guide and XAPI mailing list for details.

I look forward for the day when I can ditch Xen Center and use libvirt-based software to manage all virtualization platforms.

Ant for system administration

2009-11-01T17:32:00.004+01:00

Ant is a popular build tool, especially for Java development. What most people don't realize is it can be used to automate other tasks. Lately I started using it for system administration purposes. Don't get me wrong, I'm not saying it should replace good old fashioned scripts. On the contrary, I estimate that I use simple bash scripts for 90% of the tasks, real languages like Python or PHP for 9%, remaining 1% is everything else, including Ant. And even this jobs I could do with other tool. But if a 10-line buildfile does the job of 100-line Python script, I'll go with the simpler solution.

What sysadmin tasks can you do with Ant?

Ant has support for:
- File operations, like copy, delete, move, mkdir, chown, chmod. Useful as a part of a more complicated task. If you only need file operations, shell script is way easier to write.
- Archives: usual like tar, gzip, bzip2, zip and few others, including rpm.
- Network-related: sending email, remote execution with SSH and telnet, FTP and SCP client, HTTP client (with ant-contrib)
- Revision control systems: CVS built-in, many (if not all) others with a plugin

When I use Ant?

I found two tasks for which Ant excels. Both are often needed in modern environments - with virtualization and cloud platforms in place, one administrator often controls hundreds of systems.

Uploading files to remote systems

Ant supports FTP and SCP. Nothing special, except it's easy to send only modifled files. Obviously with ordinary script you can read the timestamps, store them somewhere and compare them yourself, but it's more tedious. Here's an FTP example:

<?xml version="1.0" encoding="UTF-8"?>
<project name="whatever" default="ftpexample">

<target  name="ftpexample">
<ftp server="www.example.com"
userid="admin"
password="YourVerySecretPassword"
passive="yes"
depends="yes"
verbose="yes"
remotedir="/wherever/I/want "
binary="yes">
<fileset dir="/source/directory" defaultexcludes="yes">
</fileset>
</ftp>
</target>

</project>

Let's go through the example, skipping obvious lines.
- depends="yes" - only send modifiles files. Yes, it's that simple.
- verbose="yes" - print names of the transfered files.
- defaultexcludes="yes" - exclude backups (filenames ending with ~), CVS and SVN directories and the like.

Running remote commands on multiple systems

Ant can run tasks in parallel and track dependencies between tasks (i.e. before running task2, make sure task1 was finished). Ant can run commands remotely with SSH or telnet. Combine the two and you get a simple way of writing a distributed glue code. As an example, we'll check numbers of messages in Postfix queue on several servers.

<?xml version="1.0" encoding="UTF-8"?>
<project name="remoteexample" default="all">

<target  name="checkservers">
<parallel>
<sshexec host="server1.example.com"
username="exampleuser"
trust="yes"
keyfile="${user.home}/.ssh/id_rsa"
outputproperty="out1"
command="find /var/spool/postfix/deferred  -type f -print | wc -l" />

<sshexec host="server2.example.com"
username="exampleuser"
trust="yes"
keyfile="${user.home}/.ssh/id_rsa"
outputproperty="out2"
command="find /var/spool/postfix/deferred  -type f -print | wc -l" />
</parallel>
</target>

<target name="all" depends="checkservers">
<echo message="${out1}" />
<echo message="${out2}" />
</target>
</project>

Two caveats:
- Ant has built-in support for SSH. If you run it on Linux, it will not follow your usual configuration in ~/.ssh. You need to provide all required parameters in the buildfile, like the path to keyfile in the example above.
- If you run the above example, you'll notice the output looks like this:

all:
[echo] find /var/spool/postfix/deferred -type f -print | wc -l : 2
[echo] find /var/spool/postfix/deferred -type f -print | wc -l : 0

The command you ran is a part of the outputproperty. It's a known bug of Ant, local exec doesn't behave that way. It was fixed lately, but the new build is not available yet. If you want to process the return value (e.g. sum the outputs), you either have to filter the command out or compile Ant from subversion. Tried both ways and filtering in the shell script is the easier one.

The downsides of Ant

Ant requires Java. Not a problem if you need it anyway, but JRE is a little too big to install just for one or two scripts.
Ant buildfile.xml has a completely different syntax than a script. You don't write a sequence of commands. Instead, you declare what you want to achieve and what it requires using simple XML syntax. Still, it only takes an hour or two to grasp the concept and start writing buildfiles.

Shrink your logs with 7zip

2009-10-28T21:36:00.005+01:00

A busy server generates gigabytes of logfiles. Law requires to store some of them for years. I was once hired as a consultant by a company operating several Linux servers. One of my task was to increase capacity of a log archive. I proposed to shrink logfiles instead and saved the company the cost of extra disk and a few hours of downtime.

Logs are very redundant, which means they can be compressed efficiently. Standard Linux way is to use logrotate, which compresses old logs with gzip. For example, my typical daily Squid access.log turns from 2 GB into mere 50 MB. Can we do even better?

With 7zip we can. This useful tool supports several archive formats: zip, arj, rar and it's own, extremaly efficient LZMA algorithm. The downside: it's also painfuly slow and much more CPU intensive. Quick comparison on a small 52 MB sample of logs, using maximum compression setting in 7zip:

	gzip	7zip
archive size:	4.5MB	2.5MB
compress time:	1.8s	38.4s
decompress time:	0.7s	0.9s

Over 20x slower! That explains why 7zip is not used for on-the-fly compression. But old logfiles are only written once and cutting the storage requirements almost in half is worth the hassle.

In the setup I got, every server rotated the logs daily and kept 7 days of gzipped files. Once a week, a cron job downloads them to a central log archive (not a separate machine, just some space on one of the servers), to a directory named by the date. Here's /etc/cron.weekly/log-archive:

#!/bin/sh

cd /home/log-archive
BASEDIR=`date '+%Y.%m.%d'`
mkdir $BASEDIR
mkdir $BASEDIR/server1
mkdir $BASEDIR/server2
# put more servers here and use real names

cd $BASEDIR/server1
rsync  -a -R -e ssh log@server1:'`find /var/log -name "*.gz"`' .
cd ..

cd server2
rsync  -a -R -e ssh log@server2:'`find /var/log -name "*.gz"`' .
cd ..
# more servers here...

[download script]

The script uses SSH key-based authentication for password-less login. User "log" on each server has read privileges to all logfiles. You can use root, but this way is safer.
Now, let's process the logs. All we need is a another simple script:

#!/bin/bash

# decompress gzipped files and compresses them again with 7z
# set to maximum compression
# recursively recompresses directory tree

function recompress_file()
{
filename=`basename "$1" '.gz'`
echo "    $filename"
nice -n 19 gunzip "$1"
nice -n 19 7z a -bd -mx=9 "$filename.7z" "$filename"  >/dev/null &amp;&amp; rm "$filename"
}

function recompress_dir()
{
cd "$1"
echo "in $1"
for i in *
do
ext=${i##*.}
if [ -d "$i" ]
then
recompress_dir "$i"
elif [ $ext = 'gz' ]
then
recompress_file "$i"
fi
done
cd ..
}

if [ ! -d "$1" ]
then
echo "Usage: $0 directory_to_recompress"
exit 1
fi

recompress_dir "$1"

[download script]

Let's see how it works. Function recompress_dir iterates over all entries in the specified directory. If a given entry is a subdirectory, it recursively calls itself on it. You've probably heard it's better to avoid recursion in bash, but directory traversal is an exception - it works correctly (provided your directory tree is sane; you can check recursion level if you want to be extra safe) and greatly simplifies the script. For all files with extension .gz (multiple extensions like apache.log.2.gz work fine, only the last one counts), it calls recompress_file. This function is even simpler: gunzip, 7zip, delete uncompressed file. I used quotes around arguments everywhere in case a filename containts spaces.

Parameters for 7z:

a - Add files to archive. Only one file in our case, but 7zip works like zip or rar, multiple files are possible.
-bd - Disable percentage indicator. We redirect output to /dev/null anyway.
-mx=9 - Sets compression level to maximum. Lower levels are faster, but we're here for maximum storage savings.

Compression algorithm is not set, so default LZMA will be used. I also set 7z to the lowest possible priority with "nice -n 19". I don't care how long the job takes, as long as it's less than a week when the next one starts. And it's better to leave CPU to other processes. Incidentally, 7zip supports parallel processing. With LZMA it uses 2 threads on an SMP system. My server had 4 CPU cores, which left half of the processing power to other jobs (like, serving websites and stuff).

Save this file as /usr/local/sbin/log-recompress. Than, to automatically process new logs, append one line to /etc/cron.weekly/log-archive:

/usr/local/sbin/log-recompress /home/log-archive/$BASEDIR

What about the old logs? I created several files like this:

cd /home/log-archive/ && log-recompress 2009.02.11 && log-recompress 2009.02.18 && log-recompress 2009.02.25

Saved them as log-job1, log-job2 etc. Next, I used at to run them for a few nights in the row:

at -f log-job1 23:00 24 Sep
at -f log-job2 23:00 25 Sep
at -f log-job3 23:00 26 Sep
...

With nice -n 19 I could probably run them any time, but just to be sure I've chosen low utilization hours. Each night the server processed about 6 GB of logs shrinking them by 2.5 GB. In the end I got enough disk space for about 2 years worth of logfiles.

EC2 news: MySQL service, High-Memory instances

2009-10-27T16:56:00.003+01:00

Several annoucements from Amazon

Since November 1st, all types of EC2 instances will be slightly cheaper. For example, small Linux VM dropped down from 10 to 8.5 cents per hour. Only the largest Windows instance is now above $1.

Amazon also introduced two new instance types called High-Memory:

Double Extra Large: 34.2 GB memory, 13 ECU (4 virtual cores with 3.25 ECU each), 850 GB storage, 64-bit platform
Quadruple Extra Large: 68.4 GB memory, 26 ECU (8 virtual cores with 3.25 ECU each), 1690 GB storage, 64-bit platform.

Pretty impressive. But they need a new naming scheme or in a few years we'll get "Enormously Extra 64x Huge".

A new service made it into a public beta: Relational Database Service. Basically, it's an EC2 instance with preconfigured MySQL database. Notable features include scalability and snapshoting. Until now, you had a choice of running your own MySQL on EC2 instance (with all administrative burden) or using Amazon SimpleDB, incompatible with most existing apps. RDS made it easier to quickly deploy a LAMP application. For long-term use, you still should probably configure your own database server. RDS instances are more expensive than ordinary VMs and you're also billed for data transfer.

XenServer and Xen again

2009-10-24T13:33:00.003+02:00

Citrix has just announced it's going to completely open source XenServer. Not only the hypervisor, which shares most code with normal Xen anyway, but also administrator's console.

That's good news, but I'm still not convinced XenServer is better than ordinary Xen for my use. What I particularly don't like is that it uses completely different API and administration tools. Xen has xm, xend and high-level tools built on top of them. XenServer has xe and XAPI. Neither is clearly better or worse than the other, the problem is they're different. Not only it means you need to learn two sets of commands if you want to manage both, it also means that XenServer won't work with Eucalyptus and the like.

Simple example - few days ago I spent way too much time figuring out how to connect to virtual machines's console in XenServer without using XenCenter. With xen, it's as simple as "xen console [vm number]". With XenServer, the procedure takes 3 steps and is never mentioned in the docs. Citrix want you to use XenCenter for everything, if you just physically sit at the server and don't have any Windows machine nearby - well, sucks to be you. Luckily, I was able to find answer in the forums:

1. List your virtual machines to get VM id:

xe vm-list

2. Match your VM id with domain id (why XenServer uses two types of ids, one of them painfully long, remains a mystery):

list_domains

3. Run the console, you have to use full path:

/usr/lib/xen/bin/xenconsole [dom_id]

Run Windows on Linux with VirtualBox

2009-10-21T12:16:00.008+02:00

In my previous comparison of virtualization platforms I wrote I prefered VMware Player for home use. Not any more. Few weeks ago I installed the newest VirtualBox on my Ubuntu laptop. Turned out it now provides the same ease of use and performance, stability seems even a bit better (although I didn't have much problems with VMware either). Unlike VMware Player, it has a built in support for creating VMs. You can also save snapshots and restore the VM to previous state - very useful with Windows VMs and test/development environments. What's more, VirtualBox is open source. I'm quite pragmatic, given a choice of a better closed source product and inferior but free alternative, I'd go for closed source. But if all other metrics are equal, open source is a nice bonus.

Installing VirtualBox on Ubuntu takes only a few minutes. Hardware virtualization support is not required, but if it's available, your VM will run much faster.

Step 0. Uninstall VMware (skip this step if you don't have it)
I didn't need VMware anymore, so I removed it with:

sudo vmware-uninstall

1. Add apt repository for Virtual Box.
That way, you'll get automatic upgrades in future.

wget -q http://download.virtualbox.org/virtualbox/debian/sun_vbox.asc -O- | sudo apt-key add -
sudo echo 'deb http://download.virtualbox.org/virtualbox/debian jaunty non-free' > /etc/apt/sources.list.d/virtualbox.list

2. Install packages

sudo aptitude update
sudo aptitude install virtualbox-3.0 dkms

That will also install a C compiler and a whole bunch of packages related to kernel development, unless you already had them. They are necessary to compile Virtual Box kernel modules. Don't worry, it happens automatically.

3. Add yourself to vboxusers group
Users who want to run Virtual Box have to be in vboxusers group. Type the command:

sudo adduser igor vboxusers

replacing "igor" with your username.

4. Run VirtualBox
You can either type VirtualBox in your terminal or find it in the system menu.

5. Click "New"
Set some options. Give your machine large RAM, but leave enough for your host system. Another important option is storage type. You have a choice between:
- Fixed size: if you choose 10 GB, it will occupy 10 GB even if most of the disk drive is empty.
- Dynamically expanding: will save disk space, but make your VM slower.
I only use this VM occasionally, so I went with dynamic.

6. Configure sound
For Ubuntu host, the best choice is PulseAudio.

7. Install Windows
Put your Windows CD in the drive. Run your VM, then from Devices menu, select Mount CD/DVD-ROM / Host Drive. Another option is to use .iso file (CD image).

8. Activate Windows
You do have a genuine copy, right?

9. Install Guest Additions
VirtualBox comes with a set of drivers and utilities that make your VM run much faster. They also provide some convenience features:
- your mouse automatically switches between host and guest system,
- the clipboard is shared,
- full screen and seamless mode.
I highly recommend installing them. You'll get a warning the driver is not signed by Microsoft, but you can safely ignore it.

Zemanta

2009-10-21T12:08:00.005+02:00

Image by chucks via Flickr

I've just installed Zemanta: a Firefox extensions that aids blog posting by automatically finding relevant contents: images, related articles etc. I'll give it a try for a few days and see if it's useful. I suppose with blog like mine - highly technical and specialized - it's relatively easy to decide what's relevant. For the sake of experiment, I'll include some links from Zemanta in that very post.

Hosted Xen

2009-10-07T16:54:00.003+02:00

Xen usual mode of work is Type 1 or abre metal hypervisor - your computer boots into Xen, which then runs another operating system (usually Linux) into a first, specially privileged virtual machine. This is typical server way, it gives better security and performance, but it's harder to setup.

Now you can also run an experimental version called Hosted Xen (HXEN for short). Currently it runs on Windows or MacOS X. You can download source and binary version from the Xen Projects website and read the announcement on Xen Blog.

Why would you want a hosted Xen? One reason that comes to mind is using the same VM image and similar setup on your workstation for testing and on the server for production.

Nexus: a GPU development environment from Nvidia

2009-10-05T19:39:00.003+02:00

Another interesting announcement from Nvidia: a release of Nexus, integrated CPU/GPU software development environment for Visual Studio. Nexus is aimed both on game developers and programmers interested in general-purpose GPU computing. It contains several tools that help to analyze performance of both CPU and GPU programs:
- source code debugger,
- Nexus Analyzer, a tool that shows GPU and CPU events on a single timeline,
- Nexus Graphic Inspector, tool that works on frames rendered with Direct3D.

GUI should be familiar to all Visual Studio programmers. I, not being one of them, won't have much use of Nexus, but I certainly welcome a new addidtion to a GPU computing toolbox. Nexus public beta will be available for download in a few days.

Open source Xen vs. XenServer

2009-09-29T17:23:00.007+02:00

So, I already convinced you to choose Xen hypervisor for your servers. Great. Now you're wondering whether to choose a community-supported Xen or commercial (but free of charge) XenServer? The usual answer is: it depends.

Similarities

Both projects share most of the code. Xen is developed as an open source project, but Citrix - the company behind the XenServer - is the main contributor. Once a year or so, the current version of Xen is bundled with some additional software, tested thouroghly and branded as XenServer. You should get the same performance from both versions. With open source Xen, you can run the bleeding-edge development version with more features, but doing so on a production server is a really bad idea. XenServer is a complete product, open source Xen is more like a component - only coupled with your Linux distribution and some additional software it will provide a full solution. In short - more work to set it up properly, but more flexibility.

XenServer pros

The best point of XenServer is extremally easy setup. You can have your software installed and virtual machines running in literally 10 minutes. Advanced features like live migration don't take much longer. All you have to do is:
- download 2 CDs - base installer and Linux guest support.
- install them, following on-screen instructions, on as many servers as you like - the installer looks just like the one in RedHat
- install XenCenter management console on a Windows(!) workstation,
and you're ready to go. You can put all your servers in one group and manage them centrally. Live migration only requires a shared network storage. Spawning a new VM is just a few clicks, as is snapshoting, suspending/restoring and other typical administration tasks.

XenServer cons

Starting with minor issues: XenCenter management console only works on Windows. Many sysadmins prefer to work with Linux, but there's usually a Windows machine somewhere, so it's not a big deal. Upgrade process is awkward - you can only do it from XenCenter. I wish Citrix used something like Yum or apt-get, so it could be initiatied from the server or in many cases completely automated. On the plus side, it's easy to perform a rolling upgrade: migrate your VMs out of the box you're upgrading to other machines, upgrade it, restart, move machines back, empty another box, repeat until done. XenServer doesn't use xm tool familar to many sysadmins. It uses xe instead as a CLI interface. It has more features, but a different syntax - yet another thing to learn.

Middle-weight problem: XenServer only works on 64-bit CPUs. New servers are usually 64-bit anyway, but many companies have few year old servers which are completely fine, except they're 32-bit. Mine has.

The biggest drawback of XenServer is it's closed nature. Open source Xen management tools don't work with Xen Server. That includes provisioning apps like xen-tools, all monitoring tools based on RedHat's libvirt and goes up to private cloud systems. XenServer offers it's own management tools: command-line xe, text-mode menu system xsconsole and GUI XenCenter (pictured above). If you're not satisfied with them, tough luck. Many advanced or even not-so-advanced features like e-mail alerts are possible with community Xen, but blocked in XenServer unless you buy Citrix Essentials license. There are also some 3rd party products that would give you more features. Neither them nor Citrix Essentials offer as much as Eucalyptus or Nimbus. Of course, also neither of them requires 2 weeks to learn and setup properly.

What to choose?

If you only want to run a few servers for typical business app (none of private cloud fancy stuff) and all of them are 64-bit, XenServers is a great choice. It will save you many days of work. Otherwise, you have no choice but to run community Xen. Choose wisely, because it's not easy to switch. Converting VM image between these versions of Xen is complicated manual process. In fact, it's easier to migrate a physical server.

More info on XenServer:
Citrix website
IBM Redbook article

Graphic cards will finally be useful

2009-09-29T17:10:00.007+02:00

Nvidia just released first public version of OpenCL drivers for Windows, Linux and MacOS. OpenCL is a standard for writing parallel software that can run on GPUs, ordinary x86 CPUs, Cell CPUs and specialized hardware like DSPs.

In addition, Nvidia provides OpenCL Visual Profiler. This tool allows to see what functions takes the most time to execute and find bottlenecks. Linux version of Visual Profiler is certified on RHEL only, but I ran it succesfully on Debian. Read release notes before installing, there's one tricky step involving LD_LIBRARY_PATH. Visual Profiler only runs on Nvidia hardware at the moment. If you ever tried to find why your GPU program doesn't run as fast as it should, you know how hard it was without that tool.

Update: I've played a bit with Visual Profiler and it looks really useful. I particularly like one feature: timing how much it takes to copy data between GPU memory and system memory. It's often the biggest bottleneck.

That is, in my opinion, the biggest breakthtrough in the short history of GPU computing. Just one month ago, when I first wrote about this subject, I said that lack of high-lever tools and incompatible implementations are the biggest barriers of adoption. OpenCL specification was already available since late 2008. ATI was the first major vendor who adopted it and dropped it's own technology. Few days after my post later Apple released MacOS X Snow Leopard, the first operating system to support OpenCL. Nvidia, the biggest player, was falling behind. Today it still supports CUDA library, but adds OpenCL support as well. This technology will be implemented in Windows 7 and is already available for Linux.

In short, we can now use the same language on:
- 3 most important operating systems,
- GPUs by 2 biggest vendors, in addition to x86 and Cell Broadband Engine
The tool support is light years from what we have for ordinary CPUs, but it's getting closer every month. For maximum performance, you still have to tweak your code to your hardware, but the amount of low-level worked was significantly reduced. I think we're getting very close to a GPU-accelerated Photoshop or video encoder.

More info:
OpenCL overview (PDF)

Downloads (require free registration):
- Nvidia
- ATI/AMD

Both contain compiler, libraries, documentation and sample code. Only Nvidia provides visual profiler.

Eucalyptus. 6: Troubleshooting

2009-09-28T18:08:00.006+02:00

Eucalyptus: troubleshooting

1. Are the nodes present?
Command: euca-describe-availibility-zones verbose
(on any client) will show types of VMs and registered nodes:

AVAILABILITYZONE proba2 UP localhost
AVAILABILITYZONE |- vm types free / max cpu ram disk
AVAILABILITYZONE |- m1.small 0002 / 0002 1 128 2
AVAILABILITYZONE |- c1.medium 0002 / 0002 1 256 5
AVAILABILITYZONE |- m1.large 0000 / 0000 2 512 10
AVAILABILITYZONE |- m1.xlarge 0000 / 0000 2 1024 20
AVAILABILITYZONE |- c1.xlarge 0000 / 0000 4 2048 20
AVAILABILITYZONE |- node1 certs[cc=true,nc=true] @ Thu Aug 20 13:31:43 CEST 2009
AVAILABILITYZONE |- node2 certs[cc=true,nc=true] @ Thu Aug 20 13:31:43 CEST 2009
...

If you can't see a node, login to your Cloud Controller, deregister the node and register it again:

euca_conf -delnode node_ip
euca_conf -addnode node_ip

2. Are the virtual machines running?

euca-describe instances

Machines that are just starting will have the status "pending". If Eucalyptus fails to run them, they show up as "terminated" for a few seconds. Admin will see the machines belonging to all users.

RESERVATION r-53C80A43 admin default
INSTANCE i-41630842 emi-9F770FB8 0.0.0.0 0.0.0.0 terminated mykey 0 m1.small 2009-08-20T11:25:57+0000 proba2 eki-90581388 monitoring-false
INSTANCE i-467607D0 emi-9F770FB8 0.0.0.0 0.0.0.0 terminated mykey 1 m1.small 2009-08-20T11:25:57+0000 proba2 eki-90581388

3. Log files

They're in /var/log/eucalyptus. Careful, they're very verbose. Grep is your friend.

4. 32 or 64 bits?

Eucalyptus doesn't check if the image you're uploading is 32 or 64 bit. You need to provide the right information yourself. Uploading a 64-bit image to a 32-bit only system is a common mistake.

5. Double-check IDs

Or make it triple. It's easy to make a mistake. You'll probably want to write scripts soon.

6. Can you run your VM manually?

Check with ordinary Xen installation.

7. Check your /etc/fstab

Eucalyptus uses /dev/sda1 as a root device, sda2 for swap and sda3 for "ephemeral" (empty filesystem you can use for temporary files). If you converted your existing Xen image, you might have something else, like /dev/xvda1.

8. Error message: certificate has expired

This is usually caused by invalid timezone. Eucalyptus only works correctly if the controllers use UTC. Try this command: ln -sf /usr/share/zoneinfo/UTC /etc/localtime

Howto: downgrade Ubuntu Karmic to Jaunty

2009-09-26T18:54:00.006+02:00

Today, something unrelated to HPC and virtualization. Few weeks ago I upgraded my system to Ubuntu Karmic (9.10) alpha. I thought, 2 months before release date, it would be fairly stable. Well, I was wrong and some days later I had to downgrade it back to Jaunty (9.03). Upgrading is usually well-tested and quite simple process, but there's no official, one true way to downgrade an Ubuntu system. Turned out it required some manual tweaking. I hope this howto will help others with similar problems.

I based it on this 4 year old post about debian downgrade and added some stuff specific to Ubuntu Karmic and Jaunty.

Caution

Using this guide, you may completely screw up your system. Especially all the commands with "--force" option are dangerous. I ended up with a usable system because I have years of experience with Linux, so I knew what I was doing. Or I was lucky. Either way, it worked for me, no guarantee it will work for you.

Prerequisites

You will need:
- about an hour (more if you aren't lucky to sit on a very fast connection),
- about 3GB free disk space,
- root account unlocked (if you haven't done it before, do "passwd root" now),
- a way to connect to the internet without Network Manager if you find yourself without X. I'll show how to manually connect through wired LAN later. Alternatively, you should be able to use Ubuntu CD, but I didn't try this way.

First steps

1. Edit your /etc/apt/sources.list file and everything in /etc/apt/sources.list.d
Replace every occurence of "karmic" with "jaunty".

2. Edit /etc/apt/preferences, create the file if it doesn't exist. Enter the following lines:

Package: *
Pin: release a=jaunty
Pin-Priority: 1001

Package: *
Pin: release a=karmic
Pin-Priority: 600

3. Type "sudo su -" to drop into a root shell, so you won't have to use sudo on every command.

4. Run "apt-get update"

The Big Cycle

Now, you will need to repeat it a few times

5. Run: apt-get dist-upgrade. This will inform you it has to downgrade/install/remove hundreds of packages. However, something WILL break in the middle of the process.

6. When the upgrade is interrupted, try: apt-get -f install.

7. If it didn't help, see what package is causing the problem and why it can't install properly. Usually that's because the package tries to overwrite the file also owned by another package. This situation is handled properly during upgrades, but sometimes apt fails to resolve it when downgrading. However, it will spill the name of the package. At that point, you should probably install it by hand, using dpkg and full path to the package, eg.:
dpkg -i --force-all /var/cache/apt/archives/apturl_0.3.3ubuntu1_all.deb
Note that it will only downgrade this single package, you have to run "apt-get -f install" to resolve dependencies next.

8. Other possible problems:
- One package complained about the lack of "mktemp". I installed it using "apt-get install mktemp".
- I had issues with config files from one package. I completely uninstalled it with "dpkg --purge" and installed it back with "apt-get install", reverting it to the right version and default configuration.
- One package was installed incorrectly event though apt thought it was fine. I reinstalled it with "apt-get --reinstall install libkrb53". Luckily it almost never happen, but if it does, it's damn hard to spot. Text mode programs will complain to the terminal about a missing library, but GUI software or daemons will just write it to a logfile somewhere.

9. When "apt-get -f install" doesn't show any unresolved dependencies or unconfigured packages, do "apt-get autoremove". This will remove all the libraries that were required by a Karmic version of some package, but now aren't needed anymore when you downgraded them to Jaunty.

10. Go back to the the beginning of the cycle. Repeat until all packages are downgraded, then reboot.

Fixing Jaunty

11. Karmic uses completely new init system. As usual, upgrade from Jaunty works fine, downgrade doesn't. After reboot, I found that some of my daemons weren't running. Turned out some scripts in /etc/init.d were replaced with the symlinks, now leading to nowhere. Proper scripts were installed with ".dpkg-new" appended to their name. Fix it like this:

cd /etc/init.d
mv cron.dpkg-new cron

Repeat for all files ending with dpkg-new

12. I also found my system with unusable X. Mouse and keyboard were locked on the login screen. After some googling, I found that this issue struck more people, not necessarily during downgrade. Solution:
- reboot in single user mode (or you can do that before rebooting in Jaunty, just to be sure)
- I normally use WLAN and Network Manager, without X i resorted to wired network that I brought up with "ifconfig eth0 up; dhclient eth0". If eth0 doesn't work for you, try eth1, eth2 etc.
- Remove some packages with: dpkg --purge --force-all hal udev xserver-xorg dbus
- Install them back with: apt-get -f install
Probably I didn't have to reinstall all 4 packages, but I didn't bother checking which one it was.

Possible improvements

If you want to keep some packages from karmic, you could enter them in /etc/apt/preferences like this:

Package: zim
Pin: release a=karmic
Pin-Priority: 1001

Eucalyptus. 5: Creating and running VMs

2009-09-22T13:05:00.010+02:00

What you need to know

• VM images are stored in Walrus - a storage subsystem of Eucalyptus cloud controller.
• Walrus is divided into several areas called buckets.
• Objects inside Walrus are identified by a bucket name and object name (think: path and filename).
• To make your VM image available in Eucalyptus, you need to register it. If you want to delete it, you should deregister it first.
• That means that storing object is a 3 step process: create, upload, register.
• A registered object gets a unique identifier. Eucalyptus (non-Walrus) commands use this identifiers instead of filenames.
• There are 3 types of objects: kernel, root filesystem and initrd (the last one is optional).

Type	Example ID	Where stored
kernel	eki-C5881420	kernel-bucket
initial ramdisk	eri-C5985326	initrd-bucket
root filesystem	emi-C4581434	image-bucket

• You can assign kernel and initrd to a filesystem image, otherwise Eucalyptus will use a default one. You can also choose another kernel/ramdisk at runtime. Multiple filesystems can use the same kernel and initrd.
• Each image can be stored as either 64-bit (default) or 32-bit (forced with -r i386 when creating).
• At runtime, in addition to root filesystem, each VM will also get a swap and ephemeral partition. The latter is an empty filesystem to store temporary files.

Adding a new VM image

I assume the client is configured according to "Eucalyptus client configuration" post. I also assume you already created a Xen filesystem image, for example with xen-tools or by modyfing a pre-existing image. It's a good practice to check the image using local Xen installation first.

Adding kernel image

Example:

euca-bundle-image -i /boot/vmlinuz-2.6.26-2-xen-686 --kernel true -r i386
euca-upload-bundle -b kernel-bucket -m /tmp/vmlinuz-2.6.26-2-686.manifest.xml
euca-register kernel-bucket/vmlinuz-2.6.26-2-686.manifest.xml

Description:

• -r i386 switch means we create a 32-bit image. Omit it for 64-bit.
• -i in euca-bundle-image is followed by a path to a kernel file. We use the same filename (without directories!) in the next steps.
• euca-bundle-image stores temporary files in /tmp - watch out for free space! You can change location with -d /some/where/else
• euca-register will return Eucalyptus Kernel ID (eki-xxxxx). We'll need it later to assign to filesystem.

Adding initrd image

Example:

euca-bundle-image -i /boot/initrd.img-2.6.26-2-xen-686.bak --ramdisk true -r i386
euca-upload-bundle -b initrd-bucket -m /tmp/initrd.img-2.6.26-2-xen-686.bak.manifest.xml
euca-register initrd-bucket/initrd.img-2.6.26-2-xen-686.bak.manifest.xml

Description:

• The commands are very similar to what we used before. Differences: –ramdisk instead of –kernel in bundle-image, bucket name in the next two.
• euca-register will return Eucalyptus Ramdisk ID (eri-xxxxx). We'll need it later to assign to filesystem.

Adding filesystem image

Example:

euca-bundle-image -i vm-proba.img -d ~/tmp --kernel eki-6D2D12A0 --ramdisk eri-632C1665 -r i386
euca-upload-bundle -b image-bucket -m ~/tmp/vm-proba.img.manifest.xml
euca-register image-bucket/vm-proba.img.manifest.xml

Description:

• vm-proba.img is a filesystem image I created with xen-tools.
• kernel and ramdisk parameters of bundle-image are IDs we get from euca-register in the previous steps.
• euca-bundle-image will store a few GBs of temporary files. I don't have that much space in /tmp, so I pointed it to ~/tmp with -d switch.
• euca-register will return Eucalyptus Machine Identifier (emi-xxxxx). We will use it to run a VM.

Running virtual machines

A running machine is called an instance in Eucalyptus lingo.

Starting

euca-run-instances -k keyfile -n number emi-identifier

Example:

euca-run-instances -k mykey -n 2 emi-9F0C0FAC

Number of instances defaults to 1 if ommited.

Checking

euca-describe-instances

Example output:

RESERVATION r-3F820853 admin default
INSTANCE i-4E22077A emi-9EFE0FB2 0.0.0.0 0.0.0.0 running mykey 0 m1.small 2009-08-26T14:33:22.292Z proba2 eki-C5881420 eri-6384166E
RESERVATION r-4A7B0A09 admin default
INSTANCE i-43AD0888 emi-9EFE0FB2 0.0.0.0 0.0.0.0 pending mykey 0 m1.small 2009-08-26T14:36:09.046Z proba2 eki-C5881420 eri-6384166E

running is obviously a working instance, pending - will run in a moment, Eucalyptus is moving files around, shutting-down - in the process of halting the system, terminated - that shows for a few seconds after shutdown.

Stopping

euca-terminate-instances instance-id

Example:

euca-terminate-instances i-30FE0671

System will be cleanly halted, running all the usual shutdown scripts first.

Listing and removing images in Walrus

euca-describe-images
euca-deregister id
euca-delete-bundle -b bucket_name -p file_name

Eucalyptus. 4: client configuration

2009-09-21T14:09:00.004+02:00

I assume you want to use Eucalyptus tools. You can also use Amazon EC2 API and AMI tools instead. Installation instruction is for Debian/x86, everything else would work on every Linux distribution.

1. Unpack the tools somewhere and enter the directory:

tar -xvzf euca2ools-1.0-lenny-i386.tar.gz
cd euca2ools-1.0-lenny-i386

2. Add the directory to APT and install the tools:

echo deb file://${PWD} ./ >> /etc/apt/sources.list
apt-get update
apt-get install euca2ools

3. You need to install client certificates. I'll use admin credentials in the example (they allow you to interact with other users' VMs), you can also install regular user's certificates the same way. Fire up your web browser and enter URL: https://ip.of.cloud.controller:8443/ Note: HTTPS, not HTTP.

4. Login with your username and password, click Credentials and Download Certificate. Download file euca2-admin-x509.zip to your home direcory.

5. Issue commands:

cd
mkdir .euca
mv euca2-admin-x509.zip .euca
cd .euca
unzip euca2-admin-x509.zip
. eucarc
echo '. ~/.euca/eucarc' >> ~/.bashrc

6. Client tools should now be able to connect to the controller. Check with euca-describe-availability-zones verbose. It should display something similar to this:

AVAILABILITYZONE proba2 UP localhost
AVAILABILITYZONE |- vm types free / max cpu ram disk
AVAILABILITYZONE |- m1.small 0001 / 0002 1 128 2
AVAILABILITYZONE |- c1.medium 0001 / 0002 1 256 5
AVAILABILITYZONE |- m1.large 0000 / 0000 2 512 10
AVAILABILITYZONE |- m1.xlarge 0000 / 0000 2 1024 20
AVAILABILITYZONE |- c1.xlarge 0000 / 0000 4 2048 20
AVAILABILITYZONE |- node1 certs[cc=true,nc=true] @ Wed Aug 26 13:21:28 UTC 2009
AVAILABILITYZONE |- node2 certs[cc=true,nc=true] @ Wed Aug 26 13:21:28 UTC 2009

7. Just one step remaining: you need a keypair to encrypt your communication with the controller. Type the command, using any name you like in place of "mykey":

euca-add-keypair mykey > mykey.private

Eucalyptus. 3: Initial configuration

2009-09-21T14:00:00.007+02:00

1. Login to you Cloud Controller box. Register Cluster Controller with Cloud Controller using:

euca_conf -addcluster clustername localhost

clustername is whatever name you choose.

2. You need to be able to login from Cluster Controller to Node Controller without passwords, using username eucalyptus on both sides. Eucalyptus shoud set it up automatically, but in my experience it sometimes failed to do that. It's better to send SSH keys manualy, they're already generated. On your Cluster Controller, enter commands:

su - eucalyptus
scp .ssh/id_rsa.pub root@[ip of the node]

Repeat the last command for all the nodes.

3. Login to all Node Controllers and issue commands:

mkdir -p ~eucalyptus/.ssh
cat id_rsa.pub >> ~eucalyptus/.ssh/authorized_keys

4. Test your setup - login to Cluster Controller and type:

su - eucalyptus
ssh [ip of the node]

If you can login to node that way without being asked for a password, everything's fine.

5. On your cloud controller:

euca_conf -addnode [ip of the node]

Repeat for all the nodes.

6. Run a web browser and enter URL: https://ip.of.cloud.controller:8443/ Note: HTTPS, not HTTP.

7. Your username and password is "admin", you need to change it after first login.

Eucalyptus. 2: Server installation

2009-09-16T15:36:00.003+02:00

Installing Eucalyptus (using Xen as a hypervisor) on Debian Lenny

Howto based on open.eucalyptus.com/wiki/EucalyptusInstallationDebianLenny_v1.5.2

Prerequisites
1. All Eucalyptus controllers use loop devices extensively. By default, Linux limits the number of loop devices to 8. To increase it, edit /etc/modprobe.d/whatever_file_ending_with.conf and add line:

options loop max_loop=64

2. All controllers need to have the time synchronized. Make sure you use Network Time Protocol:

aptitude install ntp

3. Some client tools fail to connect if nodes use any timezone other then UTC (note: it's not enough to set your hardware clock to UTC). Run a command:

ln -sf /usr/share/zoneinfo/UTC /etc/localtime

Node Controllers

1. Install Xen:

aptitude install linux-image-xen-686 xen-hypervisor-3.2-1-i386

2. Check Xen configuration. We'll start with the basic network setup (bridging), you can switch to Eucalyptus-managed mode later. Make sure /etc/xen/xend-config.sxp contains the following lines:

(xend-http-server yes)
(xend-unix-server yes)
(xend-unix-path /var/lib/xend/xend-socket)
(xend-address localhost)
(network-script network-bridge)
(vif-script vif-bridge)
(dom0-min-mem 196)
(dom0-cpus 0)
(vncpasswd '')

3. Restart your machine, make sure Xen is running (xm list show Domain 0).

4. Unpack eucalyptus-1.5.2-lenny-i386.tar.gz somewhere and enter the directory:

tar -xvzf eucalyptus-1.5.2-lenny-i386.tar.gz
cd eucalyptus-1.5.2-lenny-i386

5. Add this directory to APT by running:

echo deb file://${PWD} ./ >> /etc/apt/sources.list
apt-get update

6. Eucalyptus docs suggest to run aptitude install eucalyptus-nc, but that would install other virtualization methods we won't need. My solution is the following:
• run aptitude
• press "/" and type "eucalyptus"
• navigate to eucalyptus-nc, select it with "+"
• press "g", aptitude will show list of suggested packages to install,
• deselect qemu and kvm by pressing "-" on them
• press "g" again
7. Aptitude will warn you that packages don't come from the trusted source. Ignore the warning.

Cloud and Cluster Controller

1. Unpack eucalyptus-1.5.2-lenny-i386.tar.gz somewhere and enter the directory:

tar -xvzf eucalyptus-1.5.2-lenny-i386.tar.gz
cd eucalyptus-1.5.2-lenny-i386

2. Add this directory to APT by running:

echo deb file://${PWD} ./ >> /etc/apt/sources.list
apt-get update

3. Install Java. For some strange reasons, Eucalyptus recommends having 2 versions: OpenJDK and Sun's JRE. You also need to install CA certificates unless you already have them:

apt-get install openjdk-6-jdk ca-certificates sun-java6-jre

4. There's a small bug with SSL certificates in Java package, easy to fix with this simple commands:

mkdir -p /etc/ssl/certs/java/
ln -sf /etc/java-6-sun/security/cacerts /etc/ssl/certs/java/cacerts

5. Eucalyptus docs suggest to run aptitude install eucalyptus-cc eucalyptus-cloud, but that would install a bunch of GCJ packages we won't need and that may even screw up your setup (haven't checked if they do). My solution is the following:
• run aptitude
• press "/" and type "eucalyptus"
• navigate to eucalyptus-cc, select it with "+", same with eucalyptus-cloud
• press "g", aptitude will show list of suggested packages to install,
• deselect all packages with "gcj" in their name by pressing "-" on them
• press "g" again
6. Aptitude will warn you that packages don't come from the trusted source. Ignore the warning.

GPU computing re-revisited

2009-09-16T15:32:00.002+02:00

There was a conference on parallel processing and applied mathematics in Wroclaw (where I live), GPU computing was one of the most prominent topics. The presentations in PDF format are available at gpgpu.org/ppam2009

Another interesting place is Claudio Albanese's website www.level3finance.com. As the domain suggests, dr Albanese uses GPUs for financial computations, but many of his articles apply to other areas as well.

If you know the open-source statistic software R, you can rapidly get started with GPU computing. A library provides GPU-accelerated equivalents to the most common R functions. The tests suggests 50-100x speedups are possible, check for yourself: brainarray.mbni.med.umich.edu/brainarray/rgpgpu/

Finally, a company called Hoopoe (www.hoopoe-cloud.com) provides several interesting solutions. One of them is GPU cloud - you can run your CUDA (nVidia) or OpenCL (ATI) programs on their hardware. For a limited time, the service is free. They're also the authors of very useful libraries - CUDA bindings for Java and .NET and OpenCL bindings for .NET. All of these allow you to write GPU-accelerated programs more easily. As an added bonus, your software will work on Windows, Linux and MacOS/X (using Mono instead of .NET) without changes.

Eucalyptus. 1: Introduction

2009-09-15T19:01:00.002+02:00

For a few months I've been working with Eucalyptus. I found it to be very promising, yet still somewhat incomplete platform. I decided to share my experiences. This is and introductory post, others will be howtos for specific tasks.

Eucalyptus is an open-source cloud computing platform designed to closely resemble (from the user's point of view) Amazon EC2. Standard Amazon tools can work with Eucalyptus and vice versa. This allows for very interesting setups: hybrid systems using both local and public VMs.

Eucalyptus started as a university project in 2008. In the beginning of 2009, it got commercial support. As you can see, the project is in the early phase. And yet there already some large scale deployments, with most famous being Ubuntu Public Cloud. Other noteworthy user of Eucalyptus is NASA. The project progresses at a rapid rate, in a few weeks we can expect a new version with more features.

Pros:
• Mostly (not 100%) compatible with EC2.
• Works with Xen and KVM, can use hardware virtualization support but doesn't require it. Note: I will only cover Xen setup in the howtos.
• Multi-user setup. Users can create their own VMs without administrator's intervention.
• Security groups: VMs within one group can communicate with each other, but groups are separated (not only by IP subnetwork, but also by VLAN!). Default is one security group per user, you can create more or share a group if you like.
• Fully scriptable.
• Authentication by X.509 certificates.
• Open source, easy to customize and modify.
• Relatively easy (for such complex system) installation

Cons:
• Sometimes incomplete documentation.
• Some planned functionality is still missing, namely: multi-cluster grids, ability to restart controller.
• Requires custom scripting for many typical administration tasks. This should also change in the new release.
• Some small, but annoying issues. For example, you should use the same Linux distribution on all controllers (note: not VMs, they can run anything your hypervisor supports); controller timezone should be set to ETC.

Eucalyptus from the user's perspective

First time user needs to login to a web interface and request an account. After administrator's approval (this is the only part that requires human interaction), she downloads a personal certificate. User also has to install client software: either Eucalyptus euca2ools, Amazon's EC2 API/AMI tools or a GUI client, eg. EC2Dream. Whatever the choice, the software has to be configured by providing the certificate plus all the URLs, usernames etc.

From that point, user can begin working with the system. There are commands for:
- running a new instance of virtual machine,
- stopping a running instance,
- configuring instances (eg. assigning network resources, security groups),
- storing, retrieving and moving around files inside Eucalyptus storage service,
- creating and modifying VM images.
They work exactly the same for EC2 and Eucalyptus.

Architecture of Eucalyptus

Eucalyptus system consists of 3 parts:
• Cloud controller - to manage the whole installation.
• Cluster controller - in future, it will be possible to divide the installation into several clusters. Cluster is a group of machines connected to the same LAN, cloud can be distributed. However, current version (1.5.2) doesn't support it yet. For now, cloud controller and cluster controller has to be the same host.
• Node controller - on each physical machine that will run VMs.

The client only interacts with cloud controller, which then sends commands and reads replies from the nodes. Note that for a publicly available cloud, you only need a public IP address on a cloud controller.

Eucalyptus uses a storage service called Walrus - an open source software using the same public API as Amazon S3. All VM images are initially stored in Walrus in compressed and encrypted form (Walrus can temporarily cache unencrypted version for performance reasons). Images can be either private (readable only for the user who stored it) or public. Whenever the client requests to run a new instance, the controller decrypts the VM image and sends it the node. Node controller then stores it on a local disk and attempts to run it.

Is EC2 API the right choice for you?

The system closely follows EC2, which can be a disadvantage in some scenarios:
• VM instances are treated as disposable resources. If they fail, it's up to the user to respawn them. Whenever they are shut down, all the data is lost (you have to use S3/Walrus for persistent storage). There's no support for failover (although the user can made one inside the VM), migration etc.
• Node controllers store VMs on local disks. Obviously, you can use a network drive instead, but Eucalyptus will be unaware of it. Even if you mount the same volume on both node and cloud controller, Eucalyptus will still copy the files from one machine to the other. This can severely affect your network performance if you spawn a lot of VMs.
• Some systems allow to configure VMs while deploying them, but Eucalyptus doesn't care at all about what's hapenning inside the VM. It's an intentional design choice: it's a multi-user cloud, so VM configuration is a user's job. If you need a centrally-configured system of interoperating VMs, there might be other choices. Eucalyptus allows for that, but doesn't provide any support.

There is, however, one major advantage of using EC2 comptabile platform: ability to create a hybrid cloud system. Sure, you need to write your custom scripts to deploy VMs. But once you have them, switching from local to EC2 resources and back is a matter of a few shell variables.

Unified cloud API

2009-09-11T12:16:00.001+02:00

I've already written about blurring the border between cloud-based and private servers. EC2-compatible platforms, like Eucalyptus, do exactly that. RedHat's project Deltacloud goes a step further.

Deltacloud will provide unified, RESTful API for Amazon EC2, RedHat Enterprise Virtualisation, VMware ESX and RackSpace. First two are already available. Deltacloud Portal is a web interface for Deltacloud.

As a famous aphorism of David Wheeler goes: All problems in computer science can be solved by another level of indirection. VMware won't let go of it's API, nor will EC2. Other vendors might do that, but only if market forces them. The only solution is to wrap all the APIs in another layer that presents a common interface. Deltacloud also promises the interface won't change even if the underlying API is upgraded. If you need to run scripts that can manage both local and distributed VMs, Deltacloud seems like a good idea.

(disclaimer: I didn't try it yet, but it's on my agenda)

Xen News

2009-09-06T11:09:00.002+02:00

This summer wasn't much of a holiday season for Xen developers. In the last two months, I saw many exciting changes around this platform and more are coming.

• Xen 3.4.1 and 3.3.2 were released, after a series of release candidates. Just maintenance realeses without important new features.
• Xen users FAQ was updated. It's a good idea to skim this file regularly, the solutions are very useful.
• A series of video interviews with developers have been posted on Xen Blog
• Dozens of new and updated howtos on Xen Wiki, see

These are changes that every user can see, but more interesting stuff happens under the hood - in the development version, or soon-to-come Xen 4. New features include:
• Virtual switch. I had it covered in a separate post.
• Ability to restart dom0 without affecting VMs. Read: smooth upgrades without the need to migrate all VMs elsewhere.
• Much better graphics support, both device-indepented way (for better compatibility) and device-dependent (for performance). How about a remote gaming or virtual 3D design workstation?
• Xen Cloud Platform. Again, covered elsewhere.

Private clouds are coming

2009-09-06T11:05:00.003+02:00

I've been reading news form WMworld for the last few days. Announcements are somewhat vague and cloudy (pun intented), but after filtering out all marketing buzz with obligatory Whatever-as-a-Service, there's a lot of interesting information. I believe it's a beginning of a new paradigm: private cloud.

Today we see two main uses for virtualization. One is inside the company datacenter: server consolidation. The other is cloud computing, with Amazon EC2 being the most prominent example. Which means you can either deploy a new service on your own machine or on a public one, but you have to decide beforehand and it's hard to switch.

Now the two are beginning to blend. Both Citrix (the company behind Xen) and VMware are launching new initiatives called Xen Cloud Platform (XCP) and VMware vCloud Express. They promise exciting new features, including ability to live-migrate VM between your own server and the cloud! In addition XCP will contain Virtual Switch I wrote about a few days ago and it will run virtual machines created by other systems, including VMware.

One of the possible uses is extending your hardware base to cover for temporary flukes. Most of the time you use your own servers, having the benefit of a fast LAN and not being charged for traffic/CPU time. When you need more resources for a few hours, offload some jobs to external systems. It's possible today, but requires careful planning, experienced admins and lots of scripting now. Soon it will become easier and more common.

I've been playing with a similar setup for a few weeks now with a software called Eucalyptus. It's an open-source but commercial-backed system compatible with EC2. Basically, it allows you to setup your own system that has most of EC2 features (sans billing) and works with the same client tools. Some users made really cool stuff with it. My favourite example is automated build and test system. Very common thing in software development companies that traditionally requires buying several servers. In that case however, Eucalyptus runs it's instances on developers' workstations, using minimal amount of resources. When they need more power (eg. a new release is coming), they add some EC2 instances.

Updated drivers for Xen and KVM

2009-08-26T19:43:00.001+02:00

RedHat released a new version of drivers for Windows guests running on KVM - available at
http://www.linux-kvm.org/page/WindowsGuestDrivers/Last_WHQLed_drivers

Also Windows drivers for Xen were updated recently - download site http://www.meadowcourt.org/downloads/. Be careful, these are experimental.

These drivers greatly increase performance of Windows guests by paravirtualizing some operations. It's not the same as complete paravirtualization possible for Linux guests, but much better then nothing. Beware, the new version is considered experimental.

Virtual switches

2009-08-26T19:39:00.002+02:00

Directing traffic between VMs and physical interfaces can get quite complicated. For a simple setup with several VMs, the usual bridge configuration works fine. This is exactly like the physical network with a simple hub. But for complex scenario on a physical network you would use managed switches supporting VLANs, spanning tree, link aggregation, monitoring , maybe even some QoS or security features. Virtual networks need virtual switches.

VMware already got it. The built-in solution is not much better than a simple bridge of Xen, but you can buy a really powerfull Cisco Nexus 1000V. In addition to having more features, it also uses standard Cisco interface familiar to most network administrators. With Nexus your virtual networks behave like a part of your physical network. Anything you might need - VLAN separation for security, IGMP snooping for directing multicast traffic, High Availibility, you name it - is there. Let's say, VM1 and VM2 need to use one VLAN, while VM3, 4 and 5 use another. Can you spread them around your physical network and still assure they will connect to the right network and the network traffic won't get anywhere it's not supposed to? With a mixture of virtual and physical switches it's possible.

The biggest selling point is blurring the border between the virtual and physical world. When all your network appliances are configured and monitored centrally, you don't really care if you're really dealing a huge box with humming fans or a piece of software running on a hypervisor. Just like it happened with the servers.

Xen is still missing a virtual switch. Not for long. The new software called OpenVSwitch is going to be included in the next release. How does it compare to Cisco?
• It supports various virtualization platforms, at the moment Xen (both community version and Xen Server), KVM and VirtualBox. Possibly even VMware in future. There aren't, as far as I know, any technical reasons not to support VMware, only political (VMware is closely cooperating with Cisco).
• It's free. Nexus 1000V is most definitely not.
• At the moment it's not as feature rich. The most requested functions (VLANs, per-VM policy and ability to monitor intra-VM traffic) are already there. More will come, the project
• Due to it's open nature, it will easilly interoperate with various hardware and software appliances.

GPU computing revisited

2009-08-23T12:16:00.005+02:00

Just a few days ago I wrote about computations on video cards. Looks like this technology will be built into Windows 7. Developers will have a new API called DirectX DirectCompute.

Nvidia promises improvement in performance of multimedia and graphics applications. Technically it's possible, but quick look at the developer documentation shows there's nothing really new. GPU programming will continue to be a daunting tasks, which will discourage many programmers. I'm still waiting for the day when it becomes as easy as ordinary programming.

Links:

High-Performance Computing as a Service

2009-08-20T19:51:00.003+02:00

Will HPCaaS be a new addition to the well-known family of cloud buzzwords: SaaS, PaaS, IaaS? A San Francisco-based company Penguin Computing offers on-demand access to a supercomputing cluster. Unlike in the usual cloud techniques, tasks are executed on the physical machines which use typical cluster technologies, like low-latency Infinband interconnects and high-speed storage.

Notable features:
• In addition to hardware, the offer includes service of of HPC experts.
• Nodes are optionally equipped with Tesla units.
• Data can be send and received in two ways: usual, over the internet (I almost wrote: traditional) or by fedexing the hot-swappable 2TB hard drives. Looks funny at a first glance, but it's the fastest way to send really large amounts of data.

Businesses which only occasionally need HPC can find this offer interesting. Hard core users of clusters, like pharmaceuticals corporations or video animation companies would probably keep using in-house facilities, but for them it's a good way to temporarily increase processing power.

A cluster of mobile phones?

2009-08-20T19:01:00.007+02:00

An emerging trend in datacenters today is Green IT. An ordinary user running a PC or laptop, or even a business with few dozens of them dont't think much about energy they consume. But a server room can easilly exceed 1 MW of power draw. Add another MW for air conditioners or your equipment will melt under its own heat.

No wonder that in every technical magazine and conference I see articles about new technologies aiming to reduce power consumption in large installations. Possibilities are countless. For example, unused hardware in cluster nodes can be powered off and CPUs scaled down, like in laptops. By properly aligning data, we can even power off some disks in the storage devices (google for MAID: Massive Array of Idle Disks). Simple redesign of a rack can greatly reduce the cost of cooling. And so on.

There's a limit on how far we can go if we decide to stay within today's architecture. What are other options? Browsing the archives of Usenix Technical Summit looking for something else, I accidentally found a talk by Adrian Cockcroft about using mobile hardware for supercomputing. Why it's a good idea? He gives a simple analogy.

In the 80's and 90's, the speed of PCs improved much faster then their bigger cousins. Today most servers don't run PA-RISC or SPARC anymore, they use the same x86 architecture as the home computers. We even almost forgot about special supercomputer or mainframe CPUs, it's better to throw in a bunch of Athlons or Xeons. But for the last few years, PC hasn't been the fastest growing child anymore. Mobile hardware used in smartphones and PDAs is improving at much higher rate. It seems likely that in a few years, for a given amount of processing power, we'll have an option of using 100 PCs or 500 smartphones. And these units were designed with low power consumption in mind.

A group of enthusiast picked up this idea called millicomputing and built several systems. The most interesting is MilliCluster. It consists of 14 ARM CPUs running at 1 GHz. Each has it's own 256 MB RAM and 4 GB Flash. They're interconnected with USB, external connection is dual gigabyte Ethernet. Peak power usage is 20 W, idle only 3 W! I suppose the Ethernet port is responsible for half of the idle power consumption. Need more processing power? 8 millicluster fit easily into single 1U rack module, together with PSU and some networking hardware.

Today it's just a proof-of-concept. You don't get the most bang-for-the-buck with mobile CPUs (although you get the most bang-for-the-watt). Theoretical performance can be impressive, but it's in form of numerous low-speed nodes, connected with relatively slow network. That's acceptable for some type of problems (think SETI@Home, Boinc and other distributed projects), but a huge issue for others. However, it's a system built by enthusiast with of-the-shelf technology like USB. My guess is the proper internal bus would really boost the system like this. Don't worry about the number of CPUs. Ordinary PCs get more cores every year, the software we'll have to make more use of parallel architecture anyway.

See Adrian Cockroft's blog for more information and news.

(Image from Adrian Cockroft's presentation published under Creative Commons Attribution Share-Alike 2.5)

Turning your video card into a supercomputer

2009-08-19T20:17:00.005+02:00

Using GPUs for computation is not a new idea. Actually, games have been doing that for years. Older video cards only accelerated functions directly related to displaying images, but they were given more features every year. Today, most of the 3D processing and even some game physics calculation are done by GPUs. They outperform CPUs for this kind of tasks because of their massively paralell (SIMD) architecture: same operations are made on numerous variables at a time.

Which is basically what scientist do with their clusters and grids. No wonder someone came with the idea of using GPU for the same purpose. At first, programmers were using tricks like disguising their code as OpenGL. Later, Nvidia and ATI provided libraries for generic-purpose calculations. First time I read about it, I was really excited: they promised supercomputer-like performance on a cheap video card. Turns out it's not that bright.

At my institute we tested Nvidia CUDA library with several cards, both ordinary GeForce and special purpose Tesla units like the one pictured. Disclaimer: I haven't done much work with that technology, my colleagues have. Most of what I wrote here is based on their experience. What do they say?

• Programming a GPU looks like a trip to the past. You have to deal with low-level stuff like registers, buses and memory locations. Compiler won't do that for you.
• Some operations are blazingly fast, but only when using registers. Mostly your GPU will waste it's clock cycles waiting for the data.
• GPUs are optimised for single-precision arithmetic without some standard features like NaN (see Wikipedia article on floating point if you don't know what it means). Scientific calculations mostly use double precision arithmetic and require all standard features. CUDA supports them, but at the great performance penalty.
• The overhead of copying data between CPU and GPU sometimes eats up all the benefits of faster calculation.

Still, in some applications they saw about 20x performance increase compared to CPU. That obviously depends on what hardware you use, but generally you get the similar rate if you compare high-end Xeon to Tesla and entry-level CPU to cheap GeForce.

Will it have an impact on ordinary users? Maybe. If GPU vendors provide better tools, we'll see this technology used more often. Some typical home applications can be parallelized. Imagine encoding a video file 20x faster than now. Or an image processing software, like Gimp or Photoshop, could use it to speed up operations. But I don't suppose many developers would like to spend hours tweaking their RAM-to-registers loading code. Assembler is rarely used on ordinary CPUs, because the compiler can do the same job good enough, sometimes even better than human. We need the same for GPUs, otherwise the technology won't be used outside the small realm of scientific computations.

Lots of information on this technology can be found in Wikipedia, see:
http://en.wikipedia.org/wiki/GPGPU
http://en.wikipedia.org/wiki/CUDA

Alacricity

2009-08-18T19:08:00.003+02:00

The world of Linux virtualization solutios doesn't get any less active. Just a few weeks ago we got an interesting addition. Alacricity is a project that aims to provide much faster I/O for virtual machines.

I/O performance in VMs is noticably slower compared to real machines. My favourite hypervisor Xen works great for pure-CPU workloads, computations takes maybe 0.1% longer compared to native machine, but it gets noticably worse when doing lots of network or disk activity. For a typical server environment it usually causes about 5-10% decrease in performance, acceptable, but improvements are always welcome. However, form some tasks, like high performance file server, it gets much worse. I/O issue is also one of the major obstacles of adopting virtualization in cluster environments. After all, nobody wants to invest millions (literally!) in high-bandwidth, low-latency InfiniBand or 10G Ethernet interconnects to waste it all on virtualization overhead. Then there are also real-time systems in which every nanosecond counts.

AlacricityVM is based on KVM, but introduces a new I/O fabric called Virtual-Bus. It greatly reduces the layers of indirection, thus allowing for greatly increased thoughput and - what's even more important for High-Performance Computing - greatly reduced latency. See Vbus introduction pdf if your interested technical details and the article on lwn.net for general info.

The project is at en early stage, definitely not ready for production environments. I'm looking forward till it's mature, I'll probably make a test deployment in a few months. Judging by the information I've read, it can really be a breakthrough. I mostly use Xen, but it might be a reason to switch (good thing I tend to use hypervisor-agnostic management tools). It's also quite possible for the technology to be integrated into Xen - didn't try it, but it seems possible.

Filesystem performance on Xen

2009-08-18T19:05:00.003+02:00

Few days ago I decided to test filesystem performance on Xen. I read lots of Linux fs benchmarks, but none of them concerned domU. My thought was: Xen I/O performance is noticably worse than native and differences between filesystems might be amplified.

I won't give you all the details, I didn't even recorded them. My aim was to get some hints about which technology should I choose, not the exact numbers. I ran some real-life workloads, which have an obvious disadvantage: they weren't designed to test any particular component in isolation. Fine for me, I wanted to know if a particular fs would let me do my work faster, not if it looked better in the benchmark.

Filesystems I checked:
All mounted with noatime:
• ext3 created with default options
• ext3 with dir_index,
• ext3 with data=ordered, writeback or journal
• JFS with default options

Why I didn't check some of the others:
• btrfs, ext4 - not stable enough,
• reiserfs - I had very bad experience in the past (severe data loss) and I'm not giving it a second chance,
• XFS - known for poor unlink performance.

My main test was several tasks running in parallel, I was interested in total execution time. They were using files of different sizes, from few hundred B to few dozens MB and also doing intensive CPU work. I specifically didn't try to compensate for external factors like caches. The conclusion was suprising: there was no signifacant difference. And, if any fs performed better, it was ext3 with default options, but the differences were below 1%. One exception was ext3 with dir_index, which was noticably slower.

I was astounded. I've seen so many benchmarks, performance tuning howtos, even flamewars. Turned out that old, boring ext3 without any meddling was the best choice. I always chose it anyway, for stability and compatibility. I knew if anything go wrong, all data recovery tools work with ext3. It has it's shortcomings, especially very long fsck time on large partitions, but there's always a tradeoff.

I also tried a low level test - bonie++. This time I could see some differences. Generally, each FS or option performed slightly better in one place and worse in the other. If your needs are specialized, eg. you use millions of small files or few very large ones, you need low seek time and don't care much about latency or vice versa, it might be worth to try. For a general-purpose machine, all these strong and weak points seemed to had compensated for each other. My advice, if you still want to find the fastest fs, use a simulated workload instead of low-level benchmark.

There werse also significant (10-20%) differences between the runs of bonie++. I also tried the same test (ext3/defaults only) on a real machine. The results not only were better, but also more similar on each run. The "real-life" test finished about 20% faster.

Xen reliability

On a slightly related note, one of my spare VMs has been running crashme for days. Nothing happened. The test could have failed for 3 reasons: bug in Linux kernel, Xen hypervisor or hardware issue, but all components worked flawlessly.

Xen vs. KVM vs. the rest of the world

2009-08-14T17:17:00.012+02:00

No website about Xen can be considered complete without an opinion on this topic. KVM got included into the Linux kernel and is considered the right solution by most distributions and top Linux developers, including Linus Thorvalds himself. This made many people think Xen is somehow inferior or is on the way to decline. The truth is, these solutions differ both in terms of underlying technology and common applications.

How Xen works

Xen not only didn't make it to the main tree of the Linux kernel. It doesn't even run on Linux, although it looks like it. It's a bare metal hypervisor (or: type 1 hypervisor)- a piece of software that runs directly on hardware. If you install a Xen package on your normal Linux distribution, after rebooting you will see Xen messages first. It will then boot your existing system into a first, specially privileged virtual machine called dom0.

This makes the process quite complex. If you start experimenting with Xen and at first attempt make your machine unbootable, don't worry - it happened to many people, including Yours Truly. You can also download Xen Server - commercial, but free distribution of Xen which comes with a simple to use installer, a specially tailored, minimal Linux system in dom0 and enterprise-class management tools. I'll write some more about diffences between XenServer and "community" Xen in a few days.

It also means you won't be able to manipulate VMs using ordinary Linux tools, e.g. stop them with kill and monitor with top. However, Xen comes with some great management software and even greater 3rd-party apps are available (be careful, some of them don't work with Xen Server). They can fully utilize interesting features of Xen, like storing snapshots of VMs and live-migration between physical servers.

Xen is also special for its use of technology called paravirtualization. In short, it means that the guest operating systems knows it runs on a virtualized system. There is an obvious downside: it needs to be specially modified, although with open source OSes that's not much of an issue. But there's also one very important advantage: speed. Xen delivers almost native performance. Other virtualization platforms use this approach in a very limited way, usually in form of a driver package that you install on a guest systems. This improves the speed compaired to completely non-paravirtualized system, but is still far from what can be achieved with Xen.

How KVM works

KVM runs inside a Linux system, not above it - it's called type 2, or hosted hypervisor. This has several significant implications. From technical point of view, it makes it easier to deploy and manage, no need for special boot-time support; but it also makes it harder to deliver good performance. From political point of view, Linux developers view it as superior to Xen because it's a part of the system, not an outside piece of software.

KVM requires CPU with hardware virtualization support. Most new server, desktop and laptop processors from Intel and AMD work with KVM. Older CPUs or low-power units for netbooks, PDAs and the like lack this feature. Hardware-assisted virtualization makes it possible to run an unmodified operating system with an adequate speed. Xen can do it too, although this feature is mostly used to run Windows or other proprietary guests. Even with hardware support, pure virtualization is still much slower than paravirtualization.

Rest of the world

Some VMware server platforms and Microsoft Hyper-V are bare-metal hypervisors, like Xen. VMware's desktop solutions (Player, Workstation) are hosted, as well as QEMU, VirtualBox, Microsoft Virtual PC and pretty much everything else. None of them employ a full paravirtualization, although they sometimes offer drivers improving the performance of guest systems.

KVM only runs on machines with hardware virtualization support. Some enterprise platforms have this requirement too. VirtualBox and desktop versions of VMware work on CPUs lacking virtualization support, but the performance is greatly reduced.

What shoud you choose?

For the server, grid or cloud

If you want to run Linux, BSD or Solaris guests, nothing beats the paravirtualized performance of Xen. For Windows and other proprietary operating systems, there's not much difference between the platforms. Performance and features are similar.

In the beginning KVM lacked live migration and good tools. Nowadays most open source VM management applications (like virt-manager on the screenshot) support both Xen and KVM. Live migration was added in 2007. The whole system is considered stable, although some people still have reservations and think it's not mature enough. Out of the box support in leading Linux distributions is definitely a good point.

VMware is the most widespread solutions - as they proudly admit, it's used by all companies from Fortune 100. Main disadvantage is poor support from open source community. If free management software from VMware is not enough for you, you usually have no choice but to buy a commercial solution - and they don't come cheap. Expect to pay several thousand $ per server or even per CPU.

My subjective choice would be: 1 - Xen, 2 - KVM, 3 - VMware ESXi.

For the personal computer

While Xen is my first choice for the server, it would be very far on the list of "best desktop virtualization platforms". One reason is poor support for power management. It slowly improves, but still I wouldn't install Xen on my laptop. Also the installation method is more suitable for server platforms, but inconvenient for the desktop.

KVM falls somewhere in the middle. As a hosted hypervisor, it's easier to run. Your Linux distribution probably already supports it. Yet, it lacks some user-friendliness of true desktop solutions and if your CPU doesn't have virtualization extensions, you're out of luck.

VMware Player (free of charge, but not open source) is extremaly easy to use, when you want to run VMs prepared by somebody else (hence the name Player - nothing to do with games). Creating a new machine requires editing configuration file or external software (eg. this web-based VM creator). What I really like is convenient hardware management (see screenshot) - just one click to decide if your USB drive belongs to host or guest operating system, another to mount ISO image as guest's DVD-ROM. Another feature is easy file sharing between guest and host. Player's bigger brother is VMware Workstation (about $180). It comes with the ability to create new VMs as well as some other additions. Due to the number of features it slightly harder to use, but still very user-friendly.

VMware offers special drivers for guest operating systems. They are bundled with Workstation, for Player they have to be downloaded separately (or you can borrow them from Workstation, even demo download - license allows it). They are especially useful if you want to run Windows guest, even on older CPUs without hardware assist it's quite responsive.

VirtualBox comes close to VMware. It also has the desktop look&feel and runs on non-hardware-assisted platforms. Bundled guest additions improve performance of virtualized systems. Sharing files and hardware is easy - but not that easy. Overall, in both speed and features, it comes second.

My subjective choice: 1 - VMware Player or Workstation, 2 - VirtualBox, 3 - KVM

EDIT: I later found out that new version of VirtualBox is superior to VMware Player.

What other people think?

Read some more comparisons:

Xen vs. KVM at gridvm.org (very interesting site, by the way)
Xen vs. KVM
Xen vs. KVM from LinuxWorld 2007

Why virtualize servers? 10 reasons.

2009-08-10T15:03:00.006+02:00

People who didn't experience the benefits of virtualization often ask what's the big deal. The often repeated argument about consolidation is flawed: after all, instead of running 10 VMs, each serving a single app, you can have one multi-purpose server. Having only one system to maintain means less work. True? Not quite. There are reasons for separating applications in either physical or virtual computers. The first option is clearly too expensive in so many ways (not only initial costs, but also power, cooling, space, maintenance) and VMs even have some advantages over real computers.

1. Isolation

I believe it is possible to have a machine serving HTTP, FTP, DNS, SMTP, POP, IMAP, 2 types of SQL and 5 types of application servers in addition to doing its own resource monitoring and intrusion detection. I used to have boxes doing 80% of that and obviously I could have just installed another app. But the whole setup is barely maintainable. Debugging a problem becomes extremely difficult, when you have to filter out all the irrelevant information. Upgrades are nightmare. There's no way of writing a comprehensive SELinux policy. And worst of all, one misbehaving app using to many resources can bring the whole system to it's knees. Yes, you can impose some limits by PAM, but not on all the resources and anyway, you can't predict anything, especially when customers are allowed to run their applications.

2. Security

On a multi-purpose server, a security hole in one application exposes all the others. An attacker who gained access using an exploit on a web application can than read your mail or alter your DNS zones. All the modern operating systems have some ways of preventing that. Applications can be isolated by running with different UIDs or using chroot jails. That has never been enough to stop the intruders. After an intrusion, you usually have no option but to reinstall the machine from scratch or at least restore a backup and hope it doesn't contain any rootkits.

A successful attack on a virtual server is much less devastating. If you followed the advice of having a single-purpose machine, there are no other applications on that system, no tools that malicious user can use to his advantage. If you want to investigate, suspend the VM and resume it on another machine without a network connection. Restoring the attacked machine is a matter of minutes (you DO have snapshots of your VMs, right?).

And that's provided you didn't avoid an intrusion in the first place. Single purpose machines are way easier to harden. You can check which binaries are run and which files are read and disallow everything else with ACLs. If you know it's only going to run a web application, just filter all the ports except HTTP.

3. Dealing with software incompatibility

Sometimes it's impossible to run 2 apps on the same machine. They can rely on a different version of a library or other component. Or they both need to use the same TCP port. Commercial apps on an open-source system tend to cause problems. They are less configurable, often expect to be installed in a specific place, sometimes require a specific OS distribution. It's easiest to have a separate system for them. There are workarounds, but they make your setup even more messy, hard to test and to maintain.

4. Maintaining legacy systems

Another issue are legacy servers running in many companies. Nobody dares to touch them, because the last guy who knew how they work retired last century. They run a different operating system that you never even heard of. They have small hardware requirements, but where you can get replacement parts for that 486? These machines are prime candidates for virtualization.

5. Easier software upgrades

Now that you have single-purpose machines, upgrading became much easier, but that's not the end of good news. You don't have to upgrade your critical business server in-place. Just create a new virtual machine, install a new version and test it thoroughly. When it's done, switch the virtual plug.

6. Easier hardware upgrades

Upgrading hardware has never been easier. Hypervisor exposes the same set of devices on every box (unless you configured it to give a VM a direct access to a specific hardware, which you shouldn't do without a good reason), that means no worries with the device drivers. When you need more processing power, plug another server and migrate some VMs from the old ones. Popular hypervisors Xen and VMWare allow a live migration with almost none downtime (usually below 100 ms).

7. High availability

HA installations used to be found only in the richest companies and even there only for the mission-critical servers. After all, you had to double all your hardware, also doubling your power, cooling and space expenses. Not any more.

With VMS, you can take snapshots. You can live migrate. You can suspend them, at which point they use virtually (pun intended) no resources, but they can be quickly resumed. I think you get the image now: all you need to run a HA installation is some more planning and configuration in the beginning.

8. Dealing with increased load

The peek load of the server is usually orders magnitude higher than the standard load. With traditional server, you had to buy one that can sustain peek traffic (meaning: expensive hardware, more power draw). Or buy one that's right for your day-do-day needs and hope nobody mentions your website on slashdot or digg. VMs, as usual, give you more choices. You can migrate them around your physical servers to balance the load. You can suspend some VMs to give more resources to the others. If your infrastructure is compatible with public clouds (eg. Amazon EC2), you can rent some processing power for a negligible cost.

9. Backup and restore

Backup is always simple, whether in a virtual of physical environment. After some initial configuration, it happens automatically. It's restoring that's difficult.

Restoring a traditional server is a long, manual process. Hours of downtime, during which both IT and financial people develop ulcers. It's also hard to test, which means the test is often postponed indefinitely until a disasters strikes, at which point you find out restore procedure fails. With VMs, you can save a complete snapshot (disk and RAM image) of a running server. Restoring is a matter of copying the image to the right place. Testing restore? Just start another VM.

10. Testing new software

In a typical corporate environment, all machines are already put to a good use. If you need one for a test deployment, it takes a lot patience, paperwork and begging to get it. If you need several machines to test a networking app, your completely screwed. Not any more. Creating a new VM is a matter of minutes. You don't need high performance for a test, which means you can squeeze dozens of VMs on one box.

Virtually A Machine

Xen vs. KVM yet again

How to run a network monitoring station on a XenServer VM

Important new releases

Xen support in mainline Linux kernel

Password storage, part 2 - my choice

Possibly related articles

Password storage, part 1 - considerations

Possibly related articles

Xen 4.1 released

Related articles

EC2 can now import VMware images

Related articles

Gartner: 6 myths about server virtualization

Related articles

What's new in Fedora?

Related articles

Amazon introduces EC2 GPU instance

Related articles

Ubuntu 10.10 - what's under the hood

Related articles

Use Amazon Web Services for free

Related articles

Creator of ZFS leaves Oracle

Related articles by Zemanta

EC2 quick news - September 2010

Related articles by Zemanta

Official EC2 Linux instances

EC2 micro instances

Live migrating Ubuntu to Debian - howto

Live migrating Ubuntu to Debian - introduction

Howto: disable eCryptFS

Virtualization news - May 2010

Related articles by Zemanta

New Apache projects, Xen news and my learning project

Related articles by Zemanta

I'm not upgrading to Ubuntu Lucid

Related articles by Zemanta

Xen 4 released

Related articles by Zemanta

Qubes: secure Xen-based desktop OS

Ubuntu Lucid window controls

Xen 4 on Ubuntu Lucid

Software to learn

IE 9 will use GPU to render pages

Eucalyptus howto

Nexenta: Power of Solaris with usability of Linux? Not yet

Introduction to desktop virtualization

Parallel processing in shell scripts

Virtualization news: KVM, Xen, AlacricityVM, VirtualBox

XenServer goes open source - again

Amazon S3 gets versioning support

Xen.org pamphlet in Polish

Solaris for Linux admins - part 3: Blastwave

Solaris for Linux admins - part 2: package management

New Xen Documentation

Solaris for Linux admins - part 1: introduction

Fedora goes towards KVM, away from Xen

Howto: two Firebirds on one Linux box

Xen.org overview slides

Free academic journals on HPC

Experimental Xen and Ubuntu. Part 2: hypervisor

0. Prerequisites

1. Get ACPICA

2. Get the sources

3. Compile hypervisor and tools

4. Create packages

5. Install packages

6. Update your Grub2 configuration

7. Run Xen daemon at boot

Troubleshooting

Extra Apt repositories in Ubuntu

MediBuntu

GetDeb

VirtualBox

What's next?

Experimental Xen and Ubuntu. Part 1: kernel

0. Prerequisites

1. Create source directory

2. Get the sources