No Name Blog

Magic SysRqs

One of the most powerful ways to debug or recover Linux is through the Magic SysRq keys.

Normally, if you’re using a serial console, you send BREAK, followed by the command you want. However, this doesn’t work with the Xen paravirtualized serial console driver. Instead, you have to use Ctrl+o, followed by the command you want.

For instance, to do an emergency sync, press Ctrl+o, then s.

Note that you use Ctrl+o instead of BREAK for both dom0 and domU serial consoles – they both are using the Xen paravirtualized driver.

Paravirtualized Clocks

In theory, Xen dom0′s are supposed to forcibly sync their system clock to the domU’s. In practice, due to some incompatibility between either Ubuntu’s version of the dom0 or domU patches, that doesn’t work, even though the feature is enabled, which leads to clock drift and occasionally weird clock lockup bugs.

The easiest way to fix this is to disable the Xen clock syncing entirely, and rely on the standard Linux clock mechanism. You can do that by adding these two lines just before exit 0 in /etc/rc.local:

echo '1' > /proc/sys/xen/independent_wallclock
echo 'jiffies' > /sys/devices/system/clocksource/clocksource0/current_clocksource

You’ll want to be sure to run NTP or some other service to keep your clock in sync.

Now for part two in my ongoing series on making Xen suck less. Last time we looked at making networking work for hardware virtualized machines. Networking for paravirtualized VMs does work out of the box, but this hint might help if you’re running into performance problems.

Paravirtualized Networking Performance

If you’re running a web server or some other server that’s sending large files (or sometimes small files), you may find that your VM seems to hang inexplicably on those transfers.

For some reason, the paravirtualized Xen networking drivers advertise that they support on-board TCP segmentation. In fact, they seem to pass the packets onto the wire un-segmented, which frequently will cause the packets to be dropped for going over the MTU.

If you’re using xen-create-image, there’s a commented out line in /etc/network/interfaces that runs ethtool -K eth0 tx off. That’s close to the right issue. You actually want to add a line to your /etc/network/interfaces so that it looks something like this:

auto eth0
iface eth0 inet static
 address 18.181.0.80
 gateway 18.181.0.1
 netmask 255.255.0.0

 post-up ethtool -K eth0 tso off

Stay tuned for more hints, including how to deal with clock issues and magic sysrqs. I’ll also be pulling walkthroughs together on converting paravirtualized to hardware virtualized VMs, and how to upgrade older Ubuntu releases to more recent ones safely.

Right now, all of my Xen dom0′s run Ubuntu Hardy with Xen 3.3 from hardy-backports. Before we even talk about making Xen work, that statement bears some looking at.

I use Xen for a variety of reasons. Some are historical – the Invirt Project was built on top of Xen, and migrating away from Xen to a solution like KVM or VMWare would require working with users that are running paravirtualized operating systems. Some are circumstantial – I still have hardware that doesn’t support hardware virtualization.

My reasons for using Ubuntu are far less logical – I know how to use it, and don’t want to learn anything else if I don’t have to.

And I use Xen 3.3 because it’s way more stable than Xen 3.2.

In any case, if you find yourself using Xen 3.3 on Ubuntu Hardy as a dom0, there are a lot of tricks I’ve picked up for making it work better. Over the next few weeks, I’ll be working my way through them. I’ll be tagging them all with xen-tips for easy retrieval later.

As a disclaimer, I have no idea if these problems have been fixed in later versions of Xen or Linux, or if they’re specific to the Xen and/or kernel shipped by Ubuntu. For me, there’s a lot of value in getting all of my software from my distribution, so these instructions are designed to help do that.

HVM Networking

I have no idea whose fault this is, but HVM networking just doesn’t seem to work out of the box. qemu-dm, which emulates the VM’s devices, hooks the VM to a tap net device, while Xen sets up networking for a vifN.0 device. As far as I can tell, the intent was to connect the tap and vif devices, but nothing does.

For Invirt, we worked around this by writing a wrapper script around qemu-dm to make sure everything was setup correctly. If you want to use this script, you can drop qemu-dm-invirt in /usr/sbin and qemu-ifup in /etc/xen. (You’ll probably want to replace vif-invirtroute in qemu-ifup with vif-bridge or vif-route or whatever networking script you’re using).

/usr/lib/xen/bin/qemu-dm is hard-coded to run /etc/xen/qemu-ifup, if it exists. Without the qemu-dm-invirt wrapper, though, qemu-ifup doesn’t have any access to the domain ID for the domain it’s setting up. qemu-ifup then sets up and triggers the normal Xen networking script, which repeats the same setup it did for the vifN.0 interface.

Then, in your Xen config file, be sure to set device_model = '/usr/sbin/qemu-dm-invirt'.