Virtualization: Notes to self

This is just things I wrote down while playing with QEMU/KVM virtualization, for my own purposes of packing two existing computers into a third one. There is no point to make here, and neither do I expect anyone to understand this. It’s published because I don’t care to.

Log files

There are definitely two files one wants to peek at every now and then

• /var/log/libvirt/qemu/{guest-name}.log (in particular when a USB device doesn’t attach)
• /var/log/audit/audit.log (SELinux audit, possibly piped to grep -E ‘^type=(AVC|SELINUX\_ERR)’ to reduce amount of junk)

Start with running virt-manager as a non-root user (it will fail with a nondescriptive message if trying to run it as root). A root password will have to be fed.

Use qcow2 disk images

If you’re going to play around with the image, and then maybe want to get rid if the changes, this is sooo simple:

#qemu-img create -F raw -b clean-hda.img -f qcow2 hda.qcow2

Note that qemu-img can also create and apply snapshots of images, which is also good.

Don’t try to use Virtual Machine manager’s clone utility for qcow2 images, since the tool will rewrite all data.

USB device passtrough

It looks like libvirt doesn’t massage the permissions for USB character devices, before adopting them, so both classic permissions and SELinux blow when trying to run passthrough. The current workaround would be to change the USB device’s classic permission manually, and run in permissive mode. Which is indeed unhealthy, but harmless given the fact that XP gives a healthy blue screen in response to the new device. I think I saw XP complaining something about USB 2.0 versus USB 1.1.

Running the same scenario with a knoppix LiveCD image, I managed to find the device (a Canon 500D camera) with lsusb. The root USB hub was shown to be a UHCI. I’m not sure whether a real PTP camera would respond to a UHCI hub, which maybe explains why Windows got confused finding such a camera under the emulated hub.

The emulator appears as /usr/bin/qemu-kvm under the SELinux domain (type) svirt_t, which is decleared in /usr/share/selinux/devel/include/services/virt.if (which you don’t want to read).

USB devices appear somewhere under /dev/bus/usb/ as character devices with SELinux type usb_device_t.

Command line

Create a guest, run it and pause it. Then dump its XML by something like (as root):

# virsh dumpxml try1 > ~eli/virt/try1.xml

Then play around with the XML file a bit, destroy the previous domain, and

# virsh create ~eli/virt/try2.xml

How can I manually manipulate the list of guests?

virsh # start demo
error: Failed to start domain demo
error: internal error unable to start guest: qemu: could not open disk image /home/eli/virt/tryxp.img: Permission denied

Reason(?): tryxp.img is of the wrong context, so SELinux prevents it to be opened…? But I ran SELinux in permissive mode. How could this happen? Or, as put in the Fedora 12 Virtualization Guide:

SELinux prevents guest images from loading if SELinux is enabled and the images are not in the correct directory. SELinux requires that all guest images are stored in /var/lib/libvirt/images.

Basically, what solved this was to move the image to the dedicated directory, and going:

# virt-install --force --name demo3 --ram 1024 --import --disk path=/var/lib/libvirt/images/tryxp.img

or even better:

# virt-install –force –name demo5 –ram 1024 –import –disk path=/var/lib/libvirt/images/hda.img

For playing around:

# virsh

“Stealing” command lines from Virtual Machine Manager

After running a machine under the GUI interface, it’s possible to do so for running on an external VNC console. Just find the command with ps aux | grep qemu-kvm. The following changes in flags apply:

• Remove the -S flag. It says that the guest should not start until commanded to do so.
• Remove the -montitor flag. We’re running on VNC only
• Change the -vnc flag’s address to point to an address known to the outside world, if necessary
• Add “-usbdevice tablet” so that the mouse is followed correctly
• Change the -net flags (two of them) to “-net nic -net user”. This is said to have a performance hit, but it’s simple and it works with an internal (fake) DHCP server

The tip of the day

In the relevant terminology, “source” refers to information seen by the host OS, while “target” to the guest OS.

Another little tip: If I try to install Windows 7, and the installation gets stuck for very long periods of time with nothing really happening, maybe it’s because the disk image is read-only? :-O

VMPlayer

Running VMplayer on a 2.6.35 kernel requires a small fix, which was published here. The thing is that some kernel symbol has changed its name, and hence the vmmon module fails to compile. How I love when people are sensitive about backward compatibility.

To make a long story short, one needs to go to where the VMplayer module sources are, and go:

$ perl -pi -e 's,_range,,' iommu.c

which is bit of a Columbus egg, I would say. Also, VMplayer repeatedly wanted to compile the modules every time I started it, because it missed vsock (which wasn’t compiled in the first place), so I followed the same page‘s hint and edited /etc/vmware/config to say

VSOCK_CONFED = "no"

By the way, I tried to figure out what this module does, and all google results tell you how to tweak and patch. Don’t people care what they do on their computers? Maybe this component is useful?

The following remark was true when this post was published, but no more:

To run VMPlayer under Fedora 12, there’s a need for a little hack, or the VMPlayer closes right after starting:

# cd /usr/lib/vmware/resources/
# mv mozilla-root-certs.crt old-mozilla-root-certs.crt

Have no idea why this is.

VMPlayer networking

The interesting stuff is at /etc/vmware/networking, which pretty much says which interface is NATed and which is host-only. To pick a certain device for bridging, additional lines configuring add_bridge_mapping should be added as explained on this page.

Also useful to play with

# vmware-networks --status
# vmware-networks --start
# vmware-networks --stop

etc. (as root)

Moving an old Linux computer

The mission: Move an old 2.4.21-kernel based computer into a VMPlayer virtual machine. The strategy was to run a LiveCD on the virtual machine, create an empty ext3 file system on it, copy the files and boot. Caveats follow.

The most important lesson learned, is that everything on the new machine has to be done with a kernel with the same or earlier version compared with the one that will be used. In simple words, the rescue disk must run a matching kernel. In particular, with a newer rescue disk, the ext3 is generated with an inode size of 256, which old kernels don’t support. Even worse, even if the file system was generated properly, newer kernels (say, 2.6.35) writes things on the disk that will confuse the old kernel. This leads to “attempt to access beyond end of device” errors during boot, from 01:00 (the ramdisk) as well as 08:01 (the root filesystem’s partition).

So fdisk, mkfs.ext3 and mkinitrd must be done with a matching kernel running. And copying the files too, of course. The rescue disk must match, as mentioned above.

The next thing to note is that all hda’s turn into sda’s. That needs to be adjusted in /etc/fstab as well as /etc/lilo.conf.

The most difficult thing to handle was the fact that SCSI drivers were not installed in the kernel by default, so the initrd image had to be adjusted. So after the file system is copied, mount it, chroot to it, and go

# mkinitrd --with=scsi_mod --with=sd_mod --with=BusLogic /boot/initrd-2.4.21-mykern.vmware.img 2.4.21-mykern

The insmods are attempted in the same order the flags appear, and the two latter modules depend on the scsi_mod. So it’s important to keep the order of the flags as above.

If these modules aren’t loaded, the generation of /dev/sda and /dev/sda1 doesn’t occur, resulting in a kernel panic with various complains (pivot_mount fails, init not found and some more).

And then fix /etc/lilo.conf and /etc/fstab and run lilo. It’s recommended to copy the /boot directory first, so that the kernel image falls within the lower 4 GB. Or pick lba32 option, as lilo will tell you.

And then boot. Mount the ISO image for VMware tools (/dev/hdc) and run ./vmware-install (going with the defaults most of time).

Converting a VMPlayer machine to VirtualBox

Make a copy of the entire directory to a new place. No point messing up things. Then create an OVF file, without converting the disks (because it takes forever, and possibly fails)

$ ovftool --noDisks CleanXP.vmx export.ovf

For some reason, ovftool doesn’t fill in the correct names of the disks (I asked not to convert them, not to ignore them). The simplest way around this is to remove the disk definitions altogether, and import them manually from Virtualbox. For a VM with three disks, the lines marked in red should be removed:

 <References>
 <File ovf:href="export-disk1.vmdk" ovf:id="file1" ovf:size="0"/>
 <File ovf:href="export-disk2.vmdk" ovf:id="file2" ovf:size="0"/>
 <File ovf:href="export-disk3.vmdk" ovf:id="file3" ovf:size="0"/>
 </References>
 <DiskSection>
 <Info>Virtual disk information</Info>
 <Disk ovf:capacity="10" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
 <Disk ovf:capacity="30" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk2" ovf:fileRef="file2" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
 <Disk ovf:capacity="40" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk3" ovf:fileRef="file3" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
 </DiskSection>

and also the other references to these vmdiskn:

<Item>
 <rasd:AddressOnParent>0</rasd:AddressOnParent>
 <rasd:ElementName>disk0</rasd:ElementName>
 <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
 <rasd:InstanceID>7</rasd:InstanceID>
 <rasd:Parent>6</rasd:Parent>
 <rasd:ResourceType>17</rasd:ResourceType>
 </Item>
 <Item>
 <rasd:AddressOnParent>1</rasd:AddressOnParent>
 <rasd:ElementName>disk1</rasd:ElementName>
 <rasd:HostResource>ovf:/disk/vmdisk2</rasd:HostResource>
 <rasd:InstanceID>8</rasd:InstanceID>
 <rasd:Parent>6</rasd:Parent>
 <rasd:ResourceType>17</rasd:ResourceType>
 </Item>
<Item>
 <rasd:AddressOnParent>0</rasd:AddressOnParent>
 <rasd:ElementName>disk2</rasd:ElementName>
 <rasd:HostResource>ovf:/disk/vmdisk3</rasd:HostResource>
 <rasd:InstanceID>10</rasd:InstanceID>
 <rasd:Parent>4</rasd:Parent>
 <rasd:ResourceType>17</rasd:ResourceType>
 </Item>

Delete the export.mf file. It contains export.ovf’s hash signature, and it’s wrong after editing the file.

Then in the VM VirtualBox Manager, pick File > Import Appliance… and choose export.ovf.

Then add the hard disks by choosing Storage > (Diskette Icon) > Add Hard Disk and pick Choose existing disk. Pick the .vmdk file with no number attached to it (not the e.g. *-s011.vmdk).

Enlarge the video memory to 18 MB (or more), or Virtualbox complains that it’s too little. Enable audio.

That was nice so far. Only problem is that Windows required re-activation because too much hardware changed, and the internet activation failed, probably because the NIC wasn’t detected. To install the driver, I’ll need to activate first, and so I was stuck, lost patience and left the whole thing for now.