While setting up a custom kernel for Ubuntu 14.04 LTS 64-bit to be able to run “virsh” / libvirt tools, many problems have been encountered (see references).
First install all the essential tools for Ubuntu the easy way first:
https://help.ubuntu.com/lts/serverguide/libvirt.html
sudo apt-get install qemu-kvm libvirt-bin
sudo apt-get install virtinst
sudo apt-get install qemu-system
sudo apt-get install virt-viewer
Next a custom kernel is needed, so download a stable kernel source from www.kernel.org. The existing config from current version of Ubuntu 14.04 is copied to “.config”, and the following additional changes made:
CONFIG_NETFILTER_XT_NAT=m
CONFIG_NF_NAT_MASQUERADE_IPV4=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_MASQUERADE_IPV6=m
and then “make oldconfig” applied, before “make” to start the compilation. Then you will need to “sudo make modules_install” and “sudo make install”.
Reboot into the new kernel, and ensure that “nat” show up in /proc/net/ip_table_names. And ensure that “kvm” is listed in “lsmod”.
So to test the setup:
a. sudo virsh net-start default
b. sudo virsh list
c. sudo virsh sysinfo
d. sudo virsh pool-list
e. sudo virsh net-list –all
f. To create a guest VM, first create a file called “guest.xml”:
<domain type='kvm'>
<name>guest</name>
<uuid>f5fe9230-6ef3-4eec-af54-65363a68f3ce</uuid>
<memory>524288</memory>
<currentMemory>524288</currentMemory>
<vcpu>1</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-1.5-qemu-kvm'>hvm</type>
<boot dev='cdrom'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<source file='/var/lib/libvirt/images/guest.img'/>
<target dev='hda' bus='ide'/>
</disk>
<disk type='file' device='cdrom'>
<source file='/home/user/ubuntu1404_x86_64/ubuntu-14.04-desktop-amd64.iso'/>
<target dev='hdc' bus='ide'/>
<readonly/>
</disk>
<interface type='network'>
<mac address='54:52:00:2a:58:0d'/>
<source network='default'/>
</interface>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
</devices>
</domain>
For each of the bold item above:
1. The unique number is generated by “uuidgen”.
2. For the machine type it must come from one of this items:
/usr/bin/qemu-system-x86_64 –machine ?
Different types of machine chosen may end up using “tcg” as the QEMU emulation mode, instead of “kvm” (which is based on hardware virtualization and thus much faster.
So beware – if you find the emulation abnormally slow. Just “ps -ef” and ensure that
the “accel=kvm” is displayed instead of “accel=tcg”.
3. Create the guest image beforehand:
sudo qemu-img create -f qcow2 /var/lib/libvirt/images/guest.img 8192
4. The cdrom ISO is just the ISO downloaded.
Now issue “sudo virsh define guest.xml”. (what if you hate XML file, and are not sure how to setup the correct XML file for your version of libvirt, which might be different from here? See below).
g. After “sudo virsh list” to ensure that the “guest” KVM image is listed, now do “sudo virsh start guest” to start the guest booting up from CDROM.
h. After starting the VM running: “ps -ef” to list the process:
libvirt+ 4737 1 5 11:44 ? 00:02:04 qemu-system-x86_64 -enable-kvm -name guest -S -machine pc-i440fx-1.5-qemu-kvm,accel=kvm,usb=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid f5fe9230-6ef3-4eec-af54-65363a68f3ce -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/kvm1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/guest.img,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/home/user/ubuntu1404_x86_64/ubuntu-14.04-desktop-amd64.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=24,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=54:52:00:2a:58:0d,bus=pci.0,addr=0x3 -vnc 127.0.0.1:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
Notice how the complexity of the qemu command line is now solved by properly entering the correct parameter in the “guest.xml” file.
i. Finally, to connect to the running VM, you either can use “sudo virt-viewer <vm_name>” where <vm_name> is the name of the guest itself (which is “guest” in our case), or “sudo virt-viewer -c qemu:///system guest”.
j. And “virsh shutdown guest” to shutdown the running VM.
k. And “virsh destroy guest” to destroy the running VM.
How to setup virtio based on the libvirt infrastructure?
An architectural visualization of the virtio and its use in VM guest setup are as follows (http://www.openstack.cn/?p=580):
In summary, the diagram essentially means some of the I/O processing can pass directly from host into the guest without going through the QEMU event loop (which uses the /dev/kvm interface inside the host. As shown in another diagram below (http://slides.com/braoru/kvm/fullscreen#/):
By default the following should be enable before compiling the custom kernel:
CONFIG_VHOST_NET=m
CONFIG_VHOST_SCSI=m
CONFIG_VHOST_RING=m
CONFIG_VHOST=m
After compiling and rebooting into the new kernel:
a. modprobe -c |grep vhost to search for all the virtio related kernel module (which essentially is just vhost, vhost_scci, vhost_net).
b. modprobe vhost, modprobe vhost_scsi, modprobe vhost_net to load the kernel module.
c. Use “sudo virt-host-validate” to check that the kernel setup:
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking for device /dev/kvm : PASS
QEMU: Checking for device /dev/vhost-net : PASS
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS
And to setup the virtio direct I/O between the host and the VM guest, you can follow through here:
https://easyengine.io/tutorials/kvm/enable-virtio-existing-vms/
How to setup the guest if no XML is given:
This is made possble by the package “virtinst” installed earlier.
And the command is:
sudo virt-install –virt-type qemu –arch x86_64 –machine ‘pc-i440fx-2.0’ –debug –name guest –ram 1024 –disk path=/var/lib/libvirt/images/guest.qcow2 –cdrom /home/user/ubuntu1404_x86_64/ubuntu-14.04-desktop-amd64.iso –boot cdrom
Again the machine type must come from one of those listed in:
/usr/bin/qemu-system-x86_64 --machine ?
And now “sudo virsh dumpxml guest” to extract out the XML file.
References:
How libvirt work internally: https://libvirt.org/internals.html
http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
http://www.linux-kvm.org/images/4/41/2011-forum-virtio_net_whatsnew.pdf
http://www.ibm.com/developerworks/library/l-virtio/l-virtio-pdf.pdf
From here: http://dpdk.org/doc/guides/sample_app_ug/vhost.html
More complex setup for virtio:
Architectural internals of vrtio:
https://jipanyang.wordpress.com/2014/10/27/virtio-guest-side-implementation-pci-virtio-device-virtio-net-and-virtqueue/
Common setup problems:
https://www.digitalocean.com/community/questions/problem-with-iptables
https://forums.gentoo.org/viewtopic-t-1009770.html?sid=7822e8eefcdb28edcedf9db7526b7b1e
http://stackoverflow.com/questions/21983554/iptables-v1-4-14-cant-initialize-iptables-table-nat-table-does-not-exist-d
http://serverfault.com/questions/593263/iptables-nat-does-not-exist/593289
My Technical Blog 