June 15, 2024

kvm virtualization howto

KVM (not keyboard/video/mouse, used for viewing multiple screens and also called KVM) is a virtualization technology where you can run lots of different operating systems if you have the right cpu for it, like a Xeon or AMD one that supports virtualization. You have to have a remote client on your LAPTOP called virt-manager (I could only find a client for Linux/BSD, there may be some for Windows/Mac though?) to load the operating systems on your SERVER, which really sucks compared to xen, but xen is limited by the number/types of guest operating system

KVM install howto

This is on Debian, though others will apply will modification. Do this on your SERVER:

egrep '(vmx|svm)' --color=always /proc/cpuinfo

you should get a few lines of code output, this means your CPU will work with KVM, if you don’t stop, you can’t install it, you’ll have to use another Virtual Machine technology, like xen/virtualbox/VMWare, etc.

apt install --no-install-recommends qemu-system libvirt-clients libvirt-daemon-system
adduser whateveryournormalusernameis libvirt-qemu
#vi /etc/libvirt/libvirtd.conf
#  # unix_sock_rw_perms = "0770" <-- comment this line out and add one below:
#  unix_sock_rw_perms = "0777"
sysctl vm.swappiness=0
echo "vm.swappiness=0" >> /etc/sysctl.conf
/etc/init.d/libvirt-bin restart (/etc/init.d/libvirtd in jessie)
vi /etc/network/interfaces (change it to look like this, but with your IP info)
  # The loopback network interface
  auto lo
  iface lo inet loopback
  auto eth0
  iface eth0 inet manual
  auto br0
  iface br0 inet static
        bridge_ports eth0
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off
/etc/init.d/networking restart
see if you can still ping google.com
mkdir /root/.ssh
touch /root/.ssh/authorized_keys
vi /etc/ssh/sshd_config
  PermitRootLogin yes
  AllowUsers root@your.laptop.i.p anotheruser someotheruser (allow here everyone you want to be able to ssh in, you can not login with virt-manager on your laptop unless you login from this IP)
/etc/init.d/ssh reload

on your LAPTOP do:

sudo apt-get install virt-manager

now you have to set up an automatic connection to your KVM SERVER, so on your LAPTOP set it up with ssh/keys (as your REGULAR USER, not root) like:

ssh-keygen -t rsa -f ~/.ssh/id_rsa -C "yourusername@"
  > hit enter 2x on passphrase
cat ~/.ssh/id_rsa.pub | ssh root@server.ip.or.hostname 'cat - >> ~/.ssh/authorized_keys'
ssh roote@server.ip.or.hostname (you should not get a password prompt, it should just log you in)

now you have to set up a machine on your SERVER from your LAPTOP, but first you have to have an install .iso on the SERVER that you can do the install from, so download your .iso on the SERVER, and put it in /var/lib/libvirt/images, in this example we get a Debian .iso from a mirror by logging into your SERVER and doing:

cd /var/lib/libvirt/images
wget http://ftp.osuosl.org/pub/debian-cdimage/6.0.6/amd64/iso-cd/debian-6.0.6-amd64-CD-1.iso

Now, back on your LAPTOP, run virt-manager -> File -> Add connection:

Connect to remote host <--check checkbox
Method: ssh (should be by default)
Username: yourusernameonSERVER
Hostname: ip.or.hostname.ofSERVER

Now right-click on your new connection in the window, and do -> Connect. It shouldn’t prompt you for a password. Now click icon above connection to “create new virtual machine”, which it will do on the SERVER over your ssh connection now. Here’s an example machine I’m creating with the Debian .iso we just downloaded on the SERVER:

Name: whateveryouwanttocallyournewVM
Local install media checked <-- here it means local on your SERVER
Next button--->
Use ISO image <- click browse, select the image we just downloaded
OS type: whatever you are installing, same with Version
Forware --->
RAM/CPU for whatever you want
Forward --->
Change disk size to what you want
Forward --->
Advanced Options -> changet to "Specify shared device name"
Bridge name: br0
Architecture: select to match .iso
Finish --->
Allocates disk...

it should be done now, and you can click on the newly created machine and start it up and go through the install on the window on your LAPTOP.

KVM gotcha’s

if there is a not a /dev/kvm, it means your processor may have virtualization disabled in the bios. Look under advanced options for your processor, and enable virtualization. You will probably have to completely power off the server and then power it back on to enable it. after that, once it boots, run:

/etc/init.d/qemu-kvm restart (you should see something like:)
  Succesfully unloaded kvm module kvm_intel.
  Loading kvm module kvm_intel.

then double-check you have something listed in /dev/kvm

Take kvm snapshot

If you just make a manual zipped backup of your image file, it might drop in size from 50GB to 20GB, which is good, but you have to do it manually, and the KVM snapshot will be around 5K, which is a HUGE hard disk space saver.

If you want to use the native KVM utilities, you have to do it from the server, so login to your SERVER and get into the virsh shell:

  virsh # list --all
  Id    Name                           State
   2     debian1                        running
   4     centOS1                        running
   -     suse1                          shut off
   -     debian2                        shut off
virsh # snapshot-list --domain debian1
 Name                 Creation Time             State

This means you don’t have any snapshots for your debian1 machine. You can take one (in this case from a running machine) like:

virsh # snapshot-create-as --domain debian1 --name "20190716.debian1" --description "Snapshot before doing stuff after install"
Domain snapshot 20190716.debian1 created
virsh # snapshot-list --domain debian1
 Name                 Creation Time             State
 20190716.debian1 2019-07-16 10:52:00 -0700 running

If you get an error a error like:

error: unsupported configuration: internal snapshot for disk vda unsupported for storage type raw

You have to convert your whatever.img to a qcow2 image with a utility, then point your config.xml to it. the convert might take 10 minutes when it looks like nothing is happening, just wait for it, there’s no progress bar. You might want to use screen in case you get disconnected. The conversion process will not destroy the original file, you’ll still have that.

cd /var/lib/libvirt/images/
qemu-img convert -f raw -O qcow2 whatever.img whatever.qcow2
vi /etc/libvirt/qemu/whatever.xml
  <source file='/var/lib/libvirt/images/whatever.img'/>
  change to
  <source file='/var/lib/libvirt/images/whatever.qcow2'/> 
ls -lah
-rw-------  1 root         root          50G Jul 15 16:00 whatever.img
-rw-r--r--  1 root         root          37G Jul 15 16:21 whatever.qcow2
virsh /etc/libvirt/qemu/ whatever.xml
Domain portal defined from whatever.xml
virsh# start whatever
Domain whatever started

So the qcow2 is significantly smaller.

If you did it while the guest was shutdown you’ll see something like:

virsh # snapshot-list --domain debian1
 Name                 Creation Time             State
 20190716.debian1 2019-07-16 10:54:59 -0700 shutoff

Now to restore from a snapshot do:

virsh # shutdown debian1
Domain debian1 is being shutdown
virsh # list --all
 Id    Name                           State
 2     centOS2                        running
 -     debian1                        shut off
virsh # snapshot-revert --domain debian1 --snapshotname 20190716.debian1 --running
virsh # list
  Id    Name                           State
   2     debian1                        running
   4     centOS1                        running
   -     suse1                          shut off
   -     debian2                        shut off

Move a kvm guest to another server

You can shut down the vm, scp it to another server, then tell kvm on the new server about it, and you should be able to use it on your new server. This worked moving a guest from a Debian Squeeze server running qemu-kvm to another server running Debian Wheezy qemu-kvm on the same subnet so I didn’t have to change the guest IP configs. Here, the source server is server1 and destination server is server2. Make sure you shut down the guest on server1 before moving it, I’ll assume you’ve already done that.

server1#: su
cd /var/lib/libvirt/images/
list --all
shutdown whateverdomainyouremoving
undefine whateverdomainyouremoving
  Domain whateverdomainyouremoving has been undefined
scp guestmachinename.img ser.ver.2.ip:/var/lib/libvirt/images/
cd /etc/libvirt/qemu/
scp guestmachinename.xml ser.ver.2.ip:/etc/libvirt/qemu/
ssh user@ser.ver.2.ip
cd /etc/libvirt/qemu/
virsh define guestmachinename.xml
list --all (should now show guestmachinename)
start guestmachinename

you should now see your new machine in virt-manager on your laptop, or in virsh if you use that from the command line, and be able to login and use it on the new box.

machine won’t start after migrating

if you get an error like:

error: Failed to start domain videobox
error: internal error Process exited while reading console log output: Supported machines are:
pc         Standard PC (alias of pc-0.12)
pc-0.12    Standard PC (default)
pc-0.11    Standard PC, qemu 0.11
pc-0.10    Standard PC, qemu 0.10
isapc      ISA-only PC

it means you have to change the machine type in your migrated domain xml, so change it to something on the list in the error message like:

vi /etc/libvirt/qemu/yourdomain.xml
  <type arch='x86_64' machine='pc-0.12'>hvm</type>
virsh define yourdomain.xml
virsh start yourdomain

Resize KVM LVM partition

You can resize a normal partition using a recent version of the KVM/qemu tools, but resizing the LVM/VG/PV/partition inside that is a bit more tricky. You should back everything up before you start, or even better, just copy the image to a new one in case everything gets screwed up.

cd /var/lib/libvirt/images
> list --all

That should show all your machines, running or not, so shutdown the one you want by either logging in and shutting it down, or doing:

> shutdown whateverdomain
  domain is shutting down 
(wait a bit and then see if it really is by doing:)
> list --all
  whateverdomain     not running

now copy your image first to a backup like:

cp whateverdomain.img whateverdomain.img.bak

Now you can resize your image. In our example we resize a 20G (or whatever) to increase it to 30G (or whatever your current size + 10 GB like:

qemu-img resize vmdisk.img +10G
ls -lah *

you should see that your image “grew”, this takes a surprisingly short amount of time, like a few seconds on a powerful Xeon machine.

Now here comes the tricky part where you have to boot to a Gparted liveCD FROM INSIDE YOUR VM YOU JUST RESIZED. To do this, you have to use a virt-manager client on a Linux laptop somewhere and connect to your server running KVM for the next steps.

1. Login to virt-manager, connect to your KVM server host and then double-click on the icon for the virtual machine you’re resizing, BUT DON’T START IT ACCIDENTALLY, it has to be shut down still

2. Go to Show Details > Boot Options, check the box that says “IDE cdrom”, then use the arrow keys to move that entry to the top of the list there. Now go to the “IDE Cdrom” button and then browse and select the Gparted live .iso, which also has to be located in /var/lib/libvirt/images on your host KVM server to be visible to the browse function, so scp it to your server or use wget to download it directly to the server. Also chown libvirt-qemu.libvirt-qemu gparted-whatever.iso, so your virt-manager will see it. Sometimes you have to exit virt-manager and go back and redo the browse if it doesn’t pick it up.

3. Now when you start that machine up, you should see the boot screen for Gparted, not whatever OS is running normally on that machine. If you don’t stop and troubleshoot, you don’t want to screw up your original OS, so be sure of this.

4. On the boot screen just hit enter in most cases, that will take you to a GUI that has icons for running stuff. You actually need a terminal, but you just double-click on the terminal icon on the Gparted desktop.

5. Now you have to figure out which disk is which, otherwise if you select the wrong disk/volume you stand a fair chance of losing your data on that disk, which is never good. So take your time here and double-check you’re working on the right thing. You either have a MBR or a UEFI, and basically if you don’t know what the difference is, assume you have a normal MBR, which means you’d use fdisk (rather than gdisk/parted), and run:


build VM from command line

you can use virt-install to build a new VM from the command line like:

virt-install \
        --name whatevername \
        --accelerate \
        --virt-type kvm \
        --vnc \
        --description "Some description you want" \
        --memory 1024 \
        --disk path=/var/lib/libvirt/images/whatever.img,bus=virtio,size=100 \
        --vcpus 2 \
        --os-type Linux \
        --os-variant debianwheezy \
        --cdrom /var/lib/libvirt/images/debian-8.5.0-amd64-DVD-1.iso \
        --network bridge=br0 \
        --console pty,target_type=serial
Starting install...
Allocating 'whatever.img'                                                                              | 100 GB     00:00     
Creating domain...                                                                                     |    0 B     00:00     
Unable to init server: Could not connect: Connection refused
Cannot open display: 
Run 'virt-viewer --help' to see a full list of available command line options
Domain installation still in progress. You can reconnect to 
the console to complete the installation process.

Now you login with your virt-manager on a remote laptop and finish the install.

KVM second interface eth1

You can add another network interface, typically eth1 to your KVM host, and then let guests use it too. First, set up the second bridge, “br1”, just add it below your other bridge at the end of the file like:

auto eth1
iface eth1 inet manual
auto br1
iface br1 inet static
        address sec.on.dary.ip
        gateway sec.on.dary.gateway
        bridge_ports eth1
        bridge_fd 9
        bridge_hello 2
        bridge_maxage 12
        bridge_stp off

Now you can either manually reboot the server, restart networking (and hope it works, sometimes doesn’t on Debian), or just bring up your second interface like:

ifconfig br1 sec.on.dary.ip netmask up

Now boot your guest VM, then once you do, make your GUEST VM /etc/network/interfaces (change to match your specific environment) look something like:

# The primary network interface
allow-hotplug eth1
iface eth1 inet static
        address pub.lic.static.ip
        gateway pub.lic.static.gateway
        post-up route del default gw whate.ver.private.ip eth0
        post-up route add default gw pub.lic.ip.gateway eth1
# The primary network interface
allow-hotplug eth0
iface eth0 inet static
        address whatever.private.i.p
        gateway private.ip.gate.way

If you don’t do the post-up delete/add, it will try to route stuff out your private IP/subnet, instead of your public one.

References: http://wiki.debian.org/KVM

bridging for IDS/promiscuous monitoring

First, set up the second interface so it will bridge to eth1:

auto eth1
iface eth1 inet manual
#Alienvault OSIM Interface
auto br1
    iface br1 inet manual
    bridge_ports eth1
    bridge_stp off


ifup eth1

then see if it’s capturing packets like:

tcpdump -i br1

you should get a ton of stuff like:

14:46:53.507328 IP > ipv4_1.cxl0.c154.sea001.ix.nflxvideo.net.https: Flags [.], ack 2897, win 5611, options [nop,nop,TS val 3160018074 ecr 2918482309,nop,nop,sack 1 {28961:46337}], length 0
 14:46:53.507402 IP ipv4_1.cxl0.c154.sea001.ix.nflxvideo.net.https > Flags [.], seq 56473:57921, ack 0, win 2050, options [nop,nop,TS val 2918482612 ecr 3160018052], length 1448

like pages and pages flowing past very quickly, this is good. Now add your bridge like:

brctl setageing br1 0
brctl setfd br1 0

Now fire up your VM and add a second NIC of > Specify shared device name > br1. You’ll probably have to reboot the VM, but then login to the VM and you should see a second NIC of eth1, which is bridged to your mirror interface and can start looking at packets. I still haven’t figured out how to get vswitch to copy mirror packets to multiple interfaces, in case you want to have multiple sensors look at the same data, so that’s my next issue. To make your brctl config persist a reboot, do:

cd /etc/network/if-up.d
touch br1-mirror
chmod +x br1-mirror
vi br1-mirror
  if [ "$IFACE" = br1 ]; then
  brctl setageing br1 0
  brctl setfd br3 0

Mad props to the help here and also also this guy, very helpful all http://www.ryanhallman.com/kvm-configure-mirrored-ports-traffic-to-be-visible-in-guest-snort/ hope this all helps someone else, tough to figure out how to bolt all the pieces together.

Also, bridge tutorial: