September 21, 2021

KVM guest traffic monitor

To get a guest VM on a KVM box to monitor traffic on a mirror/span port, you have to bridge it to an interface the guest can see. Also, you have to set up a mirror port somewhere on a switch/router that will copy packets as they traverse and send a copy to the interface on your server. So you’ll need an extra NIC on your server just to listen to traffic, then you have to configure it to do that like:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:1e:67:7c:84:2b brd ff:ff:ff:ff:ff:ff
3: ens2f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br1 state UP group default qlen 1000
    link/ether 00:1e:67:7c:84:2c brd ff:ff:ff:ff:ff:ff
4: ens2f2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:1e:67:7c:84:2d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::21e:67ff:fe7c:842d/64 scope link 
       valid_lft forever preferred_lft forever
5: ens2f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:1e:67:7c:84:2e brd ff:ff:ff:ff:ff:ff
7: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:1e:67:7c:84:2b brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/24 brd 10.1.10.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::21e:67ff:fe7c:842b/64 scope link 
       valid_lft forever preferred_lft forever
8: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:1e:67:7c:84:2c brd ff:ff:ff:ff:ff:ff
    inet 5.6.7.8/24 brd 97.90.110.255 scope global br1
       valid_lft forever preferred_lft forever
    inet6 fe80::21e:67ff:fe7c:842c/64 scope link 
       valid_lft forever preferred_lft forever

In this case, I already had the first interface ens2f0 bridged to br0 for subnet 1.2.3.4/24 (yours WILL not be 1.2.3.4), and my second bridge from ens2f1 bridged to br1, which maps to a second subnet of 5.6.7.8/24, in case other VM’s want to access the second network.

But then I added the third mirror port interface like:

ifconfig ens2f2 promisc up

Now I will check to see if traffic is being copied to this interface from my switch like:

apt install tcpdump
tcpdump -I ens2f2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens2f2, link-type EN10MB (Ethernet), capture size 262144 bytes
10:22:22.016793 IP ipv4-c223-sea001-ix.1.oca.nflxvideo.net.https > 097-.55649: Flags [.], seq 1951756359:1951759279, ack 676778573, win 2050, length 2920
...
bunches of gobbleygook
...
 
26 packets captured
248566 packets received by filter
248451 packets dropped by kernel

This means it’s seeing traffic, so it’s working 🙂

Now we have to port that traffic to a new bridge I’ll call br2. So I add the following lines to the end of my /etc/network/interfaces file:

auto ens2f2
iface ens2f2 inet manual
 
auto br2
iface br2 inet static
        address 0.0.0.0
        bridge_ports ens2f2
	#up /sbin/ifconfig br2 promisc on

If this worked (you’ll probably have to reboot), you should see packets when you do:

tcpdump -i br2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br2, link-type EN10MB (Ethernet), capture size 262144 bytes
11:25:29.418894 IP 192.168.25.110.60025 > map2.hwcdn.net.https: Flags [P.], seq 1734478228:1734478487, ack 2019253996, win 4096, options [nop,nop,TS val 366971595 ecr 252608166], length 259
...
gobblygook
...
19 packets captured
226350 packets received by filter
226241 packets dropped by kernel

Now you should be able to add br2 as an interface and your sensor should start seeing packets 🙂