December 5, 2022

libreqos / cake

Cake (and earlier fq_codel) directs traffic on internet connections so that it works much better by reducing congestion by a lot. It’s really cool and pretty much just works. libreqos puts a gui on it.

I’m building this in Debian Bullseye, though it will certainly work on other things. This is a riff based on this. Get console and figure out which interfaces you want to use, don’t get this wrong or nothing will work!

The first thing I’ll do is get rid of the stupid newer names of interfaces like enp2s0, etc. by editing grub and making this line look like:

vi /etc/default/grub
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

then update grub and reboot:

update-grub2
apt install bridge-utils
reboot

When you reboot, you should now have names like eth0, eth1, etc. which will be tons easier to not screw up in this config.

libreqos cake

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:67:10:91:88 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:67:10:91:89 brd ff:ff:ff:ff:ff:ff
    altname enp2s0
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:e0:67:10:91:8a brd ff:ff:ff:ff:ff:ff
    altname enp3s0
    inet 192.168.10.5/24 brd 192.168.32.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::2e0:67ff:fe10:918a/64 scope link 
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:e0:67:10:91:8b brd ff:ff:ff:ff:ff:ff
    altname enp4s0

This shows my management interface is the third port from the right on my device, yours probably will be different.

make your /etc/network/interfaces look something like (note I assigned a bogus subnet for the bridge interface, which doesn’t do anything), change below to reflect your actual network, there’s 100% chance it won’t be like mine):

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
source /etc/network/interfaces.d/*
 
# The loopback network interface
auto lo br0
iface lo inet loopback
 
# The primary network interface
allow-hotplug eth2
iface eth2 inet static
	address 192.168.10.5/24
	gateway 192.168.10.1
	# dns-* options are implemented by the resolvconf package, if installed
	dns-nameservers 8.8.8.8
 
iface eth0 inet manual
	tc qdisc add dev eth0 root cake
 
iface eth1 inet manual
	tc qdisc add dev eth1 root cake
 
#Bridge setup
iface br0 inet static
	bridge_ports eth0 eth1
	#bridge_stp on
		address 192.168.3.150
		broadcast 192.168.3.255
		netmask 255.255.255.0
		gateway 192.168.3.1

How we modify some properties of interfaces:

apt install ethtool

now create a couple scripts to modify interfaces:

vi /usr/src/offloadinterfaces.sh
  #!/bin/sh
  ethtool --offload eth1 gso off tso off lro off sg off gro off
  ethtool --offload eth2 gso off tso off lro off sg off gro off
vi /etc/systemd/system/offloadOff.service
  [Unit]
  After=network.service
 
  [Service]
  ExecStart=/usr/src/offloadinterfaces.sh
 
  [Install]
  WantedBy=default.target
chmod 664 /etc/systemd/system/offloadOff.service
chmod 744 /usr/src/offloadinterfaces.sh
systemctl daemon-reload
systemctl enable offloadOff.service
Created symlink /etc/systemd/system/default.target.wants/offloadOff.service → /etc/systemd/system/offloadOff.service.
reboot

Now install libreqos:

apt install build-essential python3-pip clang gcc gcc-multilib llvm libelf-dev git
python3 -m pip install ipaddress schedule influxdb-client requests flask flask_restful flask_httpauth waitress psutil binpacking
cd /usr/src/
git clone https://github.com/rchac/LibreQoS.git
cd LibreQoS/v1.2
git submodule update --init
cd xdp-cpumap-tc/
git submodule update --init
cd src
make
make[1]: Entering directory '/usr/src/LibreQoS/v1.2/xdp-cpumap-tc/libbpf/src'
make[1]: pkg-config: No such file or directory
  MKDIR    staticobjs
  CC       staticobjs/bpf.o
  CC       staticobjs/btf.o
  CC       staticobjs/libbpf.o
  CC       staticobjs/libbpf_errno.o
  CC       staticobjs/netlink.o
  CC       staticobjs/nlattr.o
  CC       staticobjs/str_error.o
  CC       staticobjs/libbpf_probes.o
  CC       staticobjs/bpf_prog_linfo.o
  CC       staticobjs/btf_dump.o
...
cd ../../
git clone https://github.com/pollere/pping.git
cd pping
git clone https://github.com/mfontanini/libtins.git
cd libtins
sudo apt-get install libpcap-dev libssl-dev cmake
mkdir build
cd build
cmake ../ -DLIBTINS_BUILD_SHARED=0 -DLIBTINS_ENABLE_CXX11=1 \
 -DLIBTINS_ENABLE_ACK_TRACKER=0 -DLIBTINS_ENABLE_WPA2=0 \
 -DCMAKE_INSTALL_PREFIX=`dirname $PWD`
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
...
git submodule init && git submodule update
make
Scanning dependencies of target tins
[  1%] Building CXX object src/CMakeFiles/tins.dir/address_range.cpp.o
[  2%] Building CXX object src/CMakeFiles/tins.dir/arp.cpp.o
[  4%] Building CXX object src/CMakeFiles/tins.dir/bootp.cpp.o
[  5%] Building CXX object src/CMakeFiles/tins.dir/crypto.cpp.o
[  7%] Building CXX object src/CMakeFiles/tins.dir/detail/address_helpers.cpp.o
[  8%] Building CXX object src/CMakeFiles/tins.dir/detail/icmp_extension_helpers.cpp.o
[  9%] Building CXX object src/CMakeFiles/tins.dir/detail/pdu_helpers.cpp.o
...
make install
[100%] Built target tins
Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /usr/src/LibreQoS/v1.2/pping/libtins/lib/pkgconfig/libtins.pc
-- Installing: /usr/src/LibreQoS/v1.2/pping/libtins/lib/cmake/libtins/libtinsConfig.cmake
-- Installing: /usr/src/LibreQoS/v1.2/pping/libtins/lib/cmake/libtins/libtinsConfigVersion.cmake
-- Installing: /usr/src/LibreQoS/v1.2/pping/libtins/lib/cmake/libtins/libtinsTargets.cmake
-- Installing: /usr/src/LibreQoS/v1.2/pping/libtins/lib/cmake/libtins/libtinsTargets-relwithdebinfo.cmake
-- Installing: /usr/src/LibreQoS/v1.2/pping/libtins/lib/libtins.a
cd ../..
vi Makefile (change line 3 to)
  LIBTINS = /usr/src/LibreQoS/v1.2/pping/libtins
make
g++ -I/usr/src/LibreQoS/v1.2/pping/libtins/include -std=c++14 -g -O3 -Wall -o pping pping.cpp -L/usr/src/LibreQoS/v1.2/pping/libtins/lib -ltins -lpcap
pping.cpp: In function ‘tsInfo* getTStm(const string&)’:
pping.cpp:175:19: warning: catching polymorphic type ‘class std::out_of_range’ by value [-Wcatch-value=]
  175 |     } catch (std::out_of_range) {
      |                   ^~~~~~~~~~~~
apt install influxdb
reboot

to see if it’s working do:

watch -n .1 tc -s qdisc show dev eth0
Every 0.1s: tc -s qdisc show dev eth0                                  hostname: Sat Oct 29 16:26:51 2022
 
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 341732 bytes 1578 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0