APU2 and Debian Buster as a home router

This (terse!) guide shows:

how to get a working serial console with Debian on the APU2
setting up basic network configuration (including full 1500 MTU on PPPoE)
setting up firewalling using nftables
setting up DHCP, DNS, NTP
setting up QoS using CAKE

This guide doesn't cover WiFi in any form - for WiFi I recommend a separate AP that you can position for best coverage.

I don't use this set up any more as the APU2 isn't fast enough for gigabit connections with PPPoE - I moved to a Mikrotik RB5009, also purchased from linitx.com.

The router

The PC Engines APU2 is a single-board x86 computer with a reasonable 64-bit quad core CPU, up to 4GB of RAM, storage via mSATA or SD card, and up to 4 Intel NICs.

It draws <10W in normal operation - which allows it to be passively cooled, and it is fairly unusual in x86 platforms in that it doesn't have any display out - only a serial terminal.

I have the apu2c4 model - which has 4GB of ECC DDR3 and 3x Intel i210AT NICs. I've fitted a 250G mSATA SSD for persistent storage.

I got mine from linitx.com - a good UK distributor of PC Engines hardware.

On a 330/30 service, the APU2 works well. On faster services that require PPPoE it can't keep up - even with the newest firmware which allows frequency boosts to ~1.4GHz. About 400mbit was attainable for me out of the box, with 700mbit after some sysctl tweaks.

The network setup

I use Andrews & Arnold as my internet provider. They're a small ISP, but they provide a high quality service with static IPv6 and easy access to static blocks of IPv4. This guide should be reasonably useful for most UK internet connections, however.

I have a FTTP service using PPPoE that's presented as a gigabit ethernet port - but I've also used this router with a FTTC service using a separate FTTC modem with exactly the same setup. If your modem doesn't support RFC4638 (mini jumbo frames for PPPoE), then you'll need to adjust the MTU appropriately. A good modem that does is the Openreach branded Huawei HG612.

My network is connected as follows:

enp1s0 is connected to the FTTP ONT or FTTC modem
enp2s0 is connected to my main LAN - in this example, this is 192.168.1.0/24 and 2001:8b0:db8::/64
enp3s0 is used for a live IP range - in this example, I use 192.0.2.0/29 and 2001:8b0:db8:1::/64

Installing Debian

Grab the netinstall image from https://www.debian.org/CD/netinst/. Make sure it's the amd64 image. Once downloaded, write it to a flash drive. I use dd for this:
```
# replace /dev/sdX with the device node for your USB drive
dd if=debian-10.6.0-amd64-netinst.iso bs=1M of=/dev/sdX
```
Connect up your serial console, and start your terminal emulator using 115200 baud, 8 data bits, no parity, one start bit and one stop bit.
Apply power to the APU2, and use the serial console to boot from USB - press F10 to enter the boot menu, then pick your flash drive from the list.
When booting from the flash drive, you'll need to provide some extra boot parameters. Add console=ttyS0,115200u8 to the command line, and press enter to boot.
Follow the installer through as normal.

Initial post-install configuration

I set /etc/apt/sources.list to the following:

deb http://deb.debian.org/debian/ buster main
deb-src http://deb.debian.org/debian/ buster main

deb http://deb.debian.org/debian-security/ buster/updates main
deb-src http://deb.debian.org/debian-security/ buster/updates main

deb http://deb.debian.org/debian/ buster-updates main
deb-src http://deb.debian.org/debian/ buster-updates main

deb http://deb.debian.org/debian/ buster-backports main
deb-src http://deb.debian.org/debian/ buster-backports main

And then run:

apt update

Install packages we'll need for this to work:

# a kernel from buster-backports so we have up to date cake, nftables, and wireguard modules available
apt install -t buster-backports linux-image-amd64

# flashrom is used for firmware updates for the apu2
apt install flashrom

# a variety of useful tools for this setup
apt install chrony curl ethtool irqbalance isc-dhcp-server jq mtr-tiny nftables ppp pppoe radvd unbound

# if you intend to use bridges or vlans (optional)
apt install bridge-utils vlan

Reboot into the new kernel now.

Networking setup

Set up basic networking - set /etc/network/interfaces to:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# WAN
auto enp1s0
iface enp1s0 inet manual
    description wan
    # this allows us to do full 1500 byte frames inside the PPPoE tunnel
    mtu 1508
    # this stops NIC hangs under some 5.x kernels
    up ethtool -K enp1s0 tx off rx off gso off gro off tso off

auto pppoe-aaisp
iface pppoe-aaisp inet ppp
    provider aaisp
    # ensure the underlying ethernet interface is up and give it some time for everything to settle
    pre-up ifup enp1s0
    pre-up sleep 5

# LAN
auto enp2s0
iface enp2s0 inet static
    description lan
    address 192.168.1.1/24
    mtu 1500
    # this does some basic fair queuing
    up tc qdisc add dev enp2s0 root sfq perturb 10
    # this stops NIC hangs under some 5.x kernels
    up ethtool -K enp2s0 tx off rx off gso off gro off tso off

iface enp2s0 inet6 static
    address 2001:8b0:db8::1/64
    mtu 1500

# LIVE
auto enp3s0
iface enp3s0 inet static
    description live
    address 192.0.2.1/29
    mtu 1500
    # this does some basic fair queuing
    up tc qdisc add dev enp3s0 root sfq perturb 10
    # this stops NIC hangs under some 5.x kernels
    up ethtool -K enp3s0 tx off rx off gso off gro off tso off

iface enp3s0 inet6 static
    description live
    address 2001:8b0:db8:1::1/64
    mtu 1500

Set /etc/ppp/peers/aaisp to:

user example@a.1
plugin rp-pppoe.so enp1s0
noipdefault
defaultroute
replacedefaultroute
hide-password
lcp-echo-interval 1
lcp-echo-failure 10
noauth
persist
maxfail 0
mtu 1500
noaccomp
default-asyncmap
+ipv6
ipv6cp-accept-local
ifname pppoe-aaisp

logfile /var/log/ppp-aaisp.log

This sets up PPP for connecting to A&A - it sets up a connection that will:

set the IPv4 default route
does a LCP ping every second and reconnects after 10 failures
uses a MTU of 1500 bytes
enables IPv6
renames the PPP interface to pppoe-aaisp

Set /etc/ppp/chap-secrets to:

# Secrets for authentication using CHAP
# client	server	secret			IP addresses
example@a.1	*	ExampleSecret

Create /etc/ppp/ipv6-ip.d/0000-defaultroute with these contents:

#!/bin/bash

# add a default v6 route
ip -6 route add default dev $1

And make it executable:

chmod +x /etc/ppp/ipv6-ip.d/0000-defaultroute

Create /etc/sysctl.d/router.conf with these contents:

# Enable reverse-path filter (source address spoofing protection)
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1

# Enable syncookies
net.ipv4.tcp_syncookies = 1

# Enable IPv4 forwarding
net.ipv4.ip_forward = 1

# Enable IPv6 forwarding
net.ipv6.conf.all.forwarding = 1

# Disable ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0

Set /etc/resolv.conf to:
```
nameserver 127.0.0.1
```

Set /etc/nftables.conf to:

#!/usr/sbin/nft -f

flush ruleset

table inet firewall {
    chain input {
        type filter hook input priority 0; policy drop;

        # allow localhost
        iif "lo" accept

        # allow from wan interface
        iifname "enp1s0" accept

        # allow return traffic
        ct state established,related accept

        # allow ICMP
        meta l4proto { icmp, ipv6-icmp } accept

        # allow DHCP from LAN
        iifname { "enp2s0" } udp dport 67 accept

        # allow DNS from LAN and live
        iifname { "enp2s0", "enp3s0" } tcp dport 53 accept
        iifname { "enp2s0", "enp3s0" } udp dport 53 accept

        # allow NTP
        iifname { "enp2s0", "enp3s0" } udp dport 123 accept

        reject with icmpx type admin-prohibited
    }

    chain forward-internet-to-lan {
        # allow ICMP
        meta l4proto { icmp, ipv6-icmp } accept

        # allow return traffic
        ct state established,related accept

        reject with icmpx type admin-prohibited
    }

    chain forward-internet-to-live {
        # allow ICMP
        meta l4proto { icmp, ipv6-icmp } accept

        # allow all traffic
        accept
    }

    chain forward {
        type filter hook forward priority 0; policy drop;

        # allow ICMP
        meta l4proto { icmp, ipv6-icmp } accept

        iifname "pppoe-aaisp" oifname "enp2s0" jump forward-internet-to-lan
        iifname "pppoe-aaisp" oifname "enp3s0" jump forward-internet-to-live

        # allow from lan to internet, live
        iifname "enp2s0" oifname { "pppoe-aaisp", "enp3s0" } accept

        # allow return traffic
        ct state established,related accept

        reject with icmpx type admin-prohibited
    }
}

table ip nat {
    chain postrouting {
        type nat hook postrouting priority 0;

        ip saddr 192.168.0.0/16 oifname "pppoe-aaisp" masquerade fully-random
    }
}

Enable nftables:
```
systemctl enable nftables
```

The easiest way to apply this all is to reboot - once you've rebooted you should have basic routing (but no DHCP)!

Setting up network services

DNS

Create /etc/unbound/unbound.conf.d/local.conf with these contents:

server:
    interface: 0.0.0.0
    interface: ::
    interface-automatic: yes
    prefer-ip6: yes
    access-control: 192.168.0.0/16 allow
    access-control: 192.0.2.0/29 allow
    access-control: 2001:8b0:db8::/48 allow

Restart unbound:
```
systemctl restart unbound
```

NTP

Edit /etc/chrony/chrony.conf and add this to the end:

allow 192.168.0.0/16
allow 192.0.2.0/29
allow 2001:8b0:db8::/48

Restart chrony:
```
systemctl restart chrony
```

SLAAC for IPv6

Set /etc/radvd.conf to:

interface enp2s0
{
        AdvSendAdvert on;
        MinRtrAdvInterval 3;
        MinRtrAdvInterval 10;

        AdvManagedFlag off;
        AdvOtherConfigFlag off;

        prefix 2001:8b0:db8::/64
        {
                AdvOnLink on;
                AdvAutonomous on;
        };

        RDNSS 2001:8b0:db8::1 {
        };

        DNSSL your.domain.here {
        };
};

This enables distributing DNS server and DNS search list via SLAAC, and allows devices to configure their own IP addresses from the advertised prefixes.

Restart radvd:
```
systemctl restart radvd
```

DHCP

Edit /etc/dhcp/dhcpd.conf and add this to the end:

# DNS settings
option domain-name "your.domain.here";
option domain-name-servers 192.168.1.1;

# The length of the DHCP lease - 10 minutes
default-lease-time 600;
max-lease-time 7200;

# This DHCP server is authoritative for the local network
authoritative;

subnet 192.168.1.0 netmask 255.255.255.0 {
    option routers 192.168.1.1;
    option domain-name-servers 192.168.1.1;
    option ntp-servers 192.168.1.1;
    range 192.168.1.100 192.168.1.199;
}

Restart the DHCP server:
```
systemctl restart isc-dhcp-server
```

QoS

Setting up CAKE on both ingress and egress provides us with a responsive internet connection that provides fair bandwidth allocation where possible - not allowing any one device to abuse the connection and maintaining low-latency conditions for interactive services like SSH.

In addition, A&A provide an API that can be used to get your current line rate. This is very useful for setting up CAKE - especially if you use FTTC!

We use an ifb interface to bring ingress traffic into so we can apply CAKE to it.

Create /usr/local/sbin/qos-setup with the contents:

#!/bin/bash

# Your A&A control panel credentials
LOGIN="example@a"
PASSWORD="ExampleControlPanelPassword"

# Your line ID - this ID is the one at the end of the URL when you view circuit details.
# for example: https://control.aa.net.uk/editline.cgi?ID=12345 has ID 12345
SERVICE="12345"

# 8 bytes for pppoe + 4 bytes for BT VLAN. This seems to be correct for FTTP, experimentally.
# For FTTC you'll want to tweak this!
OVERHEAD="12"

# load ifb
modprobe ifb

# clear existing tc qdiscs
tc qdisc del dev pppoe-aaisp root
tc qdisc del dev pppoe-aaisp handle ffff: ingress
tc qdisc del dev ifb0 root
tc qdisc del dev ifb0 ingress
ip link set dev ifb0 down

echo "fetching service info"
INFO="$(curl -s "https://chaos2.aa.net.uk/broadband/info?control_login=${LOGIN}&control_password=${PASSWORD}&service=${SERVICE}" | jq '.info[0]')"
# these rates are from A&A's PoV, so tx = them transmitting to us, rx = them receiving from us
INGRESS_RATE=$(echo "$INFO" | jq .tx_rate_adjusted -r)
EGRESS_RATE=$(echo "$INFO" | jq .rx_rate -r)
echo "ingress $INGRESS_RATE egress $EGRESS_RATE"

# bring up ifb0
ip link set dev ifb0 up

# create ingress filter
echo "create ingress filter"
tc qdisc add dev pppoe-aaisp handle ffff: ingress

# forward all ingress traffic to ifb0
echo "forward all ingress traffic to ifb0"
tc filter add dev pppoe-aaisp parent ffff: protocol all u32 match u32 0 0 action mirred egress redirect dev ifb0

# set up ingress cake
echo "ingress cake"
tc qdisc add dev ifb0 root cake bandwidth $INGRESS_RATE nat overhead ${OVERHEAD} ingress dual-dsthost diffserv4 regional

# egress
echo "egress cake"
tc qdisc add dev pppoe-aaisp root cake bandwidth $EGRESS_RATE nat overhead ${OVERHEAD} dual-srchost diffserv4 regional

And make it executable:

chmod +x /usr/local/sbin/qos-setup

Create /etc/ppp/ipv6-ip.d/0001-qos with these contents:

#!/bin/bash

# initialise QoS
/usr/local/sbin/qos-setup

And make it executable:

chmod +x /etc/ppp/ipv6-ip.d/0001-qos

Execute it once to test:
```
/usr/local/sbin/qos-setup
```

Conclusion

At this point, you should have a nicely working Linux-based router, with working IPv4 and IPv6, DHCP for IPv4, SLAAC for IPv6, DNS, NTP, and QoS!