IPv6 Impact on nspawn Container Startup

I measured nspawn container startup with various IP configuration options. Test setup: Debian bullseye/sid, systemd 246, /var/lib/machines is a directory, physical network is Ethernet to Google WiFi 1st gen, container image is minbase debootstrap built with packer-builder-nspawn.

Test sequence: start Wireshark capture on br0, send pings to container’s IP address every millisecond with sudo ping -i0.001, start the container with sudo machinectl start, match log entries in journalctl -oshort-precise to the timestamp of the first ping reply in the packet capture.

Total time from start to host0 carrier is consistently around 0.5s. This is a lot, and I didn’t dig deeper into what systemd is doing with all that time. Mounting /var/lib/machines to tmpfs made no difference. I couldn’t get veth to work with global IPv6 addresses, but with IPv4 using veth instead of bridge also made no difference. Typical time breakdown from one of the runs:

systemd starting container: 18ms
vb-dev link up:             30ms  (+12ms)
systemd started container:  240ms (+210ms)
container systemd started:  244ms (+4ms)
systemd-networkd started:   482ms (+238ms)
host0 link up:              484ms (+2ms)
br0 port forwarding state:  487ms (+3ms)
host0 gained carrier:       507ms (+20ms)

Total time from carrier to first ping reply varied a lot:

dhcpv6:         3832ms 3352ms 5309ms
static ipv6:    1557ms 1168ms 1228ms
dhcpv4:           17ms   23ms   28ms
static ipv4:       9ms    3ms    4ms

Even with static IPv6 configuration (Address=, Gateway=, DNS=), it still takes more than 1s before container begins to respond, bringing the total start to reply time to almost 2s. DHCPv6 adds more round-trips with similarly excessive timings and takes another 2-4s.

Enabling optimistic_dad sysctl (RFC 4429) on host and in the container made no difference.

ARO (Address Registration Option) from the NDP optimizations (RFC 6775) could in theory speed this up, but it isn’t implemented in Linux yet (according to Stefan Schmidt’s report at LPC 2019 IoT Microconference), and even when it is it might only apply to IPv6 over IEEE 802.15.4.

Compared to that, the extra 10-20ms that DHCPv4 adds to container startup looks quaint. There might be lower hanging fruit in systemd-nspawn that could reduce that 500ms start to carrier time to the point where faster IPv4 configuration would begin to make a difference.

The ridiculously long time it takes IPv6 stack to initialize makes me sad and wondering if there’s anything wrong with my setup. As it stands, it’s unsuitable for on-demand containers that get started to serve requests from interactive applications, and wasteful with containers that only need to run for a few seconds at a time as part of a low-frequency compute pipeline.