Benchmarks
2026-10-08 — QUIC demux: profile and batching experiment (rejected)
Host: development VM (8 vCPU EPYC 7642). 3 instances (CPUs 0-2), demux + controller on
CPU 3, 4 × scalws-h3-probe (4 connections × 4 streams each, /hello.txt) on CPUs 4-7.
bench/demux-profile.sh (perf + strace), bench/demux-ab.sh (alternating builds).
Profile (current demux)
- 91 000 inbound datagrams/s at 59 800 req/s (perf attached), 8.2 µs CPU per datagram.
- Receive is already batched:
quinn_udpusesrecvmmsg(with GRO); strace counted 1 553recvmmsgagainst 49 625sendmsg— about 32 datagrams per receive call. The assumption in earlier notes (“onerecvmsgper datagram”) was wrong. - The cost is the forward: one
sendmsgon the instance’sSOCK_SEQPACKETconnection per datagram, 89 % of syscall time. In the kernel profile the top items are per-message work of the Unix socket (skb allocation and accounting__memcg_slab_post_alloc_hook,memset,check_heap_object,unix_dgram_sendmsg, copies). User space is small (CID routing HMAC ≈ 2.6 %, the routing loop ≈ 1.9 %).
Experiment: one sendmmsg per instance per receive batch
Datagrams of a receive batch grouped by target connection and sent with one sendmmsg
each (plus per-batch counter updates and no per-datagram string allocation for the
reason counters). 6 alternating rounds:
| Build | datagrams/s (median) | demux µs/datagram (median, min–max) | req/s | p99 ms | errors |
|---|---|---|---|---|---|
| before | 103 624 | 7.69 (7.62–8.38) | 74 413 | 2.35 | 0 |
after (sendmmsg) | 99 796 | 7.99 (6.83–8.55) | 73 940 | 2.54 | 0 |
No improvement (within the spread). sendmmsg saves syscall entries, but the cost is the
per-message work of the Unix socket, which stays. Rejected and reverted.
What would move the ceiling (after 1.0, if one host needs more than ~110 000 inbound datagrams/s)
Several datagrams per Unix message (one skb per batch and instance, a length-prefixed frame per datagram), which changes the demux ↔ instance protocol on both sides; or shared-memory rings. Neither is a correctness issue today: the mixed soak served 15.2 M HTTP/3 requests with 0 errors and 3 instances reach 76 k req/s.