Decision records

ADR-0027: Zero-copy static file bodies with sendfile(2)

• Status: Accepted (2026-10-06)

Context

Large static files were 0.42× nginx (docs/benchmarks/2026-10-05-static-profile.md): the profile is dominated by copying file data into user space (pread) and back into the socket (write), plus page zeroing; nginx uses sendfile. hyper owns the connection and only accepts body data as buffers, so the file descriptor never reaches the socket write.

Decision

A placeholder protocol between the static file body and the connection’s I/O layer:

• scalws_core::sendfile: a per-connection SendfileSlot (FIFO of file regions) and placeholder buffers — Bytes slices of one static all-zero array, recognisable by address.
• Plain-TCP HTTP/1.x connections on Linux get a slot (request extension) and an I/O wrapper. When the static handler streams a file larger than its in-memory threshold to such a connection, its body registers (file, offset, length) in the slot on first poll and then yields placeholders of exactly that total length.
• hyper’s HTTP/1 writer queues body buffers without copying when the I/O supports vectored writes (WriteStrategy::Queue; forced with writev(true)), so placeholders reach poll_write_vectored unchanged. The wrapper writes ordinary buffers with writev and replaces placeholder bytes, in order, by sendfile(socket, file, offset) from the front region. Byte counts and ordering are exactly what hyper expects, so Content-Length, keep-alive, pipelining and the write timeout keep working.
• Invariant: placeholder bytes are never sent. Placeholders without a matching region, or a file that shrank (sendfile returns 0), fail the write and close the connection.
• Consumers of body data must not see placeholders: when an application’s cache stores responses without caching headers (default_ttl, which covers static files), the cache removes the slot from the request; in every other case its tee refuses to store a body that contains a placeholder (the client still gets the right bytes). TLS, h2c and HTTP/3 connections never get a slot.
• Like nginx’s sendfile on without aio threads, a read of uncached data can block the worker briefly. posix_fadvise(WILLNEED) on registration was measured and dropped (≈11 % fewer requests per second on cached files, which is the common case); MSG_MORE on the response head was measured with no gain and dropped too.
• server.sendfile: false turns it off.

Consequences

• No unsafe, no mmap, no change to hyper.
• Depends on a hyper implementation detail (queue strategy passes buffers through); the integration tests compare bytes of large files, ranges, pipelined and keep-alive responses, so a change in hyper fails loudly rather than corrupting output.
• TLS stays on the copy path (kTLS would be a separate decision).