TCP Tuning & Congestion Control

Cross-datacenter Kafka replication crawls at 200 Mbps on a 1 Gbps link with 50ms RTT. The link is not saturated, the brokers are not CPU-bound, and iperf3 between the same hosts shows the same bottleneck. A single random packet drop cuts throughput in half and recovery takes seconds.

The bandwidth-delay product is 1 Gbps times 50ms, which equals 6.25 MB, but the default tcp_rmem max is only 6 MB. The receiver cannot advertise a large enough window to keep the pipe full, so the sender stalls waiting for window updates. CUBIC congestion control makes it worse: it interprets any packet loss as congestion and halves the congestion window, even though random loss on WAN links is inevitable and not a congestion signal.

Increase socket buffers to at least the BDP so the receiver can advertise a window large enough to fill the pipe. Switch from CUBIC to BBR, which measures actual bandwidth and RTT independently and maintains close to line rate even with 1-2% packet loss. After tuning, replication throughput typically jumps from 200 Mbps to 800+ Mbps on the same link.

Nginx serving 50,000 responses per second shows 200ms of unexplained latency on small HTTP redirects. The server processes each redirect in microseconds, but the 200-byte response takes 200ms to reach the client. Large file downloads, oddly, perform fine.

Nagle's algorithm is the culprit. It holds small writes in the send buffer, waiting for either an ACK or enough data to fill a full TCP segment before sending. A 200-byte redirect is too small to trigger a send, so it sits in the buffer for up to 200ms. Large file transfers are unaffected because they immediately fill segments.

Nginx enables TCP_NODELAY by default so every response leaves the buffer immediately, eliminating the Nagle delay. During sendfile() transfers of large files, Nginx temporarily enables TCP_CORK to batch the HTTP headers and the first chunk of file data into one optimal-size segment, avoiding the tiny-packet overhead of sending headers separately. This combination of TCP_NODELAY for interactive responses and TCP_CORK for bulk transfers gives Nginx both low latency and high throughput from the same socket.

A web application issuing 2,000 small queries per second reports that every query takes 200ms at the client, even though PostgreSQL logs show 0.5ms execution time. The 400x discrepancy is consistent across all queries returning a few hundred bytes. Meanwhile, connection slots slowly leak as clients disappear without closing cleanly.

Nagle's algorithm is buffering each small result in the kernel send buffer, waiting for an ACK or enough additional data to fill a TCP segment. Since query results are small and no more data follows immediately, each result sits for up to 200ms before being sent. The leaking connection slots come from clients that crash or lose network without sending FIN, leaving backends blocked on recv() indefinitely against the default max_connections limit of 100.

PostgreSQL sets TCP_NODELAY on all client connections so results leave the buffer immediately, eliminating the Nagle penalty entirely. It also enables TCP keepalive probes with configurable intervals so that dead sessions are detected automatically and their connection slots reclaimed rather than held indefinitely.

Redis benchmarks show 200ms p99 latency on GET operations that the server processes in 0.1ms. The 2,000x discrepancy affects every command uniformly, regardless of key size or data type. At 100,000 operations per second, the cumulative wasted client wait time is staggering.

A 20-byte GET response is far too small to trigger an immediate TCP send. Without TCP_NODELAY, Nagle's algorithm holds it in the kernel send buffer, waiting for either an ACK from the previous segment or enough additional data to fill a full segment. Since Redis commands are independent and responses are tiny, neither condition is met quickly, and the response sits for up to 200ms.

Redis sets TCP_NODELAY unconditionally on every client connection, ensuring every response is pushed to the wire immediately. This is not a micro-optimization. It is the single setting that makes sub-millisecond p99 latency possible. At 100,000 operations per second, removing that 200ms Nagle delay saves 20,000 seconds of cumulative client wait time per second of real time.

A Node.js HTTP/2 server shows sluggish stream interleaving. Multiple streams on a single connection appear to stall for up to 200ms at a time, even though the server processes each frame in microseconds. Under peak connection bursts, new connections are silently dropped with no error visible in application logs.

HTTP/2 multiplexes many small HEADERS and DATA frames over a single connection. Nagle's algorithm batches these tiny frames together, waiting for ACKs before sending, which destroys the multiplexing benefit and adds up to 200ms of artificial delay per frame. The silent connection drops happen when the listen backlog overflows: the default somaxconn of 4096 means at most 4096 completed handshakes can queue before the single-threaded event loop calls accept().

Node.js enables TCP_NODELAY by default so each frame leaves the buffer immediately and streams interleave properly. For high-traffic servers, increase the listen backlog and net.core.somaxconn to match the peak connection burst rate, since connections beyond the limit are silently dropped with no application-level notification.