mmap & Memory-Mapped Files

A Kafka broker handling 200,000 consumer fetch requests per second struggles with offset lookup latency. Each fetch needs to resolve a byte offset in a log segment, and using read() syscalls for every lookup means 400,000 syscalls per second plus kernel-to-user copies on every request.

Kafka mmaps each partition index file (typically 10 MB) so the offset-to-position translation becomes a direct pointer dereference into the page cache. No syscall is needed. Because index files are small and accessed repeatedly, their pages stay hot in the TLB, making each lookup cost under 100 ns compared to 2-5 us for a read() path.

Use mmap for small, frequently accessed index files where random lookups dominate. At 200,000 fetches per second, eliminating the seek-plus-read path saves 400,000 syscalls per second and removes the kernel-to-user copy overhead entirely.

Nginx needs to serve the same 50 KB CSS file to 30,000 concurrent clients. Without page cache integration, each request triggers a disk read and a kernel-to-user copy, meaning 30,000 redundant disk reads and 1.5 GB of wasted memory bandwidth per second.

Nginx uses sendfile() which transfers data directly from the page cache to the socket, eliminating the user-space copy entirely. The file loads into the page cache once and stays there. Combined with open_file_cache that holds file descriptors and metadata for 1,000 hot assets, Nginx avoids repeated open() and stat() syscalls on every request.

Enable sendfile and open_file_cache for static asset serving. A single 4 GB server can saturate a 10 Gbps link serving static files because the hot working set lives entirely in page cache RAM, and zero-copy transfer from page cache to socket eliminates all user-space memory bandwidth overhead.

A PostgreSQL instance with 200 backend connections needs each process to access the same 16 GB buffer pool. Without shared memory, each backend would need its own copy, requiring 3.2 TB of RAM for 200 connections, which is obviously impossible.

MAP_SHARED maps the shared_buffers segment so every backend's page table entries point to the same physical pages. A dirty buffer written by one backend is instantly visible to all others with zero copying. The 200 processes share a single 16 GB physical copy of the buffer pool, and coherence is maintained by the kernel at the page cache level.

Combine MAP_SHARED with huge_pages=on to reduce the 16 GB region from 4 million TLB entries to just 8,192. With buffer cache hit rates above 99%, most queries never touch disk at all, and the shared mapping ensures zero memory duplication across all connections.

Loading a 10 GB RDB snapshot via sequential read() takes 3 minutes because the entire file must be copied through kernel buffers into user space before parsing can begin. During a restart after a crash, this 3-minute reload window means extended downtime for every client.

When rdbload uses mmap, the RDB file is mapped directly into the address space and parsed as an in-memory array. Demand paging fetches only the pages the parser touches, and the kernel readahead prefetches sequential pages in 128 KB batches. During BGSAVE, the forked child shares the parent's entire dataset through COW pages, allocating near-zero additional memory until the parent modifies keys.

Use mmap-based RDB loading to cut restore time from 3 minutes to 45 seconds for a 10 GB snapshot. The combination of demand paging and kernel readahead eliminates the upfront copy, and COW sharing during BGSAVE means the snapshot process runs without doubling memory consumption.

A Go program running 500,000 goroutines appears to need 4 TB of stack memory because each goroutine can grow to 1 GB. Monitoring tools report massive virtual memory usage, and operators worry the process is about to exhaust the machine.

The Go runtime calls mmap with MAP_NORESERVE to create virtual address reservations that cost zero physical pages. Each goroutine starts with a 2 KB stack, and physical frames are committed on demand at roughly 1 us per page fault as stacks grow. BoltDB uses the same technique, mmapping its entire database file and accessing B+ tree nodes as direct pointer dereferences at under 500 ns per random read.

Trust the virtual-to-physical distinction when monitoring Go services. The 4 TB of virtual reservations costs nothing until goroutines actually grow their stacks, and demand paging ensures physical RAM consumption matches actual usage rather than worst-case potential.

Elasticsearch needs to search across 500 GB of Lucene index segments on a 64 GB node. Loading them into the JVM heap would create massive GC pressure, constant stop-the-world pauses, and the heap cannot even hold that much data.

MappedByteBuffer maps each segment file directly into off-heap virtual memory, turning index lookups into pointer dereferences resolved by page faults. The kernel manages eviction through the page cache LRU, so hot segments stay resident while cold ones are reclaimed automatically. The GC never sees this data because it lives outside the managed heap entirely.

Use MappedByteBuffer for large read-heavy datasets that exceed heap capacity. On a 64 GB node indexing 500 GB, only 40-50 GB of the hottest segments remain in RAM, and query p99 latency stays under 50 ms because the kernel's page cache handles eviction more efficiently than any application-level cache could.

A user opens 40 Chrome tabs and each process loads 200 MB of shared libraries, fonts, and ICU data. Without memory sharing, this would consume 8 GB of physical RAM just for duplicated read-only resources, leaving little room for actual page content.

MAP_PRIVATE maps these read-only assets so all 40 processes share identical physical pages through copy-on-write. The 40 times 200 MB collapses to just 200 MB of physical RAM. For inter-process rendering, Chrome uses MAP_SHARED on /dev/shm so the renderer paints directly into a buffer the compositor reads at memory speed, eliminating kernel copies entirely.

Leverage MAP_PRIVATE for shared read-only resources and MAP_SHARED for zero-copy IPC between processes. At 60 fps and 8 MB per frame, the shared memory approach saves 480 MB/s of memory bandwidth that would otherwise go to pipe-based IPC between the renderer and compositor.

Running git log on a repository with a 4 GB packfile would take seconds if Git had to read() and seek through gigabytes of packed objects, issuing a syscall for each object lookup. Users would experience painful delays browsing even recent history.

Git mmaps the entire packfile into virtual memory, making every object lookup a pointer offset calculation followed by a direct memory dereference. Demand paging ensures only the pages Git actually touches consume physical RAM -- browsing recent commits might fault in just 20 MB of the 4 GB file. The pack index is also mmapped, so resolving a SHA to a byte offset is a binary search over an in-memory array.

Use mmap for large read-only data files that are accessed via random lookups. Git completes the log operation in milliseconds because demand paging loads only the 20 MB of relevant pages out of 4 GB, and each SHA-to-offset resolution costs roughly 200 ns as a direct memory dereference instead of a syscall.