Inotify & fanotify: File System Events
Mental Model
A doorbell instead of checking the front porch every five minutes. The delivery driver rings it the instant a package lands. No wasted trips, no missed deliveries. inotify works the same way for files -- register the paths that matter and the kernel signals the moment something is created, changed, deleted, or renamed. Each doorbell costs a bit of memory, and the building only has so many wiring slots.
The Problem
A Node.js project with 150,000 files in node_modules cannot start: "ENOSPC: System limit for number of file watchers reached." Disk is fine -- the error is about inotify watches, not space. The default limit sits at 8,192, and one watch per directory eats that instantly. On a separate front, a Kubernetes ConfigMap update goes unnoticed because the application watches the file itself -- but Kubernetes swaps ConfigMaps via symlink replacement, leaving the watched inode untouched.
Architecture
A file is saved. Within milliseconds, webpack recompiles, the browser refreshes, and the changes appear. No click needed. Nobody polled anything.
The kernel saw the write happen -- because it is the one doing the writing -- and it tapped the dev server on the shoulder: "hey, this file changed."
That tap is inotify. And it is behind more things than most developers realize.
What Actually Happens
Linux provides two file system notification mechanisms, both built on the kernel's internal fsnotify framework.
inotify (Linux 2.6.13+) is the per-file/per-directory watcher. Creating an inotify instance with inotify_init1() returns a file descriptor. Watches are added with inotify_add_watch(fd, path, mask) -- specifying which path and which events matter (create, modify, delete, move, etc.). Each watch is tied to an inode, not a path name.
When a VFS operation triggers a matching event, the kernel queues an inotify_event on the fd. A read() call retrieves it. For directory watches, the event includes the filename of the affected child, identifying which file changed.
Here is the key limitation: inotify does not recurse. Watching /etc only reports events in /etc itself, not in /etc/nginx/conf.d/. To monitor a directory tree, the code must walk it and add a watch to every subdirectory. Then watch for IN_CREATE with IN_ISDIR to catch new subdirectories and add watches dynamically.
This is exactly what chokidar (used by webpack) does. And it is why large node_modules trees -- with 50,000+ directories -- exhaust the default watch limit.
fanotify (Linux 2.6.37+) takes a completely different approach. Instead of per-file watches, it can mark an entire mount point or filesystem with a single call (FAN_MARK_MOUNT or FAN_MARK_FILESYSTEM). And it delivers open file descriptors in events, not just names, which eliminates TOCTOU race conditions.
But fanotify's killer feature is permission events. When initialized with FAN_CLASS_CONTENT, the kernel blocks the file access entirely until the handler writes FAN_ALLOW or FAN_DENY back. The accessing process is frozen until a decision is made. This is how on-access antivirus scanning works -- ClamAV's clamonacc intercepts every open(), scans the file, and only then lets the process continue.
Under the Hood
fsnotify is the shared engine. Both inotify and fanotify are frontends to the kernel's fsnotify subsystem. fsnotify hooks into VFS functions -- vfs_create(), vfs_write(), vfs_unlink(), vfs_rename(). When a VFS operation completes, it calls fsnotify_*() helpers that iterate over all notification groups with marks on the affected inode or mount, and queue events accordingly.
Event coalescing prevents flooding. If a process writes to a file 1000 times before the application reads the inotify fd, only one IN_MODIFY event comes back, not 1000. The kernel merges consecutive identical events (same wd, same mask, same name). Distinct events are never reordered -- IN_CREATE followed by IN_MODIFY always arrives in that order -- but exact modification counts cannot be derived from inotify events.
The atomic replacement trap catches almost everyone. Editors like vim, and orchestrators like Kubernetes, do not modify config files in place. They write to a temp file and then rename() it over the target. This fires IN_MOVED_TO on the directory, NOT IN_MODIFY on the file. An inotify watch on the file itself follows the old inode -- which is now the temp file nobody cares about. The watch becomes stale and every future update is missed.
The correct pattern: watch the containing directory for IN_MOVED_TO and IN_CREATE.
Integration with epoll makes inotify practical for event loops. The inotify fd is a regular file descriptor -- adding it to an epoll instance with epoll_ctl() means epoll_wait() wakes the loop when filesystem events arrive. Zero CPU burned while waiting.
Common Questions
How does Kubernetes ConfigMap updating work with inotify?
Kubernetes uses an atomic symlink swap. The ConfigMap volume is a symlink chain: /etc/config -> ..data -> ..2024_01_15/. On update, kubelet writes new files to a new timestamped directory, then atomically updates the ..data symlink. Applications watching a file directly via inotify will miss the update because the inode did not change -- only the symlink target did. Watch the directory for IN_MOVED_TO, or watch the ..data symlink for IN_DELETE_SELF and re-read.
Why does webpack need fs.inotify.max_user_watches increased?
A typical Node.js project with node_modules can contain 50,000+ directories. chokidar adds an individual watch per subdirectory (because inotify cannot recurse). The default max_user_watches limit (8192 on some distributions) is quickly exhausted. The ENOSPC error from inotify_add_watch() is misleading -- it has nothing to do with disk space. The fix: sysctl fs.inotify.max_user_watches=524288, which costs about 540 bytes per watch in kernel memory (~270 MB for the full 524,288).
What is the TOCTOU race and how does fanotify fix it?
inotify delivers a filename in the event, but by the time the application processes it, the file may have been renamed, deleted, or replaced. Time-of-check-time-of-use. fanotify solves this by delivering an open file descriptor (event->fd) to the actual file, opened by the kernel at event generation time. The fd always refers to the correct file regardless of subsequent renames.
How to build a recursive directory watcher?
Walk the tree with nftw(), adding an inotify watch to every directory. Watch for IN_CREATE with IN_ISDIR to catch new subdirectories and add watches. Handle IN_MOVED_FROM/IN_MOVED_TO for directory renames by removing the old watch and adding a new one. When a directory is deleted, the kernel sends IN_IGNORED as the watch is automatically removed. This is exactly what inotifywait -r and chokidar implement internally.
How Technologies Use This
A containerized application using inotify to watch its ConfigMap-mounted config file for IN_MODIFY never detects updates. Kubernetes pushes new configuration, but the app keeps running with stale settings, causing silent misconfigurations in production.
The hidden cause is that Kubernetes does not modify config files in place. It writes new content to a timestamped directory and atomically swaps a symlink. The original inode never changes, so IN_MODIFY never fires. The inotify watch follows the old inode, not the path name.
Watch the parent directory for IN_MOVED_TO instead of watching the file for IN_MODIFY. This fires instantly when Kubernetes swaps the ..data symlink, giving sub-millisecond detection at zero CPU cost while idle, compared to a polling loop that burns 1-2% CPU and still misses rapid updates.
A Kafka broker's TLS certificate expires and the broker needs the renewed keystore. A restart would trigger partition leader elections, ISR shrinkage, and minutes of consumer lag recovery across the cluster.
Kafka uses Java WatchService, backed by inotify on Linux, to monitor the keystore and config files. When the certificate is renewed and the file changes, inotify fires an event within milliseconds. The broker reloads its SSL context and begins serving with the new certificate without ever going offline.
Configure ssl.keystore.location to point to the watched path and ensure the certificate renewal pipeline writes to that location. This gives zero-downtime certificate rotation with zero polling overhead. Kafka Connect uses the same mechanism to detect connector config changes and reconfigure tasks automatically.
Nginx is running with a stale configuration because the cron job that checks for changes only runs every 30 seconds. Traffic routing updates are delayed, and a manual SIGHUP is needed every time the config file changes.
Nginx itself has no built-in file watching and only reloads on SIGHUP. Sidecar tools like confd and consul-template bridge this gap using inotify to watch nginx.conf. The moment the file is written, inotify delivers an event within 1ms, the sidecar validates the new config and sends SIGHUP to Nginx, triggering a graceful reload.
Deploy an inotify-based sidecar instead of a polling cron job. In Kubernetes, the Ingress Controller watches for ConfigMap updates via the API server, writes the new config, and signals reload with the entire cycle completing in under 200ms compared to a 30-second polling interval.
A service needs to start the instant a file appears in a spool directory. A cron job checking every 60 seconds is too slow and wastes CPU, while writing a custom polling daemon adds maintenance burden for a simple trigger.
systemd .path units solve this with inotify under the hood. PathExists= watches for file creation, PathModified= watches for content changes, and PathChanged= catches attribute changes. When the kernel delivers the inotify event, systemd activates the associated .service unit within milliseconds. The process sleeps in epoll_wait with the inotify fd registered, consuming zero CPU while waiting.
Create a .path unit paired with a .service unit instead of writing cron jobs or custom daemons. This replaces polling scripts that check every 60 seconds, eliminating both the latency gap and the wasted CPU cycles.
Webpack crashes with the confusing "ENOSPC: no space left on device" error in a project with 30,000 source files, even though the disk has plenty of free space. Alternatively, polling 30,000 files burns 5-10% CPU and still misses sub-second changes.
chokidar (used by webpack and Vite) wraps inotify via fs.watch for instant notification when any file is saved. The catch is that inotify has no recursive watching, so chokidar must add a separate watch to every subdirectory. A typical node_modules tree has 50,000+ directories, blowing past the default max_user_watches limit of 8192 and producing the misleading ENOSPC error.
Run sysctl fs.inotify.max_user_watches=524288 to raise the limit. This costs about 270 MB of kernel memory but eliminates the cap for even the largest monorepos. With the fix applied, webpack recompiles within 200ms of saving a file at zero CPU cost while idle.
Same Concept Across Tech
| Technology | How it uses inotify | Key gotcha |
|---|---|---|
| Node.js (chokidar/fs.watch) | fs.watch wraps inotify on Linux. chokidar adds recursive watching | fs.watch is unreliable across platforms. chokidar is the standard |
| Kubernetes | ConfigMap updates use symlink atomic swap, not in-place file edit | Watch the directory, not the file. IN_CREATE on directory catches the swap |
| Docker | Host bind mounts do not generate inotify events inside the container on some setups | Use polling-based watchers inside containers as fallback |
| VS Code | Uses inotify for file watching. Large workspaces hit max_user_watches | VS Code docs recommend setting limit to 524288 |
| systemd (path units) | PathChanged= and PathModified= directives use inotify internally | Simpler than writing inotify code for service triggers |
| Go (fsnotify) | fsnotify wraps inotify on Linux, kqueue on macOS | Same event types across platforms but behavior differs |
Stack layer mapping (file watcher not triggering):
| Layer | What to check | Tool |
|---|---|---|
| Application | Is the watcher watching the right path? File or directory? | Application logs, debug output |
| Library | Does the library handle symlink swaps? (Kubernetes ConfigMap issue) | Library docs, test with ln -sf |
| Kernel | Is max_user_watches exhausted? Are events being dropped? | /proc/sys/fs/inotify/max_user_watches |
| Filesystem | Does the filesystem support inotify? (NFS, FUSE may not) | Check mount type with df -T |
| Container | Are bind-mount events propagating into the container? | Test from inside container |
Design Rationale inotify replaced dnotify because signal-based delivery was unreliable in multi-threaded programs and limited to one signal per directory. Delivering events through a regular file descriptor was the breakthrough -- it plugs directly into epoll and select without any signal handler gymnastics. Recursive watching was deliberately left out: letting the kernel auto-watch an entire subtree would mean unbounded memory growth as other processes create directories, with no user-level knob to cap it. fanotify came later for mount-wide monitoring, and its permission event model exists because on-access antivirus scanning needs to block file access at the kernel level -- something inotify cannot do.
If You See This, Think This
| Symptom | Likely cause | First check |
|---|---|---|
| ENOSPC error but disk has space | inotify watch limit reached, not disk space | cat /proc/sys/fs/inotify/max_user_watches |
| Dev server does not detect file changes | Watch on file, not directory, and file was replaced (not edited in place) | Watch the parent directory instead |
| Kubernetes ConfigMap change not picked up | App watches the file. K8s replaces via symlink swap | Watch directory for IN_CREATE, or use polling |
| File watcher works locally but not in Docker | Bind mount inotify propagation issues | Use polling mode (e.g., chokidar usePolling: true) |
| Events dropped silently under heavy load | max_queued_events exceeded, kernel drops events | Increase max_queued_events, check for IN_Q_OVERFLOW |
| Recursive directory watching is slow to set up | inotify does not support recursive watching natively | Each subdirectory needs its own watch. Use fanotify for mount-wide |
When to Use / Avoid
Use inotify when:
- Watching specific files or directories for changes (dev servers, config reloading)
- Building real-time file sync tools (Dropbox-like behavior)
- Triggering actions on file events without polling (log rotation, build pipelines)
Use fanotify when:
- Watching an entire mount point (antivirus, audit logging)
- Need to intercept and approve/deny file access (permission events)
Watch out for:
- inotify does NOT watch subdirectories recursively (must add watches for each subdirectory manually)
- Kubernetes ConfigMap/Secret updates use symlink swaps, so watch the DIRECTORY, not the file
- Default max_user_watches is 8,192 (too low for large projects with node_modules)
- Renaming a file generates MOVED_FROM + MOVED_TO events, not a single "rename" event
Try It Yourself
1 # Watch a directory for all events in real time
2 inotifywait -m /tmp
3
4 # Recursively watch with specific event types
5 inotifywait -m -r -e create,modify,delete,move /etc
6
7 # Check current inotify watch limits
8 cat /proc/sys/fs/inotify/max_user_watches
9
10 # Increase watch limit for large projects (webpack/vite)
11 echo 524288 | sudo tee /proc/sys/fs/inotify/max_user_watches
12
13 # Make the limit persistent across reboots
14 echo 'fs.inotify.max_user_watches=524288' | sudo tee -a /etc/sysctl.conf
15
16 # Use fatrace (fanotify) to see all file writes system-wide
17 sudo fatrace -f W
18
19 # Count which processes are generating the most file events
20 sudo fatrace -t -s 10 | awk '{print $2}' | sort | uniq -c | sort -rn | head
21
22 # Monitor Kubernetes ConfigMap changes (atomic symlink swap)
23 inotifywait -m -e moved_to,delete_self /etc/config/Debug Checklist
- 1
Check current watch limit: cat /proc/sys/fs/inotify/max_user_watches - 2
Count active watches per process: find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | cut -d/ -f3 | sort | uniq -c | sort -rn - 3
Increase watch limit: echo 524288 | sudo tee /proc/sys/fs/inotify/max_user_watches - 4
Test file watching: inotifywait -m /path/to/dir - 5
Check max queued events: cat /proc/sys/fs/inotify/max_queued_events - 6
Check max instances: cat /proc/sys/fs/inotify/max_user_instances
Key Takeaways
- ✓inotify has no recursive watching. Watching /etc only reports events in /etc itself, not /etc/nginx/conf.d/. You must walk the tree and add a watch to every subdirectory manually. This is why chokidar (webpack's watcher) can exhaust watch limits on large projects.
- ✓fanotify can block file access until your handler says "allow" or "deny." This is how on-access antivirus works -- ClamAV's clamonacc intercepts file opens via FAN_CLASS_CONTENT, scans the file, and only then lets the process proceed.
- ✓The default max_user_watches is 8192 on many systems. A single node_modules tree can have 50,000+ directories. Running out gives the confusing "ENOSPC: no space left on device" error even with plenty of disk space. Fix: sysctl fs.inotify.max_user_watches=524288.
- ✓Rename detection uses IN_MOVED_FROM and IN_MOVED_TO paired by a cookie value. If you see IN_MOVED_FROM without a matching IN_MOVED_TO, the file was moved outside your watched tree. This cookie pairing is essential for file sync tools.
- ✓The kernel coalesces rapid identical events -- 1000 writes to the same file may produce only one IN_MODIFY event. You cannot count exact modifications through inotify. You can only know that something changed.
Common Pitfalls
- ✗Mistake: Watching a config file directly for IN_MODIFY and missing atomic replacements. Reality: Tools like Kubernetes and vim replace files via write-to-tmp then rename. The watch follows the old inode, not the name. Watch the directory for IN_MOVED_TO instead.
- ✗Mistake: Treating inotify events as fixed-size structs. Reality: Events are variable-length because the name field varies. You must advance the buffer pointer by sizeof(struct inotify_event) + event->len per event, or you will read garbage.
- ✗Mistake: Using inotify for filesystem-wide auditing. Reality: inotify requires per-file/directory watches. fanotify can mark an entire mount point with a single call, using far fewer kernel resources.
- ✗Mistake: Ignoring IN_Q_OVERFLOW. Reality: When the event queue fills up (default 16384 events), the kernel drops events and delivers IN_Q_OVERFLOW. Your application must handle this by re-scanning watched directories to reconcile state.
Reference
In One Line
Use inotify to stop polling for file changes -- but raise max_user_watches early and watch the directory, not the file, when atomic replacements are in play.