Hard Links & Symbolic Links
Mental Model
Phone contacts. A hard link is the same phone number saved under two names -- "Mom" and "Emergency Contact." Both dial the same person. Delete "Mom" and "Emergency Contact" still works; the number is only disconnected when every entry pointing to it is removed. A symlink is a note in the contacts: "call whoever is listed under Mom." Delete the "Mom" entry and the note still says to call Mom -- but nobody answers. Dangling reference.
The Problem
Rolling a config update across 200 servers with zero downtime. A naive write risks a crash window where the file is half-written. A 10 GB Git repo needs local cloning in under a second without copying data. Fifty containers share a 200 MB base image and cannot afford 50 separate copies. And somewhere, a file was "deleted" but df still shows 80% full -- ghost data consuming disk with no visible files to account for it.
Architecture
Two files exist: report.txt and report_backup.txt. They have the same content, the same inode number, and the same size. But there's only one copy of the data on disk.
Now report.txt gets deleted. Is the data gone?
No. The data is untouched. Only one name was removed. The other name — report_backup.txt — still points to the same inode, the same data blocks. Nothing was lost.
This is what hard links do. And the fact that it surprises people is the reason this topic matters.
What Actually Happens
Hard links. When a process calls link("a.txt", "b.txt"), the kernel creates a new directory entry in b.txt's parent directory pointing to a.txt's inode, and increments the inode's i_nlink counter. That's it. No data copy. No new inode. Both names are completely equivalent — there's no "original" and "copy." The data is only freed when i_nlink drops to zero AND no process has the file open.
Symlinks. When a process calls symlink("a.txt", "link.txt"), the kernel creates a brand new inode with type S_IFLNK whose content is the string "a.txt". When the VFS encounters this during path walk, it reads the string and restarts resolution from that path. Relative symlinks resolve from the symlink's directory. Absolute symlinks resolve from root. The kernel caps symlink depth at 40 total traversals (8 per component) and returns ELOOP if exceeded.
Atomic rename. Here's the pattern that makes crash-safe config updates work:
- Write new config to a temp file
fsync()the temp file (ensure data hits disk)rename(temp, target)— atomic on the same filesystem
rename() is a single dentry operation: it swaps directory entries without touching the inode or data blocks. If the system crashes mid-rename, either the old or new file is present — never neither, never a half-written mess. This is why every well-written daemon uses write-to-temp-then-rename.
Under the Hood
Fast symlinks. On ext4, if the symlink target path is shorter than 60 bytes, it's stored directly in the inode's i_block[] array — the same space that normally holds block pointers. No data block allocation needed. This makes short symlinks extremely cheap in both space and I/O. Longer targets require a dedicated data block.
Why hard-linking directories is forbidden. The kernel prevents it (returns EPERM) because directory hard links would create cycles in the filesystem tree. If /a/b hard-linked to /a, an infinite loop would break find, fsck, du, and every tool that walks the tree. The . and .. entries are special — they're hard links to directories, but only the kernel creates them, and they follow strict rules that maintain the tree invariant.
Hardlink protection. Since Linux 3.6, the protected_hardlinks sysctl (enabled by default) prevents unprivileged users from creating hard links to files they don't own. Without this, an attacker could hard-link to a setuid binary, then exploit a TOCTOU race when the binary is updated. The protection closes a real, exploited vulnerability class.
Symlink permissions don't matter. On Linux, ls -l shows lrwxrwxrwx for every symlink. Those permissions are always 0777 and the kernel ignores them completely. Access control is determined by the target file's permissions. lchown() can change a symlink's ownership, but lchmod() doesn't exist on Linux. The displayed permissions are cosmetic.
Common Questions
How does mv handle cross-filesystem moves?
mv first tries rename(), which only works when source and destination are on the same filesystem (same st_dev). If it fails with EXDEV (cross-device link), mv falls back to copy-then-unlink: read data, write to destination, preserve metadata with fchmod()/fchown()/utimensat(), then unlink() the source. This fallback is NOT atomic — a crash during the copy can leave data in both locations or incomplete at the destination.
What happens when rm is called on a file with multiple hard links?
rm calls unlink(), which removes one directory entry and decrements i_nlink. If other names still exist (i_nlink > 0), the inode, metadata, and data blocks are completely unaffected. The file is fully accessible through its remaining names. It's only when the last name is removed and the last fd is closed that the kernel frees everything.
How do container runtimes use symlinks for rootfs setup?
Kubernetes ConfigMaps use a clever symlink chain: the actual data lives in a timestamped directory (.2024_01_01), and a .data symlink points to it. When a ConfigMap is updated, a new timestamped directory is created and the .data symlink is atomically replaced via rename(). In-flight reads complete against the old directory. New reads see the new data. Nobody sees a torn file.
Can a hard link point to a symlink?
Yes. link() doesn't follow symlinks by default (since Linux 2.0). It creates a new directory entry pointing to the symlink's inode, producing two names for the same symlink. The flag AT_SYMLINK_FOLLOW with linkat() changes this — it follows the symlink and hard-links to the target instead.
How Technologies Use This
Running git clone --local on a 10GB repository finishes in under a second. The destination directory takes near-zero additional disk space. Without hard links, that clone would mean copying 10GB of object data byte by byte.
When cloning on the same filesystem, Git creates new directory entries in the destination's .git/objects that point to the same inodes as the source. No data blocks are duplicated. For ref updates, Git uses the atomic rename pattern: write the new SHA to a .lock file created with O_CREAT|O_EXCL, call fsync(), then rename over the target ref. Readers either see the old ref or the new one, never a partial write.
Hard links give Git instant local clones with zero data duplication, and atomic rename gives it crash-safe ref updates. Both are filesystem primitives doing the heavy lifting that would otherwise require complex application-level copying and journaling.
Fifty containers on the same host all share a 200MB base image. Without deduplication, that is 10GB of redundant data sitting on disk. And when a ConfigMap update rolls out, any window of inconsistency means containers read half-old, half-new config files.
Docker's overlay2 storage driver uses hard links to share identical files across image layers, so all 50 containers reference the same underlying inodes and data blocks. For live configuration updates, Kubernetes ConfigMap volumes use an atomic symlink swap: the kubelet writes new config to a timestamped directory, then calls rename() to re-point the ..data symlink. Because rename() is a single dentry operation, in-flight reads complete against the old directory while new reads instantly see the update.
Hard links handle deduplication. Atomic symlink swaps handle zero-downtime updates. Both are basic filesystem primitives doing work that would otherwise require a custom content-addressed storage engine and a distributed locking protocol.
Same Concept Across Tech
| Concept | Docker | JVM | Node.js | Go | K8s |
|---|---|---|---|---|---|
| File deduplication | overlay2 uses hard links for shared layers | N/A | N/A | N/A | Container runtime handles layer sharing |
| Atomic config update | N/A | AtomicMoveFileOperation (NIO) | fs.rename() (same fs only) | os.Rename() (same fs only) | ConfigMap symlink swap via kubelet |
| Symlink resolution | Follows symlinks in COPY/ADD | Files.readSymbolicLink() | fs.readlink() / fs.realpath() | os.Readlink() / filepath.EvalSymlinks() | Volume mounts follow symlinks |
| Cross-device handling | Copies across layers (no hard link) | Files.move with COPY fallback | Detects EXDEV, falls back to copy | Detects EXDEV, manual copy needed | PV mounts may span devices |
| Dangling detection | N/A | Files.exists with NOFOLLOW_LINKS | fs.lstat + fs.stat comparison | os.Lstat + os.Stat comparison | kubelet validates symlink targets |
Stack Layer Mapping
| Layer | Link Mechanism |
|---|---|
| Hardware | Disk stores data blocks; links are metadata-only |
| Filesystem (ext4/xfs) | Hard links: additional dentry to same inode. Symlinks: inode with path content |
| VFS | namei path walker follows symlinks (40-deep limit), resolves hard links transparently |
| System calls | link(), symlink(), unlink(), readlink(), rename() |
| Container runtime | Hard links for layer dedup, symlinks for rootfs setup |
| Application | Atomic rename pattern for crash-safe updates |
Design Rationale
Multiple names for one inode is the cheapest deduplication the filesystem can offer -- no data copy, no extra blocks, just a new dentry and an incremented counter. That is a hard link. Symlinks exist because hard links cannot cross filesystem boundaries or point to directories, and sometimes a path-based indirection is more useful than identity-based sharing. The 40-traversal cap on symlinks keeps circular chains from eating kernel stack.
If You See This, Think This
| Symptom | Likely Cause | First Check |
|---|---|---|
| "No such file or directory" on existing symlink | Dangling symlink (target deleted/moved) | readlink -f <symlink> and check if target exists |
| EXDEV error on rename or link | Source and destination on different filesystems | stat <src> <dst> and compare st_dev values |
| Disk full but no large files visible | Deleted files still held open by processes | lsof +L1 to find unlinked but open files |
| ELOOP error during file access | Circular symlink chain (A -> B -> A) | readlink <each link> to trace the chain |
| EPERM on link() call | Trying to hard-link a directory, or protected_hardlinks blocking | ls -ld <target> and check /proc/sys/fs/protected_hardlinks |
| File changes not reflected through symlink | Symlink points to old copy after non-atomic update | Verify atomic rename pattern: strace -e rename <process> |
When to Use / Avoid
Use when:
- Deduplicating identical files across backup snapshots (hard links via rsync --link-dest)
- Performing atomic config updates on production servers (write-temp + rename over symlink)
- Sharing base image layers across containers without data duplication (hard links in overlay2)
- Providing version-agnostic paths to versioned binaries (python3 -> python3.11 symlinks)
- Rolling back deployments by re-pointing a symlink to a previous release directory
Avoid when:
- Crossing filesystem boundaries (hard links fail with EXDEV; use symlinks instead)
- Linking directories (hard links to directories are forbidden; use bind mounts)
- The target might be moved or deleted (symlinks will dangle; hard links are immune)
Try It Yourself
1 # Create a hard link and verify both entries share the same inode number and link count of 2
2 ln /tmp/original.txt /tmp/hardlink.txt && ls -li /tmp/original.txt /tmp/hardlink.txt
3
4 # Create a symbolic link. note the different inode number and 'l' file type in ls output
5 ln -s /tmp/original.txt /tmp/symlink.txt && ls -li /tmp/symlink.txt
6
7 # Show the hard link count (Links field) for a file
8 stat /tmp/original.txt | grep Links
9
10 # Find files with multiple hard links on the /usr filesystem
11 find /usr -xdev -type f -links +1 | head -20
12
13 # Fully resolve a symlink chain to its final target (e.g., python3 -> python3.11 -> ..)
14 readlink -f /usr/bin/python3
15
16 # Find dangling symlinks (symlinks whose target does not exist) in /etc
17 find /etc -maxdepth 2 -type l -xtype l 2>/dev/nullDebug Checklist
- 1
ls -li <file> -- show inode number and hard link count - 2
stat <file> | grep Links -- check i_nlink count - 3
find / -xdev -samefile <file> -- find all hard links to the same inode - 4
readlink -f <symlink> -- resolve full symlink chain to final target - 5
find /path -type l -xtype l -- find dangling symlinks - 6
lsof +L1 -- find open files with zero link count (deleted but still held open)
Key Takeaways
- ✓Hard links can't cross filesystem boundaries — period. Inode numbers are only unique within a filesystem, so a directory entry on /dev/sda1 can't reference an inode on /dev/sdb1. Symlinks work across filesystems because they store a path, not an inode number
- ✓You can't hard-link directories (link() returns EPERM) because it would create cycles in the filesystem tree, breaking find, fsck, and every tool that assumes directories form a tree. Only the kernel creates the '.' and '..' entries
- ✓Symlink loops don't hang the kernel — it counts. ELOOP fires after 40 total symlink traversals or 8 within a single path component. If you see ELOOP, you've got a circular chain
- ✓rename() on the same filesystem is atomic because it's just a dentry operation — the inode, data blocks, and permissions are untouched. This is the foundation of every crash-safe config update: write to temp, rename over target
- ✓Symlink permissions (lrwxrwxrwx) are cosmetic — Linux ignores them entirely. Access control is always determined by the target file's permissions, never the symlink's
Common Pitfalls
- ✗Thinking unlink() deletes a file — it removes ONE name. The data survives as long as other hard links exist or any process holds an open fd. Check st_nlink and lsof to understand what's really happening
- ✗Creating symlinks with relative paths, then moving the symlink — the relative target resolves from the symlink's new location, not the old one. The symlink dangles, and you stare at "No such file or directory"
- ✗Trying rename() across filesystem boundaries — it fails with EXDEV. You must copy + unlink instead, which is NOT atomic. That's what mv does internally when crossing devices
- ✗Ignoring ELOOP when following symlinks — symlink chains or cycles cause failures that look like "file not found" if you don't check the specific error. Set O_NOFOLLOW when you want to operate on the symlink itself
Reference
In One Line
Hard links share an inode, symlinks store a path -- pick hard links when the target stays put, symlinks when it might move or lives on another filesystem.