Kernel Modules & Device Drivers
Mental Model
A power strip behind the desk. The strip is always on -- that is the kernel. Each appliance plugged in is a module: phone charger, desk lamp, monitor. Plug in and it works immediately; no need to shut down the strip. The prongs have to match the outlet shape (kernel version), or the plug will not fit. Yank a cord while the monitor is drawing power and something breaks. Unplugging has to happen in order -- disconnect the monitor before removing the adapter it chains through.
The Problem
The production server needs a GPU driver right now. Zero downtime. Recompiling a monolithic kernel takes 30-90 minutes, demands a reboot, and risks collateral breakage. Statically linking every possible driver would balloon the kernel past 200 MB, wasting memory on thousands of unused code paths. And after a routine kernel upgrade, Docker says "driver not supported" because overlay is gone -- VirtualBox VMs refuse to start because vboxdrv was never rebuilt.
Architecture
A USB device is plugged in. Five seconds ago, the kernel had no idea that hardware existed.
Now it works. No reboot. No recompilation. The kernel just absorbed new code at runtime, in ring 0, while all applications kept running.
That is what kernel modules do. And understanding them explains a surprising number of "it was working yesterday" mysteries.
What Actually Happens
When modprobe overlay runs, here is the sequence:
-
modprobe reads
/lib/modules/$(uname -r)/modules.depto find the overlay module and its dependencies. -
It loads any missing dependencies first, then opens the .ko file.
-
It calls
finit_module()-- the syscall that passes the file descriptor to the kernel. -
The kernel's
load_module()function verifies the ELF structure, checks module signatures (on Secure Boot systems), resolves symbol references against the kernel's exported symbol table, applies relocations, and allocates memory in vmalloc space. -
The kernel calls the module's
initfunction -- the one registered viamodule_init(). -
The init function registers with kernel subsystems. For a character device driver, this means
alloc_chrdev_region()for a device number,cdev_add()for the file_operations, anddevice_create()to trigger a kernel uevent. -
The uevent travels via netlink to udev in user space, which creates the
/devnode, sets permissions, and optionally creates symlinks.
From modprobe to a usable /dev entry -- milliseconds.
Under the Hood
Module parameters are runtime-configurable. module_param() creates entries in /sys/module/<name>/parameters/. Set at load time (modprobe mymod debug=1) or changed live by writing to sysfs. Parameters have types (int, charp, bool) and permission flags.
Symbol export controls the API surface. EXPORT_SYMBOL() makes functions available to other modules. EXPORT_SYMBOL_GPL() restricts to GPL-licensed modules -- the kernel enforces license compliance at the linker level.
Tainting is a social contract. When a proprietary module loads, the kernel becomes "tainted" (check with cat /proc/sys/kernel/tainted). The kernel still works, but kernel developers will refuse to debug oops/panic reports from tainted kernels. The taint persists until reboot.
Module removal is the reverse of loading. rmmod calls delete_module(), which checks the reference count (incremented when a device is opened or another module depends on it). If zero, the kernel calls the exit function, which must unregister everything in reverse order. Getting the order wrong leaks kernel memory or leaves dangling pointers that cause panics later.
Common Questions
How does modprobe differ from insmod?
insmod loads a single .ko file and fails if any symbol is unresolved. modprobe reads modules.dep to determine the dependency tree and loads prerequisites first. It also handles module aliases -- running modprobe snd-hda-intel resolves the PCI alias to find the correct module. For removal, modprobe -r also unloads unused dependencies.
What happens if a module's init function fails?
The kernel aborts the load: frees the module's memory, removes it from the module list, and returns the error to user space. Crucially, the init function must clean up anything it partially initialized before returning an error. The exit function is NOT called on init failure.
How to debug a kernel module crash?
Start with dmesg for the oops/panic trace -- it includes the instruction pointer, register dump, and call stack with symbol names. Use addr2line or gdb with the .ko's debug symbols to map the faulting address to a source line. For live debugging, ftrace or kprobes can trace function entry/exit.
Can a module be hot-reloaded with zero downtime?
Not directly. The sequence is rmmod then insmod. If the module manages an open device (reference count > 0), rmmod fails. Some subsystems support hot-swap (USB, PCI hotplug), but there is always a brief window where the device is unavailable. Live patching (livepatch) patches functions in loaded modules without unloading them -- but that is a different mechanism entirely.
How Technologies Use This
Docker suddenly fails with a cryptic driver not supported error after a kernel upgrade, and container starts that used to take 200ms now take 30 seconds. Multi-host overlay networks stop working entirely, leaving containers on different hosts unable to communicate.
The underlying issue is that the overlay kernel module was not loaded after the upgrade. Without it, Docker silently falls back to the vfs storage driver, which copies entire image layers instead of using union mounts. Similarly, missing br_netfilter, ip_tables, and vxlan modules break container networking at the kernel level, but Docker only reports vague errors.
Docker runs modprobe overlay at startup and verifies support via /proc/filesystems. The fix after a kernel upgrade is ensuring all required modules are loaded or configured in /etc/modules-load.d/. Understanding this kernel module dependency chain is the difference between chasing application bugs and solving the actual host-level configuration issue in seconds.
Same Concept Across Tech
| Concept | Docker | JVM | Node.js | Go | K8s |
|---|---|---|---|---|---|
| Dynamic code loading | Requires overlay/br_netfilter modules | ClassLoader loads .jar at runtime | require() / import() for modules | plugin.Open() for .so files | Node kernel modules via container runtime |
| Dependency resolution | modprobe loads overlay deps | Maven/Gradle resolve dependencies | npm resolves package.json | go mod download | Helm dependency charts |
| Version compatibility | Module must match host kernel | Bytecode targets JVM version | Node version constraints in engines | Go module version constraints | K8s API version compatibility |
| Hot reload | rmmod + modprobe (brief gap) | HotSwap / JRebel for classes | nodemon restart | N/A (recompile required) | Rolling pod restart |
| Signature verification | Secure Boot module signing | JAR signing (jarsigner) | npm audit / provenance | go mod verify checksums | Image signature (cosign/notary) |
Stack Layer Mapping
| Layer | Module Mechanism |
|---|---|
| Hardware | Device exposes PCI/USB IDs that trigger module autoloading |
| Kernel core | load_module() verifies ELF, resolves symbols, allocates vmalloc memory |
| Module init | Registers with subsystem (chardev, netdev, blockdev, filesystem) |
| udev / devtmpfs | Creates /dev nodes, applies permission rules, runs trigger scripts |
| Userspace tools | modprobe reads modules.dep, depmod rebuilds dependency database |
| Application | Opens /dev/xyz, reads /sys/module/<name>/parameters/ |
Design Rationale
A monolithic kernel with every driver baked in would waste memory and demand recompilation for any hardware change -- modules let code load on demand. Symbol CRC verification (MODVERSIONS) catches the case where a module compiled against one kernel would corrupt another, enforcing ABI stability at the binary level. finit_module() replaced init_module() so the kernel can verify a signature on the file descriptor before mapping untrusted code into ring 0.
If You See This, Think This
| Symptom | Likely Cause | First Check |
|---|---|---|
| "module not found" on modprobe | Module not installed for current kernel version | ls /lib/modules/$(uname -r)/kernel/ for the .ko file |
| "disagrees about version of symbol" | Module compiled against different kernel config | modinfo <module> and compare vermagic with uname -r |
| "Key was rejected by service" | Unsigned module on Secure Boot system | mokutil --sb-state and check if module is signed |
| rmmod fails with "Module in use" | Non-zero reference count (device open or dependent module) | `lsmod |
| Docker "driver not supported" after upgrade | overlay/br_netfilter module not loaded for new kernel | `modprobe overlay && cat /proc/filesystems |
| Device not appearing in /dev | udev rule missing or module init did not call device_create() | `dmesg |
When to Use / Avoid
Use when:
- Adding hardware support (GPU, NIC, storage controller) without kernel recompilation
- Enabling filesystem or networking features for containers (overlay, br_netfilter, vxlan)
- Writing custom device drivers for specialized hardware
- Debugging kernel behavior with tracing modules (kprobes, ftrace integration)
- Deploying out-of-tree drivers (NVIDIA, VirtualBox, ZFS) on production systems
Avoid when:
- eBPF can accomplish the goal (tracing, filtering, networking) without ring 0 risk
- The functionality exists as a built-in (check /proc/config.gz before writing a module)
- Security policy requires a locked-down kernel (Secure Boot + module signing only)
Try It Yourself
1 # List all loaded kernel modules with sizes
2
3 lsmod | head -20
4
5 # Show detailed info about a specific module
6
7 modinfo ext4
8
9 # Load a module with parameters
10
11 sudo modprobe loop max_loop=64
12
13 # Check module dependencies
14
15 modprobe --show-depends nvidia 2>/dev/null || echo 'Module not available'
16
17 # Watch device events from udev in real-time
18
19 udevadm monitor --property
20
21 # List character devices and their major numbers
22
23 cat /proc/devices | head -30Debug Checklist
- 1
lsmod | grep <name> -- check if module is loaded - 2
modinfo <module> -- show version, license, parameters, dependencies - 3
modprobe --show-depends <module> -- trace full dependency chain - 4
dmesg | tail -30 -- check kernel log for module init errors - 5
cat /proc/sys/kernel/tainted -- check if kernel is tainted by proprietary modules - 6
cat /proc/modules | awk '{print $1, $3}' -- show module reference counts
Key Takeaways
- ✓modprobe is smart, insmod is not. modprobe reads modules.dep (generated by depmod) and loads prerequisites first. insmod loads a single .ko file and fails on unresolved symbols. That dependency resolution is why modprobe fixes problems insmod cannot.
- ✓The 'disagrees about version of symbol' error is not a bug -- it is ABI protection. The kernel checks that exported symbol CRC signatures match (MODVERSIONS). A module compiled against a different kernel config will be rejected.
- ✓Every loaded module consumes non-swappable kernel memory. The .text section lives in vmalloc space, per-CPU data uses the per-CPU allocator. Hundreds of unnecessary modules waste precious kernel address space.
- ✓finit_module() (Linux 3.8+) loads from a file descriptor, enabling signature verification before loading. Modern insmod/modprobe use this syscall, not the older init_module().
- ✓Device numbers: major identifies the driver (8 = sd, 1 = mem), minor identifies the device instance (sda=0, sda1=1). Modern kernels use dynamic major allocation via alloc_chrdev_region() to avoid conflicts.
Common Pitfalls
- ✗Mistake: Forgetting to unregister resources in module_exit. Reality: If you register a char device, create a class, and add a device, you must undo all three in reverse order. Missing any step leaks kernel resources or leaves stale /dev entries.
- ✗Mistake: Using GFP_KERNEL allocations in interrupt context. Reality: Interrupt handlers and tasklets must use GFP_ATOMIC, which can fail. The driver must handle NULL returns gracefully.
- ✗Mistake: Dereferencing userspace pointers directly in ioctl handlers. Reality: This bypasses SMAP protection and crashes on invalid pointers. Use copy_from_user/copy_to_user -- always.
- ✗Mistake: Building modules against headers that do not match the running kernel. Reality: The .ko file either fails to load (version magic mismatch) or causes subtle memory corruption if MODVERSIONS is disabled.
Reference
In One Line
modprobe over insmod -- it handles dependencies -- and re-check every module after a kernel upgrade before wondering why Docker or VirtualBox broke.