Interrupt Handling & Softirqs

The CPU does not poll hardware for events. It gets forcibly interrupted.

A packet arrives on the NIC. A disk completes a DMA transfer. A key is pressed. In each case, the hardware yanks the CPU out of whatever it was doing -- mid-instruction if necessary -- and says "deal with this now."

That sounds fine for a keyboard. But a server receiving 10 million packets per second? If every packet triggers a full interrupt, the machine spends all its time answering the doorbell and none of it doing actual work.

What Actually Happens

When a device needs CPU attention, it signals via MSI-X -- writing a small message to a memory address that the CPU's Local APIC interprets as an interrupt vector.

The CPU immediately suspends its current execution. Saves registers. Looks up the vector in the IDT. Jumps to the registered handler.

This handler is the top-half (hardirq). It runs with the current interrupt line masked, acknowledges the hardware, grabs essential data (like a DMA completion status), and raises a softirq. The whole thing should take microseconds, not milliseconds.

The real processing happens in the bottom-half. When the hardirq returns, the kernel checks the softirq pending bitmask and runs all pending handlers inline -- before returning to whatever user-space task was interrupted. There are 10 fixed softirq types: NET_RX, NET_TX, BLOCK, TIMER, SCHED, and others.

Here is the catch: if softirq processing takes too long (more than ~2ms or 10 restart rounds), the kernel stops and defers the remaining work to ksoftirqd -- a per-CPU kernel thread running at nice 19 priority. This prevents softirqs from starving applications.

For work that needs process context -- sleeping, taking mutexes, performing I/O -- the kernel provides workqueues. A driver calls schedule_work() to enqueue a work item that executes later in a kworker thread. Unlike softirqs, workqueues can sleep.

Under the Hood

NAPI changes the game for networking. At millions of packets per second, the interrupt overhead alone can consume entire CPU cores. NAPI solves this by switching to a poll model: after the first packet interrupt, the driver disables further NIC interrupts and registers a poll function. The kernel's NET_RX softirq polls the device for batches of packets (up to 64) without interrupts. When the queue drains, interrupts are re-enabled. This is why ksoftirqd often shows high CPU on network-heavy boxes -- it is NAPI doing its job.

Interrupt coalescing trades latency for throughput. NICs can be configured (ethtool -C) to delay interrupts and batch completions. rx-usecs=50 tells the NIC to wait 50 microseconds before signaling, grouping ~100 packets per interrupt. This cuts interrupt rate 10-100x but adds latency. Adaptive coalescing (adaptive-rx on) adjusts dynamically based on load.

IRQ affinity and RSS. Multi-queue NICs have one IRQ per receive queue. RSS (Receive Side Scaling) hashes incoming packets to distribute flows across queues. Setting /proc/irq/N/smp_affinity_list pins each queue's IRQ to a specific CPU core. The goal: the application processing packets from queue N runs on the same CPU as IRQ N, keeping data in L1/L2 cache.

Threaded IRQs are now the default. Since kernel 4.x, most drivers use request_threaded_irq(). The hardirq handler is minimal -- it just wakes a dedicated kernel thread that runs the real handler in process context. This makes interrupt handling fully preemptible and improves latency predictability.

Common Questions

What happens if a hardirq handler takes too long?

Other interrupts of the same type are delayed. If the handler runs for milliseconds, NIC ring buffers overflow (packet drops), keystrokes are lost, and timers drift. The kernel's lockup detector triggers if a hardirq holds a CPU for more than 10 seconds. In RT kernels, handlers exceeding their budget generate latency traces.

How do softirqs differ from tasklets?

Softirqs are statically allocated (only 10 types) and can run concurrently on multiple CPUs -- the same handler can execute simultaneously on different cores. Tasklets are dynamically allocatable and serialized -- a given tasklet runs on only one CPU at a time. Since kernel 5.9, tasklets are deprecated for new code in favor of threaded IRQs and workqueues.

Why does ksoftirqd sometimes cause latency problems?

When softirq work is deferred to ksoftirqd, it runs at nice 19 (lowest priority). If the system has CPU-bound tasks, ksoftirqd may not get scheduled promptly, adding milliseconds of delay to network packet processing. Solutions: use cgroup CPU reservations for ksoftirqd, enable RPS to distribute work, or use busy_poll mode where the application thread directly polls the NIC.