POSIX Threads

Here is something that surprises most people: Linux does not have threads. Not really. What it has are processes that share things.

When a pthread is created, the kernel creates a new task_struct -- the exact same data structure it uses for a process. The only difference is the flags passed to clone(). A process shares nothing. A thread shares everything -- address space, file descriptors, signal handlers. The kernel does not distinguish between the two. It is the same mechanism with different knobs turned.

This design choice has profound consequences. It means threads are cheap (they are just processes under the hood). It means the scheduler treats threads and processes identically. And it means understanding how threads really work on Linux requires understanding clone(), futex(), and the FS segment register.

What Actually Happens

When pthread_create() is called, here is what happens under the hood:

1. Glibc allocates a new stack via mmap() (default 8MB + guard page).
2. At the top of the stack, it places the Thread Control Block (TCB) -- a struct pthread holding TLS pointers, cleanup handlers, cancellation state, and a futex for pthread_join().
3. The FS segment register is set to point to the TCB. This is how thread-local storage works -- each __thread variable is accessed as an offset from FS, compiling down to a single mov instruction.
4. Glibc calls clone() with flags: CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD | CLONE_SYSVSEM.

CLONE_VM shares the address space. CLONE_FILES shares file descriptors. CLONE_THREAD groups threads under the same tgid (thread group ID), which is what getpid() returns. The new thread gets its own task_struct, its own stack, its own register state, and its own signal mask.

The futex trick. This is where things get clever. Synchronization primitives -- mutexes, condvars, rwlocks, barriers -- are all built on the Linux futex() syscall. The key design is a two-level system:

• Fast path (userspace): An uncontended pthread_mutex_lock() is an atomic compare-and-swap on a 32-bit integer in userspace. No syscall. About 25 nanoseconds.
• Slow path (kernel): When the CAS fails (another thread holds the lock), glibc calls futex(FUTEX_WAIT) to put the thread to sleep in the kernel. pthread_mutex_unlock() does an atomic store and, if there are waiters, calls futex(FUTEX_WAKE).

Most mutex operations in a well-designed program are uncontended. That means most of them never enter the kernel. This is why pthreads are fast.

Under the Hood

Mutex types matter more than most people think. PTHREAD_MUTEX_NORMAL (default) silently deadlocks on recursive locking and has undefined behavior if unlocked by the wrong thread. PTHREAD_MUTEX_ERRORCHECK returns EDEADLK on recursive lock and EPERM on wrong-owner unlock -- use this during development. PTHREAD_MUTEX_RECURSIVE allows the same thread to lock multiple times (maintains a count). PTHREAD_MUTEX_ROBUST detects owner death via the kernel's robust_list mechanism -- if a thread dies while holding the lock, the next locker gets EOWNERDEAD and can recover.

Condition variable semantics. pthread_cond_wait() atomically unlocks the mutex and blocks on the condvar's futex. The word "atomically" is doing heavy lifting here. Without atomic unlock-and-sleep, a signal could arrive in the gap between unlock and sleep -- the waiter would miss it and sleep forever. This is the "lost wakeup" problem, and it is why condvars exist (instead of just using a flag + sleep).

Spurious wakeups are allowed by POSIX and do occur. Always wait in a while loop that re-checks the predicate. Using if instead of while is a classic bug.

Thread cancellation. Deferred cancellation (default) only fires at specific "cancellation points" -- functions like sleep(), read(), write(), pthread_cond_wait(). The cancelled thread runs cleanup handlers in LIFO order. Asynchronous cancellation can fire at any instruction and is almost never safe -- it can leave mutexes locked and heap corrupted. In practice, use a shared "shutdown" flag that threads check periodically.

NPTL vs LinuxThreads. The old LinuxThreads implementation (pre-glibc 2.3.2) used a manager thread for thread creation and did not properly share PIDs. NPTL (Native POSIX Threads Library) provides true 1:1 threading with correct PID/TID semantics, faster creation, and proper POSIX signal delivery. Every modern Linux system uses NPTL.

Common Questions

What is the performance difference between a mutex and a spinlock?

An uncontended mutex is about 25ns (one CAS, no syscall). A contended mutex puts the thread to sleep (context switch cost ~1-5 microseconds). A spinlock burns CPU but avoids the context switch -- it is faster for very short critical sections (under 1 microsecond) where the lock holder is running on another core. For anything longer, the mutex wins because it does not waste CPU. PTHREAD_MUTEX_ADAPTIVE_NP offers the best of both: it spins briefly, then sleeps.

How does priority inversion happen with pthreads?

A high-priority thread blocks on a mutex held by a low-priority thread. A medium-priority thread preempts the low-priority thread. Now the high-priority thread is effectively stuck behind the medium-priority thread. The fix: PTHREAD_PRIO_INHERIT (priority inheritance) temporarily boosts the lock holder to the highest waiter's priority. Linux implements this via futex(FUTEX_LOCK_PI) and rt_mutex internally. This is what saved the Mars Pathfinder mission.

Why is fork() so dangerous in a multithreaded process?

fork() clones only the calling thread. Every other thread vanishes in the child, but their mutexes stay locked. If the child calls malloc() -- which uses an internal arena mutex -- and one of the vanished threads was holding that mutex, the child deadlocks. The only safe things to do after fork in a threaded program are calling exec() or async-signal-safe functions. pthread_atfork() can register cleanup handlers, but it cannot cover every third-party library's internal locks.