Daemons & Service Management
Mental Model
A shopkeeper versus a vending machine. The shopkeeper needs someone at the counter, goes home at closing time, stops when the store shuts. The vending machine runs 24/7 with nobody present, restocks when told to, keeps dispensing even with the lights off. A daemon is the vending machine -- no terminal needed, does not die when the operator leaves, and a supervisor (systemd, the building manager) restarts it if it jams.
The Problem
A web server starts from an SSH session. The operator logs out and the server dies -- SIGHUP fires when the controlling terminal closes. Slapping nohup on it keeps it alive, but there is no restart on crash, no dependency ordering, no log management, and no signal that startup actually succeeded. Two instances can run side by side because nothing enforces single-instance.
Architecture
An engineer writes a web server. Starts it from a terminal. It works. The session closes. It dies.
That single sentence explains why daemons exist. A regular process is tied to the terminal that launched it -- when the terminal goes away, the process gets SIGHUP and dies. A daemon is a process that has deliberately cut every tie to its launching environment: no terminal, no inherited file descriptors pointing at dead ttys, no working directory holding a mount point busy.
For decades, creating a daemon required a specific six-step ritual. Fork, detach, fork again, change directory, reset the file mask, redirect stdio. Get any step wrong and the result is a process that mostly works but occasionally crashes at 3 AM when a log rotation closes a file descriptor it should not have had.
Then systemd came along and said: "just run the program in the foreground. We will handle the rest." Understanding both approaches -- and when each one matters -- is the point of this article.
What Actually Happens
The classic double-fork. Here are the six steps, and why each one matters:
-
First fork, parent exits. The shell's child exits, returning the prompt. The grandchild continues but is now an orphan, reparented to PID 1. It also has a different PID than its PGID, which is a prerequisite for step 2.
-
setsid(). Creates a new session and process group. The process becomes the session leader. It has no controlling terminal. This is the critical step -- it severs the connection to the launching terminal.
-
Second fork, first child exits. The grandchild is not a session leader (its SID != its PID). On some systems, a session leader can accidentally acquire a controlling terminal by opening a tty device. The second fork makes this impossible.
-
chdir("/"). If the daemon's working directory is on a mounted filesystem, that filesystem cannot be unmounted while the daemon runs. Changing to root avoids this.
-
umask(0). Inherited umask may prevent the daemon from creating files with the permissions it needs. Resetting to 0 gives the daemon full control.
-
Close and redirect stdio. Close fds 0, 1, 2 and reopen them pointing to /dev/null. Without this, writes to stdout hit a dead terminal and generate SIGPIPE. Reads from stdin block forever.
The systemd way. With Type=simple, none of this is needed. The daemon runs as a foreground process directly under systemd. systemd provides: session isolation (the process is in its own session), stdio redirection (stdout/stderr go to journald), working directory control (WorkingDirectory= directive), environment cleanup, and cgroup-based process tracking.
Type=notify adds readiness signaling. The daemon calls sd_notify("READY=1") after it finishes initialization -- loading config, opening sockets, warming caches. systemd waits for this signal before starting dependent services. This is strictly better than the old approach of polling a PID file or guessing with timeouts.
Under the Hood
PID 1 adoption. When the daemon's parent exits (step 1), the daemon is orphaned and reparented to PID 1 (init/systemd). PID 1 automatically calls wait() for orphans, preventing zombies. Under systemd, the daemon is reparented to the service manager, which tracks it via its cgroup.
PID file locking. Traditional daemons write their PID to /var/run/<name>.pid for management scripts. Proper implementation uses flock() or fcntl() locks -- if the lock is held, another instance is running. Without locking, a stale PID file from a crashed daemon causes confusion. systemd renders PID files unnecessary because it tracks processes by cgroup, not by PID.
Socket activation. This is the most powerful feature most people overlook. systemd opens the listening socket(s) and passes them to the daemon as pre-opened file descriptors (fd 3, 4, etc.). The daemon calls sd_listen_fds() to discover how many sockets it received and accept() on them.
The benefits are significant. The socket exists before the daemon starts, so clients queue connections during startup. Multiple daemon instances can share a socket with systemd managing the handoff. And zero-downtime restarts are free -- systemd stops the daemon but keeps the socket open. Incoming connections queue in the kernel backlog. When the new instance starts, it picks up right where the old one left off.
Watchdog integration. systemd's WatchdogSec=30 expects the daemon to call sd_notify("WATCHDOG=1") periodically. Miss the deadline and systemd restarts the service. This catches deadlocks and hangs that PID-based monitoring cannot detect.
Security hardening. systemd unit files support extensive sandboxing that would take hundreds of lines of C to implement manually: PrivateTmp=yes (isolated /tmp), ProtectSystem=strict (read-only filesystem), NoNewPrivileges=yes (no setuid escalation), CapabilityBoundingSet=CAP_NET_BIND_SERVICE (drop all capabilities except binding privileged ports), PrivateNetwork=yes (isolated network namespace).
Common Questions
Why is the second fork necessary?
After setsid(), the process is a session leader. On SVR4-derived systems, a session leader can acquire a controlling terminal by opening a terminal device without O_NOCTTY. The second fork creates a child that is not a session leader (SID != PID), making it impossible to accidentally get a controlling terminal. On Linux, ioctl(TIOCSCTTY) is explicitly required, so the second fork is less critical -- but it is still best practice for portability.
How does systemd know a Type=forking service is ready?
systemd watches for the original process to exit (the first fork). The surviving child is assumed to be the daemon. If PIDFile= is specified, systemd reads the PID from that file to identify the main process. If there is no PIDFile, systemd guesses using the remaining processes in the cgroup. This is fragile. Type=notify is more reliable because the daemon explicitly signals readiness.
What happens to inherited file descriptors on exec?
File descriptors without O_CLOEXEC survive across exec(). This is a security risk -- the daemon may inherit open sockets, files, or pipes from the launching process. Modern code uses SOCK_CLOEXEC, O_CLOEXEC, and pipe2(O_CLOEXEC) to ensure fds close on exec. systemd's ExecStart= processes get a clean fd set: only the passed sockets plus stdin/stdout/stderr.
How does socket activation enable zero-downtime restarts?
The listening socket is owned by systemd, not the daemon. During restart, systemd stops the daemon but keeps the socket open. Incoming connections queue in the socket's kernel backlog (default 128-4096 depending on configuration). When the new daemon instance starts, it receives the same socket and processes the queued connections. No connections are refused. No clients see an error. The restart is invisible.
How Technologies Use This
An operator runs systemctl stop on a service. It reports as stopped. But hours later, orphaned grandchildren are still running -- holding ports, leaking memory, and corrupting log files. The PID file pointed to a process that exited cleanly, but the daemon had double-forked children that escaped tracking entirely.
Traditional init systems track daemons by PID file, which any double-fork or setsid() call trivially escapes. The tracked PID no longer represents anything useful, and kill(-pgid) only reaches processes sharing the original process group. Rogue children in new sessions are invisible.
systemd places every service in a dedicated cgroup that no fork trick can escape. KillMode=control-group sends SIGTERM to every process in the tree, achieving 100% cleanup reliability. Type=notify lets the daemon signal actual readiness via sd_notify(READY=1), replacing PID file polling and sleep-based guessing entirely.
Upgrading dockerd on a production host severs every running docker exec, docker logs, and API call instantly. CI/CD pipelines fail, monitoring agents disconnect, and operators scramble. All because restarting the daemon closed the API socket.
The root cause: dockerd owns /var/run/docker.sock directly. When the process exits, the socket closes and every connected client gets a connection reset. There is no buffer, no handoff, no grace period. The socket lives and dies with the daemon process.
systemd's socket activation fixes this by owning the socket independently. The socket unit opens /var/run/docker.sock and passes it as fd 3 when dockerd starts. During a restart, the socket stays open and incoming requests queue in the kernel backlog (up to 4096 connections). The new dockerd picks up where the old one left off, and clients never notice the restart.
An Nginx server is handling 10,000 active connections and needs a new config deployed. A full restart would drop every one of those connections, causing user-visible 502 errors for several seconds across all traffic.
A restart kills the process and its listening socket. Every in-flight request -- file downloads, streaming responses, long-polling connections -- is severed immediately. There is no way to drain gracefully because the old process is already dead before the new one starts.
Nginx's SIGHUP-based reload sidesteps this entirely. The master process stays running, re-reads nginx.conf, forks new workers with the updated config, and lets old workers finish their in-flight requests before exiting. Zero connections dropped. Under systemd, run Nginx with daemon off and Type=simple for session isolation and log capture while keeping the SIGHUP reload path.
After an unclean shutdown, the web application starts before PostgreSQL finishes WAL recovery. Every connection attempt fails. The app crashes in a retry loop, gets restarted by its supervisor, crashes again, and triggers a PagerDuty storm -- all because the database was not actually ready yet.
WAL recovery after a crash can take anywhere from milliseconds to 30+ minutes depending on checkpoint interval and write volume. A fixed sleep or PID file poll cannot predict this. Starting dependents too early causes cascading failures; waiting too long wastes minutes of availability on every reboot.
With systemd Type=notify, PostgreSQL calls sd_notify(READY=1) only after recovery completes and the listener socket is open. Dependent services block until that exact moment. No guessing, no timeouts, no race conditions -- even after unclean shutdowns with multi-gigabyte WAL replay.
Same Concept Across Tech
| Technology | How it daemonizes | Key config |
|---|---|---|
| Nginx | Master process daemonizes, spawns workers. systemd unit uses Type=forking | daemon on/off in nginx.conf |
| Node.js | Does not daemonize. Run via systemd Type=simple or PM2 process manager | ExecStart=/usr/bin/node app.js |
| JVM | No built-in daemon support. systemd Type=simple or wrapper (jsvc) | Set -Xmx in Environment= of systemd unit |
| PostgreSQL | pg_ctl starts postmaster which forks. systemd Type=notify (signals readiness) | postmaster sends READY=1 via sd_notify |
| Docker | dockerd is a daemon. Containers run as PID 1 inside namespaces | dockerd managed by systemd on the host |
| Go | Does not fork. Run directly via systemd Type=simple | Simplest integration of any runtime |
Stack layer mapping (service keeps dying):
| Layer | What to check | Tool |
|---|---|---|
| Application | Is the app crashing on startup? Missing config? | journalctl -u service, application logs |
| systemd | Is Type= correct? Is Restart= set? Is After= ordering right? | systemctl cat service, systemctl show service |
| Process | Is the PID file stale? Is another instance running? | Check PIDFile= path, ss -tlnp for port conflicts |
| Resource | Is the service hitting cgroup memory or CPU limits? | systemctl show service --property=MemoryCurrent |
| Permissions | Is the User= able to bind ports, read config, write logs? | journalctl for EACCES or EPERM errors |
Design Rationale Without a service manager, the only way to detach from a terminal was to manually sever every tie -- session, controlling terminal, working directory, file descriptors. The double-fork-and-setsid pattern did this, but it was fragile, and PID-file tracking was trivially escaped by any process that forked again. Cgroup-based tracking replaced it because a cgroup is a kernel boundary that no amount of forking or reparenting can escape. Socket activation addressed a separate pain: when the daemon owns its socket, every restart drops in-flight connections. Let the service manager own the socket instead, and the kernel backlog queues requests transparently while the daemon restarts.
If You See This, Think This
| Symptom | Likely cause | First check |
|---|---|---|
| Service starts then immediately stops | Wrong Type= in systemd unit, or crash on startup | journalctl -u service -n 50 |
| Service dies when SSH session ends | Not daemonized, still attached to terminal | Check if running under systemd or just nohup |
| Service restart loop (NRestarts climbing) | Crash loop with Restart=always | systemctl show service --property=NRestarts,Result |
| Service not starting on boot | Not enabled or wrong WantedBy= target | systemctl is-enabled service |
| Port already in use on restart | Previous instance not fully stopped, or socket lingering | ss -tlnp, check ExecStop= and TimeoutStopSec= |
| Service running but not responding | Process alive but deadlocked or resource-starved | strace -p PID, check cgroup limits |
When to Use / Avoid
Use systemd (modern approach) when:
- Running any long-lived service in production (web servers, databases, agents)
- Need automatic restart on crash (Restart=on-failure)
- Need dependency ordering (After=network.target, Requires=postgresql.service)
- Need resource limits via cgroups (MemoryMax, CPUQuota)
Use manual daemonization when:
- Running on systems without systemd (embedded, minimal containers)
- Need to understand what systemd automates (debugging)
Watch out for:
- Type=forking in systemd when the service does not actually double-fork (causes startup detection failure)
- Type=simple when the service forks to background (systemd tracks the wrong PID)
- Not setting Restart=on-failure (service dies and stays dead with no alert)
Try It Yourself
1 # Check if a process is a daemon (no controlling terminal)
2
3 ps -eo pid,tty,comm | grep -E '\?' | head -20
4
5 # Show systemd service status with cgroup tree
6
7 systemctl status nginx.service
8
9 # View all running services
10
11 systemctl list-units --type=service --state=running
12
13 # Analyze service startup time
14
15 systemd-analyze blame | head -15
16
17 # Check socket activation file descriptors
18
19 systemctl list-sockets
20
21 # Follow service logs in real-time
22
23 journalctl -u nginx.service -f --since '5 min ago'Debug Checklist
- 1
Check service status: systemctl status <service> - 2
View recent logs: journalctl -u <service> -n 50 --no-pager - 3
Check why it failed: systemctl show <service> --property=ExecMainStatus,Result - 4
List all failed services: systemctl --failed - 5
Check service cgroup and resource usage: systemd-cgls -u <service> - 6
Check if process is a daemon (no tty): ps -eo pid,tty,comm | grep '?'
Key Takeaways
- ✓The classic double-fork: (1) fork, parent exits (returns shell prompt), (2) setsid() (new session, no terminal), (3) fork again, first child exits (grandchild cannot accidentally acquire a terminal because it is not a session leader), (4) chdir('/'), umask(0), redirect stdio to /dev/null. Six steps to escape the terminal.
- ✓systemd's Type=simple makes all of this unnecessary. The daemon runs as a foreground process. systemd provides session isolation, stdio redirection to journald, working directory control, and cgroup tracking. No forking. No PID files. No ritual.
- ✓Socket activation is the most underrated feature of systemd. It opens the listening socket and passes it to the daemon as fd 3. The daemon never calls bind/listen. Zero-downtime restarts are free because the socket stays open during restart. Clients queue in the kernel backlog.
- ✓systemd tracks all daemon processes via cgroups, not PIDs or process groups. A daemon that double-forks, reparents children, or creates new sessions cannot escape its cgroup. This is why KillMode=control-group reliably kills everything.
- ✓The second fork in the double-fork pattern has a specific purpose: it prevents the daemon from ever acquiring a controlling terminal. A session leader (created by setsid) can open a tty and get a controlling terminal. The grandchild (from the second fork) is not a session leader, so it cannot.
Common Pitfalls
- ✗Double-forking under systemd with Type=simple. The daemon forks, the original process (which systemd tracks) exits, and systemd thinks the service crashed. If your service double-forks, use Type=forking. If you are writing new code, do not fork at all -- use Type=simple or Type=notify.
- ✗Not redirecting stdin/stdout/stderr to /dev/null. A daemon that inherits open file descriptors from the terminal may write to a closed terminal (SIGPIPE) or block on reads from it. Always dup2 all three to /dev/null in the classic pattern.
- ✗Using a PID file without flock() locking. Two daemon instances can overwrite each other's PID file. Always lock the PID file with flock() or fcntl(). Better yet: use systemd's native cgroup tracking and skip PID files entirely.
- ✗Calling sd_notify(READY=1) before the daemon is actually ready. systemd trusts this signal. If you send it before opening sockets or loading config, dependent services start before yours can serve requests. Only notify after full initialization.
Reference
In One Line
Write services as foreground processes with Type=simple or Type=notify -- systemd handles session isolation, logging, and restarts so the double-fork ritual is dead code.