PAM: Pluggable Authentication Modules
Mental Model
A nightclub with a line of bouncers at the door. First bouncer checks the ID. Second checks the VIP list. Third stamps the hand and sets up the table. Rules are on a clipboard: a bouncer marked "required" means every bouncer still checks the guest after a rejection, so nobody can tell which check failed. A bouncer marked "sufficient" lets a VIP who passes skip the rest of the line entirely. The club never builds its own ID scanner -- the manager swaps in whatever scanner module fits the night, and every entrance runs the same framework.
The Problem
Adding 2FA means patching every application that checks credentials -- SSH, sudo, login, su, screen lock -- unless there is a shared framework. A Docker container hits "too many open files" at 10K connections because the host's pam_limits.so set nofile=1024, and containers blindly inherited it. A destructive sudo command at 3 AM shows uid=0 in the audit log with no way to tell which of three operators typed it. One typo in /etc/pam.d/common-auth locks out SSH, sudo, and console simultaneously -- every authentication path broken at once.
Architecture
Type ssh user@server. A password prompt appears. Enter the password. Access granted.
That interaction involved at least six different shared libraries, four evaluation phases, and a set of control-flow rules that most engineers never think about. The system orchestrating all of it is PAM.
Here is why it matters: without PAM, every program that authenticates users -- SSH, sudo, login, su, screen lock -- would need its own password-checking code. Adding LDAP? Patch every one of them. Adding 2FA? Patch them all again. PAM exists so that applications never touch authentication logic directly. They just ask one question: "is this user legit?"
What Actually Happens
When sshd needs to authenticate a user, the sequence is precise.
Step 1: Transaction begins. sshd calls pam_start("sshd", username, &conv, &pamh). This tells libpam to read /etc/pam.d/sshd, which lists every module to run and in what order.
Step 2: Auth stack fires. pam_authenticate(pamh, 0) executes every module in the auth section, top to bottom. pam_unix.so checks the password against /etc/shadow. pam_google_authenticator.so checks the TOTP code. Each module returns success or an error code.
Step 3: Account check. pam_acct_mgmt asks: is this account locked? Expired? Restricted to certain hours? If the account is valid, proceed.
Step 4: Session setup. pam_open_session runs the session modules. pam_limits.so sets ulimits. pam_loginuid.so stamps the audit UID. pam_systemd.so registers the session with logind and creates a cgroup scope.
Step 5: The shell arrives. On logout, pam_close_session tears everything down.
Under the Hood
The control flags are where PAM gets subtle.
required means the module must succeed, but PAM keeps running the remaining modules regardless. This is not a design oversight. Running all modules prevents timing attacks: an attacker cannot measure whether the failure was fast (bad username) or slow (bad password against /etc/shadow).
requisite is the strict version. Failure stops evaluation immediately and returns to the application. Faster, but it leaks timing information.
sufficient is the early-exit on success. If pam_unix succeeds and no prior required module has failed, PAM returns success immediately without trying pam_ldap or anything else. This is how "local password OR LDAP" fallback chains work.
optional only matters if it is the sole module in the stack. Otherwise its result is ignored.
The conversation function is the bridge between PAM modules and the user interface. When pam_unix needs a password, it sends a PAM_PROMPT_ECHO_OFF message through the conversation callback. For SSH, sshd forwards the prompt over the encrypted channel. For sudo, it reads from the terminal. For a GUI login manager, it shows a dialog. Same PAM module, completely different UIs.
One module deserves special attention: pam_limits.so. It reads /etc/security/limits.conf and applies resource limits via setrlimit(). Common settings: * soft nofile 65536, * hard nproc 4096, @developers hard memlock unlimited. These limits follow every login session -- SSH, sudo, console. And they propagate to containers: Docker inherits them from dockerd's PAM session. This is why Kubernetes node tuning often starts with limits.conf.
Common Questions
How does SSH 2FA work with PAM?
In /etc/pam.d/sshd, configure both auth required pam_unix.so and auth required pam_google_authenticator.so. Both are required, so both password AND TOTP must succeed. In sshd_config, set ChallengeResponseAuthentication yes and AuthenticationMethods keyboard-interactive. The SSH protocol's keyboard-interactive method maps to PAM's conversation function, forwarding the TOTP prompt to the client.
What happens if a PAM module is missing?
If a module's .so file cannot be loaded, PAM treats it as a module failure. With required control, authentication fails but remaining modules still run. A missing pam_unix.so means nobody can authenticate via password. This is why PAM config changes are high-risk operations.
How are PAM failures troubleshot?
Check /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL/CentOS). Use pamtester service username authenticate to test interactively. Verify module paths with ls /lib/security/. Enable debug output by adding debug to the module arguments. And always keep a root shell open while editing.
How does PAM differ from LDAP or Kerberos?
PAM is a framework that chains authentication modules. It does not define a protocol. LDAP, Kerberos, and SAML are protocols. PAM modules wrap them: pam_ldap.so speaks LDAP, pam_krb5.so speaks Kerberos, pam_sss.so supports LDAP + Kerberos + AD. PAM sits at the application boundary. The protocols handle the network communication with the identity provider.
How Technologies Use This
An Nginx container crashes with too many open files when handling 10K concurrent connections, even though the host can handle 1 million file descriptors. The container's Nginx configuration looks correct, and the same setup works fine outside of Docker.
The hidden cause is that the host's PAM pam_limits.so sets the default nofile limit to 1024 when dockerd starts, and every container inherits that value. Containers cannot set their own ulimits from inside because ulimit values are inherited from the parent process at fork time and can only be lowered, not raised.
The fix is either tuning /etc/security/limits.conf on the host to raise the baseline or overriding per-container with --ulimit nofile=65536. Without understanding this PAM inheritance chain, operators chase application bugs that are actually host-level PAM configuration issues.
Elasticsearch refuses to start inside a pod with the error max file descriptors [4096] for elasticsearch process is too low. The pod spec has no ulimit settings, and the Elasticsearch container image works fine on other nodes in the cluster.
The root cause is that the node's PAM pam_limits.so sets a baseline of 4096 via /etc/security/limits.conf. The kubelet and container runtime inherit these limits at startup, and every pod on the node gets the same ceiling. Pod-spec ulimit overrides cannot exceed the hard limit inherited from the container runtime, so even explicit configuration in the pod spec has no effect if the node baseline is too low.
Tuning limits.conf to set * soft nofile 65536 and * hard nofile 131072 raises the baseline for all pods on that node. The lesson is that Kubernetes pod resource limits are bounded by the node's PAM configuration, and node-level tuning must happen before pod-level overrides can take effect.
A destructive sudo rm -rf was run at 3 AM and three operators had root access that night. The audit log shows uid=0 for every sudo command, and there is no way to determine which human actually ran the command.
The problem is that sudo changes the effective UID to 0, and without identity tracking across privilege transitions, every action taken as root looks identical in the audit trail. The original login identity is lost the moment the user escalates privileges.
pam_loginuid.so solves this by writing the original login UID to /proc/self/loginuid at session start. This audit UID is immutable and follows through sudo, su, and setuid transitions, so the audit record shows auid=1003 even when the effective uid is 0. pam_systemd.so then registers the session with systemd-logind and creates a cgroup scope, enabling systemd-cgtop to show resource usage per login session with less than 0.1% overhead.
Same Concept Across Tech
| Concept | Docker | JVM | Node.js | Go | K8s |
|---|---|---|---|---|---|
| Authentication framework | N/A (inherits host PAM) | JAAS (Java Authentication and Authorization Service) | passport.js (Express middleware) | N/A (use OS-level PAM or custom auth) | ServiceAccount tokens + OIDC |
| Pluggable modules | PAM modules in /lib/security/ | JAAS LoginModules | passport strategies | N/A | Webhook token authenticators |
| Resource limits | --ulimit (overrides PAM-inherited limits) | -Xmx, -Xss (JVM-level) | --max-old-space-size (V8) | GOMEMLIMIT (Go 1.19+) | resources.limits in pod spec |
| Session tracking | N/A | N/A | N/A | N/A | Audit logging via API server |
| 2FA integration | N/A | JAAS + custom LoginModule | speakeasy / otplib packages | pquerna/otp library | OIDC provider with MFA |
Stack Layer Mapping
| Layer | PAM Mechanism |
|---|---|
| Application | Calls pam_start(), pam_authenticate(), pam_open_session() via libpam |
| /etc/pam.d/ | Per-service config files defining module chains and control flags |
| libpam | Loads .so modules, executes stacks, enforces control-flag logic |
| PAM modules | pam_unix.so (shadow), pam_ldap.so (LDAP), pam_limits.so (ulimits) |
| System resources | /etc/shadow, /etc/security/limits.conf, LDAP/Kerberos backends |
| Audit trail | /var/log/auth.log, /proc/self/loginuid (pam_loginuid.so) |
Design Rationale
Before PAM, every Unix application -- login, telnet, ftp, su -- carried its own password-checking code, and rolling out a new auth method meant patching each one. The four-stack split (auth, account, password, session) keeps identity verification, account policy, and session setup cleanly separated. Running all "required" modules even after a failure is deliberate: if PAM bailed on the first rejection, the response-time difference would reveal whether the username or the password was wrong.
If You See This, Think This
| Symptom | Likely Cause | First Check |
|---|---|---|
| "too many open files" in containers | pam_limits.so set low nofile on host, inherited by dockerd | cat /etc/security/limits.conf and ulimit -n |
| All authentication paths broken simultaneously | Syntax error in /etc/pam.d/common-auth | Boot rescue, check cat /etc/pam.d/common-auth |
| 2FA TOTP bypasses password check | pam_google_authenticator set as "sufficient" instead of "required" | cat /etc/pam.d/sshd and verify both modules are "required" |
| sudo audit shows uid=0 with no operator identity | pam_loginuid.so not in session stack | Verify session required pam_loginuid.so in PAM config |
| SSH key auth works but password auth fails | PAM auth stack misconfigured (key auth bypasses auth stack) | grep pam_ /var/log/auth.log for specific module failures |
| Account locked after failed attempts | pam_faillock triggered | faillock --user <name> to check and --reset to clear |
When to Use / Avoid
Use when:
- Adding 2FA (TOTP, U2F) to SSH, sudo, or console login without modifying applications
- Centralizing authentication across LDAP, Kerberos, or Active Directory
- Setting per-user or per-group resource limits (nofile, nproc, memlock) via pam_limits.so
- Tracking audit identity across privilege escalation (pam_loginuid.so)
- Implementing account lockout after failed login attempts (pam_faillock)
Avoid when:
- The application handles authentication entirely internally (web apps with JWT/OAuth)
- Container-internal auth that does not interact with the host PAM stack
- Automated service accounts where PAM conversation callbacks add unnecessary complexity
Try It Yourself
1 # View PAM configuration for SSH
2 cat /etc/pam.d/sshd 2>/dev/null || echo "No sshd PAM config found"
3
4 # View PAM configuration for sudo
5 cat /etc/pam.d/sudo 2>/dev/null || echo "No sudo PAM config found"
6
7 # View shared PAM base configurations
8 cat /etc/pam.d/common-auth 2>/dev/null || cat /etc/pam.d/system-auth 2>/dev/null || echo "No common auth config found"
9
10 # List all installed PAM modules
11 ls /lib/x86_64-linux-gnu/security/pam_*.so 2>/dev/null || ls /lib64/security/pam_*.so 2>/dev/null || echo "PAM modules directory varies by distro"
12
13 # Check current ulimits (set by pam_limits.so)
14 ulimit -a
15
16 # View limits.conf (enforced by pam_limits.so in session stack)
17 cat /etc/security/limits.conf 2>/dev/null | grep -v '^#' | grep -v '^$' | head -15
18
19 # Check recent PAM authentication events
20 grep pam_ /var/log/auth.log 2>/dev/null | tail -10 || grep pam_ /var/log/secure 2>/dev/null | tail -10
21
22 # Test PAM authentication (if pamtester is installed)
23 # pamtester sshd $USER authenticate
24
25 # Check failed login attempts (pam_faillock or pam_tally2)
26 faillock --user $USER 2>/dev/null || pam_tally2 --user $USER 2>/dev/null || echo "No faillock/tally tool"Debug Checklist
- 1
cat /etc/pam.d/sshd -- view SSH PAM configuration - 2
cat /etc/pam.d/common-auth -- view shared auth base config - 3
grep pam_ /var/log/auth.log | tail -20 -- recent PAM events - 4
ulimit -a -- show current limits set by pam_limits.so - 5
cat /etc/security/limits.conf | grep -v '^#' -- active limit rules - 6
faillock --user <name> -- check failed login attempt counts
Key Takeaways
- ✓PAM deliberately runs ALL required modules even after one fails. This is not a bug -- it is a timing-attack defense. If PAM stopped early on a bad username, an attacker could measure response time to distinguish "user doesn't exist" from "wrong password." Only 'requisite' short-circuits, and using it carelessly leaks exactly this information.
- ✓The 'sufficient' flag is PAM's fast lane: if pam_unix succeeds (correct local password) and no prior 'required' module has failed, PAM skips everything else and returns success immediately. This is how systems implement "local password OR LDAP" fallback chains -- try local first, fall through to LDAP only if local fails.
- ✓pam_limits.so is the reason your containers have the ulimits they have. It reads /etc/security/limits.conf during the session stack and sets nofile, nproc, memlock via setrlimit(). Docker containers inherit these from dockerd's PAM session unless you override with --ulimit. Kubernetes nodes often need limits.conf tuning for Elasticsearch and other file-hungry pods.
- ✓SSH flows through PAM in a strict order: auth stack (password or TOTP check), account stack (is this account expired or locked?), session stack (set ulimits, register with systemd-logind, set the audit UID). Here is the catch -- key-based SSH auth bypasses the auth stack entirely (sshd handles keys itself) but still runs account and session. Your PAM session modules always fire.
- ✓A PAM misconfiguration can lock you out of a system completely. If both /etc/pam.d/login and /etc/pam.d/sshd are broken, there is no way in -- not console, not SSH. Always keep a root shell open when editing PAM configs, and test with 'pamtester' before deploying. Recovery means booting into single-user mode or mounting the disk from a rescue image.
Common Pitfalls
- ✗Mistake: Setting pam_google_authenticator as 'sufficient' for 2FA. Reality: if sufficient, a correct TOTP code alone grants access WITHOUT checking the password. Anyone who knows the TOTP seed walks right in. For real 2FA, both pam_unix and pam_google_authenticator must be 'required' in the auth stack.
- ✗Mistake: Using 'requisite' instead of 'required' for pam_unix without understanding the tradeoff. Reality: requisite short-circuits on failure, which leaks timing information. An attacker can distinguish "invalid username" (fast return) from "wrong password" (slower /etc/shadow lookup). Use 'required' unless you have a specific reason not to.
- ✗Mistake: Editing /etc/pam.d/common-auth without testing first. Reality: common-auth is @included by sshd, sudo, login, su, and virtually every PAM-aware application. A single syntax error here locks out every authentication path simultaneously. Always validate with 'pamtester sshd youruser authenticate' before saving.
- ✗Mistake: Adding pam_limits.so to the auth stack. Reality: pam_limits is a session module -- putting it in the auth stack does nothing. It only processes limits when called as 'session required pam_limits.so'. Same principle applies to pam_env.so: stack placement determines when it runs.
Reference
In One Line
Let PAM handle identity checks so applications never touch auth logic directly -- and remember that pam_limits.so is where container ulimits actually come from.