Firewalls & Security Groups
Firewalls filter packets at every layer — NACLs at the subnet, Security Groups at the instance, iptables at the host, and NetworkPolicies at the pod — implementing defense in depth through stateful and stateless rule evaluation.
The Problem
Every server on a network is exposed to all traffic. Without filtering, there is no boundary. A single compromised host can scan, exploit, and exfiltrate data from every other host it can reach. Firewalls create boundaries — at the network edge, the subnet level, the instance level, and the pod level — implementing defense in depth through layered packet filtering.
Mental Model
Like building security with multiple checkpoints — the front gate (NACL) checks badges, the floor lobby (Security Group) verifies appointments, and each office door (iptables/NetworkPolicy) only opens for specific people. An intruder who passes one checkpoint still faces the next.
Architecture Diagram
How It Works
Packet filtering is the most fundamental security control in networking. Every packet that traverses a network boundary — entering a subnet, reaching an instance, arriving at a pod — passes through rules that decide: allow, deny, or drop. The mechanism varies by layer (NACLs, Security Groups, iptables, NetworkPolicies), but the principle is identical: compare packet headers against a rule set and take action on the first match.
Understanding the layered model is critical because traffic must pass every layer to reach its destination. A Security Group allow is meaningless if the NACL blocks the same traffic at the subnet boundary.
iptables: The Linux Foundation
iptables has been the default Linux packet filter for over two decades. It organizes rules into tables and chains:
Tables determine the type of processing:
filter: Accept or drop packets (the default table)nat: Network address translation (SNAT, DNAT, masquerade)mangle: Modify packet headers (TTL, TOS, mark)raw: Bypass connection tracking for performance
Chains determine when rules are evaluated:
INPUT: Packets destined for the local hostOUTPUT: Packets originating from the local hostFORWARD: Packets being routed through the host (containers, VMs)PREROUTING: Before routing decision (used for DNAT)POSTROUTING: After routing decision (used for SNAT)
# Allow inbound SSH from a specific subnet
iptables -A INPUT -s 10.0.0.0/8 -p tcp --dport 22 -j ACCEPT
# Drop all other SSH attempts
iptables -A INPUT -p tcp --dport 22 -j DROP
# Allow established connections (stateful tracking)
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# Default policy: drop everything not explicitly allowed
iptables -P INPUT DROP
Rule evaluation is sequential. The kernel walks the chain top-to-bottom and executes the first matching rule. This has two implications:
- Put the most specific rules first and the most general (default deny) last.
- At scale (10,000+ rules), iptables becomes a performance bottleneck — every packet walks the entire chain until it matches.
nftables: The Modern Replacement
nftables replaces iptables with a cleaner architecture. The key improvements:
- Unified syntax: One framework for IPv4, IPv6, ARP, and bridge filtering (instead of separate iptables, ip6tables, arptables, ebtables commands).
- Sets and maps: Native data structures for efficient multi-value matching. Instead of 1,000 iptables rules for 1,000 blocked IPs, nftables uses a single rule with a set.
- Atomic rule replacement: Swap entire rule sets in a single kernel transaction — no window where rules are partially applied.
# nftables: block a set of IPs with one rule
nft add set inet filter blocklist { type ipv4_addr \; }
nft add element inet filter blocklist { 192.168.1.100, 10.0.0.50, 172.16.0.99 }
nft add rule inet filter input ip saddr @blocklist drop
Performance at scale is significantly better than iptables because set lookups are O(1) hash-based, not O(n) chain walks.
AWS Security Groups: Stateful Instance Firewalls
Security Groups are the primary network control in AWS. Key characteristics:
Stateful: When an inbound rule allows TCP/443 from 0.0.0.0/0, the corresponding outbound response traffic is automatically allowed. The conntrack module in the hypervisor tracks the connection. This means Security Group rules only need to cover the initiating direction.
Allow-only: There are no deny rules. A Security Group defines what is permitted; everything else is implicitly denied. To explicitly block a specific IP, use a NACL or WAF.
Referencing other Security Groups: Instead of hardcoding IP ranges, rules can reference another Security Group ID. "Allow inbound on port 5432 from sg-webservers" means any instance attached to the webservers Security Group can connect to the database port. When instances scale up or down, the rule adapts automatically.
Inbound Rules (sg-database):
Protocol Port Source
TCP 5432 sg-webservers ← Only web tier can reach the database
TCP 5432 sg-migrations ← CI/CD can run migrations
Outbound Rules (sg-database):
Protocol Port Destination
All All 0.0.0.0/0 ← Default: allow all outbound (stateful return)
AWS NACLs: Stateless Subnet Firewalls
NACLs operate at the subnet boundary and are stateless — they do not track connections. Inbound and outbound rules are evaluated independently. This means:
- Allowing inbound TCP/443 does not automatically allow the response.
- The outbound rule must explicitly allow ephemeral ports (1024-65535) for return traffic.
- Rules are numbered and evaluated in order (lowest number first). The first match wins.
NACLs are coarse-grained controls best used for:
- Emergency IP blocks (add a low-numbered deny rule)
- Subnet-wide restrictions (no SSH from outside the VPC)
- Compliance requirements that mandate subnet-level controls
Kubernetes NetworkPolicy
By default, Kubernetes imposes zero network restrictions between pods. Every pod can reach every other pod in the cluster. NetworkPolicy changes this by defining allowed traffic flows.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-server-policy
namespace: production
spec:
podSelector:
matchLabels:
app: api-server
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app: web-frontend
- namespaceSelector:
matchLabels:
env: production
ports:
- protocol: TCP
port: 8080
egress:
- to:
- podSelector:
matchLabels:
app: database
ports:
- protocol: TCP
port: 5432
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
This policy says: the api-server pod accepts traffic only from web-frontend pods in the production namespace on port 8080, and can only reach the database pod on port 5432 and kube-dns on port 53. Everything else is denied.
Critical detail: the CNI plugin enforces NetworkPolicy, not Kubernetes itself. If the cluster uses a CNI that does not support NetworkPolicy (e.g., basic kubenet, or Flannel without Calico), the NetworkPolicy resource is accepted by the API server but silently ignored. Always verify enforcement with a connectivity test.
Stateful vs Stateless: When It Matters
The stateful/stateless distinction determines how much rule-writing is required and how the system handles return traffic.
| Aspect | Stateful (SG, iptables+conntrack) | Stateless (NACL, basic iptables) |
|---|---|---|
| Return traffic | Automatic | Explicit rule required |
| Rule count | Lower (one direction per flow) | Higher (both directions) |
| Ephemeral ports | Handled by connection tracking | Must allow 1024-65535 |
| Performance overhead | conntrack table memory | None (no state stored) |
| Failure mode | conntrack table overflow under DDoS | Predictable — every packet evaluated independently |
For production infrastructure, stateful filtering is the default at the instance/pod level. Stateless filtering is used at the subnet level as a broad, emergency-capable control that cannot be overwhelmed by connection state exhaustion.
Defense in Depth: Layering All Controls
A properly secured AWS + Kubernetes deployment layers four firewall levels:
- WAF / DDoS protection (L7): Blocks malicious HTTP patterns at the edge.
- NACLs (L3/L4): Subnet-level emergency controls — block entire IP ranges during incidents.
- Security Groups (L3/L4): Instance-level, stateful — defines which services communicate.
- NetworkPolicy (L3/L4): Pod-level — micro-segments east-west traffic within the cluster.
Each layer catches what the previous one missed. A Security Group misconfiguration does not matter if the NetworkPolicy blocks the traffic. A NetworkPolicy bypass does not matter if the Security Group denies the port. Redundancy is the point.
Key Points
- •Stateful firewalls (AWS Security Groups, iptables with conntrack) track connection state. Allowing inbound TCP/443 automatically permits the response packets. Stateless firewalls (NACLs) require explicit rules for both directions.
- •iptables evaluates rules top-to-bottom in each chain. The first matching rule wins. A misplaced ACCEPT above a DROP renders the DROP unreachable — rule ordering is the most common source of firewall misconfigurations.
- •AWS Security Groups are deny-by-default with allow-only rules. It is impossible to write a deny rule in a Security Group. To block specific IPs, use NACLs or WAF.
- •Kubernetes pods are unrestricted by default — every pod can reach every other pod. The first NetworkPolicy applied to a pod activates filtering; from that point, only explicitly allowed traffic passes.
- •Micro-segmentation — applying firewall rules per workload instead of per subnet — reduces the blast radius of a compromised host from the entire network to only the workloads it is explicitly allowed to reach.
Key Components
| Component | Role |
|---|---|
| iptables / nftables | Linux kernel packet filtering framework with chains (INPUT, FORWARD, OUTPUT) and tables (filter, nat, mangle) that evaluate rules sequentially against every packet |
| AWS Security Groups | Stateful, instance-level firewalls that track connection state — if inbound traffic is allowed, the return traffic is automatically permitted without an explicit outbound rule |
| AWS NACLs (Network ACLs) | Stateless, subnet-level firewalls that evaluate inbound and outbound rules independently — both directions must be explicitly allowed for traffic to flow |
| Kubernetes NetworkPolicy | Namespace-scoped rules that restrict pod-to-pod traffic based on label selectors, namespace selectors, and CIDR blocks — enforced by the CNI plugin |
| Connection Tracking (conntrack) | Kernel module that records the state of every active connection — enables stateful firewalls to allow return traffic without explicit rules for the reverse direction |
When to Use
Every production system needs firewall rules at multiple layers. Use Security Groups for instance-level controls in AWS. Use NACLs for subnet-wide emergency blocks (IP bans). Use Kubernetes NetworkPolicy in every namespace with default-deny. Use iptables/nftables for host-level hardening on bare-metal or VMs.
Tool Comparison
| Tool | Type | Best For | Scale |
|---|---|---|---|
| iptables | Open Source | Traditional Linux packet filtering with mature tooling and documentation — still the default on most distributions, though being replaced by nftables | Small-Enterprise |
| nftables | Open Source | Modern replacement for iptables with better performance, unified syntax for IPv4/IPv6/ARP, and native set/map support for efficient rule matching | Small-Enterprise |
| AWS Security Groups | Managed | Stateful instance-level firewall integrated into AWS VPC — no agents to manage, automatic connection tracking, supports referencing other Security Groups as sources | Small-Enterprise |
| Calico NetworkPolicy | Open Source | Kubernetes-native and extended network policy enforcement using iptables or eBPF — supports global policies, DNS-based rules, and application layer filtering | Medium-Enterprise |
Debug Checklist
- Check Security Group rules: aws ec2 describe-security-groups --group-ids sg-xxx shows all inbound/outbound rules — look for 0.0.0.0/0 on unexpected ports.
- Trace iptables evaluation: iptables -t filter -L -v -n --line-numbers shows hit counts per rule — a rule with zero hits is either unreachable or the traffic pattern is wrong.
- Test connectivity through firewalls: nc -zv <host> <port> from the source confirms whether the port is reachable or blocked.
- Inspect Kubernetes NetworkPolicy: kubectl describe networkpolicy -n <namespace> shows which pods are selected and what ingress/egress rules are applied.
- Check conntrack table: conntrack -L shows active tracked connections — use conntrack -D to delete stale entries causing asymmetric routing issues.
Common Mistakes
- Opening port 0.0.0.0/0 in a Security Group for debugging and forgetting to remove it. This exposes the instance to the entire internet and is the leading cause of cloud breaches.
- Adding iptables rules to the wrong table. Filtering rules belong in the 'filter' table, not 'nat' or 'mangle.' Rules in the wrong table either have no effect or break NAT/routing.
- Assuming Kubernetes NetworkPolicy works without a supporting CNI. The default kubenet CNI does not enforce NetworkPolicy. Calico, Cilium, or a similar policy-aware CNI must be installed.
- Creating overlapping Security Group and NACL rules that conflict. Traffic must pass both — NACLs are evaluated first at the subnet level, then Security Groups at the instance level. A NACL deny overrides a Security Group allow.
- Not accounting for ephemeral ports in stateless NACLs. Outbound connections use random high ports (1024-65535). A NACL allowing only port 443 outbound blocks the return traffic from any connection initiated by the instance.
Real World Usage
- •AWS enforces Security Groups at the hypervisor level, not inside the instance. Even if an attacker gains root on an EC2 instance, they cannot disable or modify the Security Group rules — the filtering happens before traffic reaches the VM.
- •Stripe uses micro-segmentation extensively — every service has a dedicated Security Group that allows only the specific ports and source services required, preventing lateral movement if one service is compromised.
- •Kubernetes clusters in production run with default-deny NetworkPolicies in every namespace. New services must explicitly declare which pods they communicate with, enforcing the principle of least privilege at the network layer.
- •Cloudflare applies iptables rules at the edge to implement rate limiting and IP blocking for DDoS mitigation, processing millions of rules per second using ipset for O(1) lookups.