DPDK (Data Plane Development Kit)

Why It Exists

XDP is fast because it hooks into the NIC driver before the kernel networking stack. But it still runs inside the kernel. The packet still goes through the driver. eBPF programs still have size limits and restricted operations. For most workloads, XDP is more than enough.

But some workloads need more. An observability collector gateway receiving telemetry from 3,000 agent nodes. A telecom virtual network function processing subscriber traffic. A financial exchange handling market data feeds. These systems need to process 15-20M+ packets per second on a single box. Even XDP starts to hit limits at that scale.

DPDK takes the opposite approach from XDP. Instead of making the kernel faster, it removes the kernel from the packet path entirely. The NIC is unbound from the kernel driver and handed to a userspace driver (called a Poll Mode Driver, or PMD). The application talks directly to the NIC hardware through memory-mapped hugepages. No syscalls. No interrupts. No context switches. No socket buffers. Nothing between the code and the wire except a thin hardware abstraction layer.

The tradeoff is real. DPDK is not a drop-in optimization. It is a different programming model. The kernel networking stack is lost completely on DPDK-bound interfaces. No TCP. No iptables. No tcpdump. The packet processing pipeline is built from scratch using DPDK's libraries for memory management, ring buffers, and packet parsing.

How It Works

Normal Linux networking:

Packet arrives → NIC raises hardware interrupt → Kernel driver copies packet to SKB
→ Kernel processes through netfilter/routing/TCP → Copies data to userspace via syscall
→ Application reads from socket

Each step: context switch, memory copy, lock acquisition
At 1M packets/sec: ~1M interrupts + ~1M syscalls + ~2M memory copies per second

DPDK networking:

Packet arrives → NIC writes packet directly to hugepage memory (DMA)
→ Application polls the NIC ring buffer in a tight loop
→ Application reads packet directly from hugepage memory

No interrupts. No syscalls. No copies. No kernel involvement.
Just the application and the NIC, talking through shared memory.

The Key Components

Poll Mode Driver (PMD). Instead of the kernel driver handling the NIC, DPDK provides a userspace driver. The PMD runs in a dedicated CPU core and continuously polls the NIC's receive ring for new packets. This eliminates interrupt overhead but means the core runs at 100% CPU even when there are no packets. That is the fundamental tradeoff.

Hugepages. DPDK uses 1GB or 2MB hugepages for all packet buffers. Normal 4KB pages would cause too many TLB misses at high packet rates. With hugepages, the TLB can map the entire packet buffer pool with a few entries.

Memory pools (mbufs). Pre-allocated pools of packet buffers. No malloc/free in the hot path. The application grabs a buffer from the pool, processes the packet, and returns the buffer. Zero allocation overhead.

Ring buffers (rte_ring). Lock-free multi-producer/multi-consumer queues for passing packets between cores. DPDK applications typically run a pipeline: one core receives packets, another core processes them, a third core transmits. Ring buffers connect the stages without locks.

Real Example: Observability Collector Gateway

3,000 servers each run eBPF agents that collect metrics and ship them via XDP. All that traffic converges on a few collector gateway boxes. Each gateway receives telemetry from 1,000 agent nodes. At 5,000 packets/sec per agent, that is 5M packets/sec hitting a single box.

Without DPDK, the kernel networking stack on the collector maxes out at 1-2M pps. Packets queue up, latency spikes, and the telemetry pipeline falls behind.

With DPDK on the collector:

3,000 agent nodes
  ↓ (XDP fast egress, 5K pps each)
Collector Gateway (DPDK)
  ↓ PMD polls NIC: 5M packets/sec, no problem
  ↓ Application parses OTLP protobuf from hugepage memory
  ↓ Writes batches to Kafka producer
  ↓
Kafka → Flink → Storage

The collector dedicates 2 cores to DPDK polling (handling 5M pps total), 2 cores to parsing and batching, and the rest to the Kafka producer and application logic. A single 16-core box handles what would otherwise require 5-8 boxes using kernel networking.

Production Considerations

• Separate management NIC. DPDK takes over the data NIC completely. A second NIC on the kernel stack is needed for SSH, monitoring agents, health checks, and everything else that expects normal sockets.
• Hugepage reservation. Reserve hugepages at boot via kernel command line (hugepagesz=1G hugepages=4). Trying to allocate later often fails due to memory fragmentation.
• CPU isolation. Use isolcpus to keep the OS scheduler off the DPDK cores. Without this, the scheduler occasionally migrates processes onto polling cores and causes latency spikes.
• NUMA awareness. Bind DPDK cores and memory to the same NUMA node as the NIC. Cross-NUMA memory access adds 50-100ns per packet, which destroys throughput at scale.
• Graceful degradation. If DPDK crashes, the NIC goes dark (no kernel fallback). Build a watchdog that detects DPDK process death and either restarts it or fails over traffic to a backup collector.

Failure Scenarios

Scenario 1: DPDK Process Crash. The DPDK application segfaults due to a buffer overflow in the packet parser. Because the NIC is bound to the userspace driver, no kernel driver takes over. The NIC simply stops processing packets. All telemetry from 1,000 agent nodes is silently dropped. There are no kernel logs because the kernel does not know about the NIC. Detection: upstream agents detect that the collector is not acknowledging batches. The health check on the management NIC (separate interface) reports the DPDK process as down. Recovery: restart the DPDK process, which re-attaches to the PMD. Packet loss during the restart window (typically 2-5 seconds). Prevention: run DPDK behind a process supervisor (systemd with Restart=always), deploy at least 2 collector gateways with agent-side failover.

Scenario 2: Hugepage Exhaustion. The DPDK memory pool runs out of mbufs because the application is processing packets slower than they arrive. New packets cannot be received because there are no free buffers. The NIC's hardware ring fills up and packets get dropped at the NIC level. Detection: DPDK counters show rx_nombuf increasing. Prevention: size the mbuf pool for peak traffic, not average. Add backpressure signaling so upstream agents slow down when the collector is full. Monitor mbuf pool utilization and alert at 80%.

Scenario 3: Core Starvation. Someone deploys a new service on the same box and it consumes CPU that DPDK polling cores need. The polling loop slows down. Packets queue in the NIC hardware ring. At 5M pps, even a 10ms stall means 50,000 packets buffered, exceeding the ring size, and packets drop. Detection: DPDK latency metrics spike. rx_missed counters increase on the NIC. Prevention: isolcpus on DPDK cores so the scheduler cannot touch them. Containers running on the box should have CPU pinning that excludes DPDK cores.

Capacity Planning

Metric	DPDK	XDP (native)	Kernel Stack
Packets/sec per core	15-20M	5-10M	0.5-1M
Latency per packet	< 1 μs	1-5 μs	20-50 μs
CPU model	Dedicated polling (100%)	In-line (proportional)	Interrupt-driven
Kernel tools available	None on data NIC	All	All
Deployment complexity	High	Medium	Low

Real-world reference numbers: Intel benchmarks show DPDK forwarding 80M 64-byte packets/sec on a single server with multiple cores. Mellanox (NVIDIA) ConnectX-6 NICs achieve 200Gbps line rate with DPDK. Telecom operators run virtual firewalls and load balancers at 40Gbps+ using DPDK-based VNFs.

Sizing formula for collector gateways: required_polling_cores = ceil(total_ingest_pps / 8M). At 5M pps from 1,000 agents: ceil(5M / 8M) = 1 core. At 15M pps from 3,000 agents: ceil(15M / 8M) = 2 cores. Add 2 cores for parsing/batching and 2 for Kafka production. A single 8-core box handles 3,000 agents worth of telemetry. Without DPDK, 5-8 boxes would be needed to absorb the same traffic through kernel networking.

Architecture Decision Record

ADR: When to Use DPDK vs XDP vs Kernel Stack

Context: Deciding how to handle network traffic at a specific point in the architecture. The wrong choice either wastes engineering effort (DPDK where XDP would suffice) or creates a bottleneck (kernel stack where DPDK is needed).

Criteria (Weight)	Kernel Stack	XDP	DPDK
Packets/sec (30%)	< 1M	1-10M	> 10M
Operational cost (25%)	Low	Medium	High
Kernel tool access (20%)	Full	Full	None on data NIC
App changes needed (15%)	None	Minimal (attach XDP prog)	Major (rewrite networking)
CPU cost (10%)	Proportional	Proportional	Dedicated cores (100%)

Decision framework:

• Application servers, API backends, normal services. Kernel stack. Do not over-engineer the network path. Most services never exceed 100K pps.
• Agent nodes shipping telemetry, DDoS edge boxes, software load balancers under 10M pps. XDP. Lightweight, no dedicated cores, works with the existing eBPF toolchain. Delivers 5-10x improvement with low complexity.
• Collector gateways receiving fan-in traffic from hundreds or thousands of sources. Telecom NFV. Financial market data. Anything above 10M pps. DPDK. The operational cost is justified because these are dedicated boxes with a single job: move packets as fast as possible. There are 3-5 of these boxes, not 3,000.
• The hybrid approach (recommended for observability). XDP on every agent node (3,000 boxes, lightweight, no ops overhead) plus DPDK on a few collector gateways (3-5 boxes, dedicated, high throughput). Best of both worlds. The full pipeline: eBPF (collect) → XDP (fast egress) → DPDK (high-throughput ingestion) → Kafka.