Elixir / BEAM

Why BEAM Exists

Ericsson built Erlang in 1986 to run telephone switches. The requirements: handle millions of simultaneous phone calls, each call isolated so one crash does not affect others, and the system must never go down (five nines uptime). The BEAM virtual machine was designed specifically for this: massive concurrency through lightweight processes, fault isolation through no shared memory, and distribution across multiple machines as a first-class feature.

Elixir (created by Jose Valim in 2012) runs on the same BEAM VM but with modern syntax, better tooling, and the Phoenix web framework. Everything that made Erlang good at phone calls makes Elixir good at WebSockets, chat, notifications, and any workload that holds millions of concurrent connections.

Processes Are Not Threads

In Java, each thread uses ~1MB of stack memory. At 1 million concurrent connections, that's 1TB of RAM just for thread stacks. Go is better with goroutines (~8KB each), but even Go needs external coordination (Redis) to track which user is on which server.

BEAM takes a different approach. Each process uses ~2KB of memory. These are not OS threads. The BEAM VM runs its own scheduler that preemptively schedules millions of processes across all CPU cores. Processes share no memory. Communication happens strictly through message passing (the actor model).

A single Elixir server with 64GB RAM can hold 2-5 million processes. Each WebSocket connection is one process. One server does what takes 100+ Java servers.

Java:    1M connections = 100 servers (10K connections each) + Redis for routing
Go:      1M connections = 20 servers (50K connections each) + Redis for routing
Elixir:  1M connections = 1-2 servers (500K-1M connections each), no Redis

The scheduler is preemptive, not cooperative. A slow process cannot block other processes (unlike Go goroutines, which are cooperatively scheduled and can starve others if they don't yield). Every process gets a fair share of CPU time through reduction counting.

Built-In Distribution

BEAM nodes find each other and connect automatically. When two nodes connect, their process registries merge. A message sent to a process on any node in the cluster reaches it by registered name. BEAM routes it to the correct node internally.

# On any node in the cluster:
send({:user, 456}, %{type: "notification", body: "New message"})

# BEAM internally:
# 1. Look up {:user, 456} in the distributed registry
# 2. Find it on node2@10.0.1.5
# 3. Send the message over the TCP link between nodes
# 4. Process on node2 receives it, pushes to WebSocket

This replaces the entire Redis connection registry pattern. No external service to maintain, no extra network hop for lookups, no TTL expiry headaches. The registry updates automatically when a node joins, leaves, or a process crashes.

Supervision Trees and Fault Tolerance

In an Elixir application, every process runs under a supervisor. If a process crashes (bad input, timeout, network error), the supervisor restarts it automatically. Restart happens in milliseconds. No other process is affected because there is no shared memory.

This is the "let it crash" philosophy. Instead of wrapping every function call in try/catch and handling every possible error, let the process crash and restart clean. The supervisor handles recovery. The code stays simple.

A supervision tree is hierarchical. The top-level supervisor watches subsystem supervisors, which watch individual worker processes. A crash at the leaf level restarts only that leaf. A crash in a subsystem supervisor restarts only that subsystem.

WhatsApp used this model to serve 2 billion users with fewer than 50 engineers. Their Erlang servers achieved 99.999% uptime (5 minutes of downtime per year) because individual process crashes never cascaded.

Phoenix Framework for WebSockets

Phoenix, the web framework for Elixir, handles WebSocket connections through its Channels abstraction with built-in lifecycle management: connect, join a topic (chat room, notification stream), send and receive messages, handle disconnection and reconnection.

Phoenix PubSub provides publish/subscribe across all nodes in the cluster. When a message is published to a topic, every subscriber on every node receives it. No external message broker needed for the pub/sub layer.

Phoenix Presence tracks which users are currently online across all nodes. It uses CRDTs internally to merge presence state without conflicts. "Who is online in this chat room" comes for free without building a separate presence service.

Discord runs Phoenix Channels for millions of concurrent users. A single Discord "guild" (server) with thousands of members runs as a set of Phoenix Channel processes across multiple BEAM nodes. Message delivery, typing indicators, presence updates, and voice state all flow through Channels.

Hybrid Architecture: Elixir + Java/Go

The idea is not to rewrite the backend in Elixir. The production pattern is to use Elixir only for the connection and delivery layer while the business logic stays in Java/Go.

The Java/Go backend decides who to notify, builds the payload, applies user preferences, and publishes to Kafka. The Elixir layer consumes from Kafka and delivers notifications through WebSocket connections. The two systems communicate through Kafka (or gRPC). The Java service doesn't need to know Elixir exists.

This is not theoretical. Discord uses Elixir for WebSocket connections with Rust and Python for backend services. Pinterest uses Elixir for notification delivery with Java for notification logic. Bleacher Report uses Elixir for real-time push with Ruby for the main application.

Scaling the Kafka-to-Elixir Bridge

The scaling question: 3 Elixir nodes consuming from a 30-partition Kafka topic. A Kafka message for user 456 lands on Node 3, but user 456's WebSocket is on Node 1. How does the message get from Node 3 to user 456?

Step 1: Elixir nodes form a Kafka consumer group. All Elixir nodes join the same consumer group using Broadway (an Elixir library for Kafka/RabbitMQ consumption). Kafka assigns partitions across nodes automatically: Node 1 gets partitions 0-9, Node 2 gets 10-19, Node 3 gets 20-29.

Step 2: Java/Go publishes a notification to Kafka. The Java backend decides user 456 should receive a notification and publishes {user_id: 456, title: "New message", body: "..."} to the notifications topic. Kafka routes the message to a partition based on the key.

Step 3: An Elixir node consumes the message and routes it. The consuming node looks up user 456 in the distributed process registry and sends the message directly to that process. This is NOT a broadcast to all nodes. It is a direct point-to-point send to one specific process.

# Broadway consumer on whichever Elixir node receives the Kafka message
def handle_message(message) do
  user_id = message.data.user_id   # 456
  payload = message.data            # the notification

  # Look up user 456's process in the cluster-wide registry
  case :pg.get_members(:users, user_id) do
    [pid | _] ->
      # Found the process. send() works across nodes automatically.
      # If pid is on Node 1 and we are on Node 3, BEAM routes
      # the message over the internal TCP link. Direct send, not broadcast.
      send(pid, {:notification, payload})

    [] ->
      # User 456 is offline (no WebSocket process exists).
      # Store the notification for delivery when they reconnect.
      store_for_later(user_id, payload)
  end
end

Step 4: The WebSocket process receives the message and pushes to the user.

# This process runs on whatever node user 456 connected to.
# It was spawned when user 456 opened the app and registered as {:user, 456}.
def handle_info({:notification, payload}, state) do
  # Push through the WebSocket to the user's phone/browser
  push(state.socket, "notification", payload)
  {:noreply, state}
end

This is not broadcast. When Node 3 calls send(pid, message) and pid lives on Node 1, BEAM sends the message directly to Node 1 over the internal TCP connection between nodes. Node 2 never sees it. No node receives messages it does not need.

NOT this (broadcast):
  Node 3 -> send to ALL nodes -> "does anyone have user 456?"

THIS (direct send):
  Node 3 -> registry lookup: "user 456 is pid #xyz on Node 1"
         -> send directly to pid #xyz on Node 1
         -> only Node 1 receives it

Two approaches for partition assignment:

Approach A: Hash-aligned partitioning. Partition Kafka by user_id hash and route WebSocket connections using the same hash. The message always lands on the node holding the connection. No cross-node forwarding needed. Simple but rigid: if a node dies, both Kafka consumers and WebSocket connections must rebalance simultaneously.

Approach B: Any-node consumption with BEAM forwarding (what Discord does). Any node consumes from any Kafka partition. Cross-node forwarding happens through the distributed registry as shown above. This is better because Kafka scaling and WebSocket scaling are independent. Add more Elixir nodes for more connections. Kafka rebalances consumers automatically. BEAM handles cross-node message routing without any external service.

Performance Characteristics

Process creation: ~1-2 microseconds. Context switch between processes: ~0.1 microseconds. Message send between processes on the same node: ~0.1 microseconds. Message send between processes on different nodes (same datacenter): ~100-500 microseconds.

A single BEAM node can handle 2-5 million concurrent WebSocket connections depending on message rate and hardware. At 10,000 messages per second per node (aggregate across all connections), CPU usage stays under 30% on a 16-core machine. The bottleneck is almost always network bandwidth, not CPU or memory.

Phoenix Channels benchmark: a single server has handled 2 million simultaneous WebSocket connections with response times under 1ms per message. This was on a 40-core machine with 128GB RAM.