Service Mesh Networking

How It Works

A service mesh is an infrastructure layer that handles service-to-service communication. Instead of each microservice implementing its own retry logic, TLS, and observability, the pattern deploys a proxy sidecar next to each service instance. These proxies form the data plane — they intercept every network call in and out of the service. A separate control plane configures all these proxies centrally.

The magic is in the interception. When Service A wants to call Service B, the request never goes directly to B. Instead, iptables rules (or eBPF hooks in newer implementations) redirect outbound traffic from A to A's sidecar proxy. The proxy applies retry policies, establishes an mTLS connection to B's sidecar proxy, and forwards the request. B's sidecar then forwards to the actual service. Neither Service A nor Service B knows the proxy exists.

The Data Plane: Envoy and Friends

Envoy is the dominant data-plane proxy, originally built at Lyft and now a CNCF graduated project. Every major service mesh (Istio, Consul Connect, AWS App Mesh) uses Envoy or a derivative. Linkerd is the exception — it uses a purpose-built Rust proxy called linkerd2-proxy that trades Envoy's flexibility for dramatically lower resource usage.

What the sidecar proxy handles on every request:

Outbound (from the service):
  1. Service A sends request to service-b:8080
  2. iptables REDIRECT → sidecar proxy (port 15001)
  3. Proxy resolves service-b via service discovery
  4. Proxy applies retry policy (e.g., 3 retries on 503)
  5. Proxy applies timeout (e.g., 5s per attempt)
  6. Proxy establishes mTLS to Service B's sidecar
  7. Proxy applies traffic split (e.g., 95% v1, 5% v2)
  8. Request reaches Service B's sidecar

Inbound (to the service):
  1. Service B's sidecar receives mTLS connection
  2. Validates client certificate (is Service A allowed?)
  3. Applies rate limiting / authorization policies
  4. Forwards to localhost:8080 (the actual service)
  5. Emits metrics, traces, and access logs

The Control Plane: Configuration at Scale

The control plane is the brain. In Istio, it is called istiod — a single binary that handles configuration distribution, certificate management, and service discovery. In Linkerd, it is linkerd-control-plane. The control plane does not touch any data traffic — it only pushes configuration to the proxies.

Key control plane responsibilities:

Certificate Authority: The control plane runs an internal CA that issues short-lived X.509 certificates to every sidecar. Certificates are typically rotated every 24 hours automatically. This means mTLS happens without any developer effort — no certificate management, no key rotation scripts, no TLS configuration in application code.

Configuration Distribution: When an engineer creates an Istio VirtualService or a Linkerd TrafficSplit, the control plane translates that into Envoy configuration and pushes it to every relevant proxy via the xDS (discovery service) API. This happens in seconds.

Service Discovery: The control plane watches the Kubernetes API server for new pods and endpoints. When Service B scales from 3 to 10 replicas, the control plane immediately updates all proxies with the new endpoints.

Traffic Management: Beyond Basic Load Balancing

This is where service meshes shine compared to basic Kubernetes services. A mesh provides fine-grained traffic control without touching application code:

Canary Deployments: Route 1% of traffic to v2 of the service. Monitor error rates. If healthy, increase to 5%, 10%, 50%, 100%. If errors spike, route back to 0%.

# Istio VirtualService — canary with 5% traffic to v2
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-service
spec:
  hosts:
    - payment-service
  http:
    - route:
        - destination:
            host: payment-service
            subset: v1
          weight: 95
        - destination:
            host: payment-service
            subset: v2
          weight: 5

Circuit Breaking: When an upstream service starts failing, the proxy stops sending it traffic after a threshold is breached. This prevents cascading failures where one failing service takes down everything that depends on it.

Fault Injection: Intentionally inject delays or errors to test resilience. Want to know what happens when the payment service has 500ms latency? Inject it at the mesh level without touching a line of code.

Sidecar vs Ambient Mesh

The sidecar model has a cost: every pod gets an extra container consuming CPU and memory. For a cluster with 1,000 pods, that is 1,000 Envoy instances, each using 50-100MB of RAM. On top of that, every request pays a latency tax — traffic goes through two extra network hops (into sidecar, out of sidecar) per service call.

Ambient mesh is Istio's answer to this problem. It splits the mesh into two layers:

• ztunnel (zero-trust tunnel): A per-node DaemonSet handling L4 concerns — mTLS, telemetry, and simple authorization. This replaces the sidecar for workloads that only need encryption and identity.
• Waypoint proxies: Shared, per-namespace Envoy instances handling L7 concerns — HTTP routing, retries, header-based policies. Only deployed when L7 features are needed.

The result: most workloads get mTLS and basic observability with zero sidecar overhead. Only the services needing advanced L7 features pay for a waypoint proxy, which is shared across the namespace.

Traditional Sidecar Model:
  [Pod A + Envoy] → [Pod B + Envoy] → [Pod C + Envoy]
  Cost: N pods × ~80MB RAM = significant overhead

Ambient Mesh Model:
  [Pod A] → [ztunnel (node)] → [ztunnel (node)] → [Pod B]
  Cost: N nodes × ~40MB RAM = much lower overhead
  
  With L7 (optional):
  [Pod A] → [ztunnel] → [Waypoint Proxy] → [ztunnel] → [Pod B]

Observability: The Free Lunch

One of the most immediate benefits of a service mesh is observability without code changes. Because every request flows through a proxy, the mesh can automatically generate:

• Golden signal metrics: Request rate, error rate, and latency (p50, p99) for every service-to-service edge. These appear in Prometheus/Grafana instantly.
• Distributed traces: The proxy adds trace headers (B3, W3C Trace Context) to every request, enabling end-to-end tracing through Jaeger or Zipkin.
• Access logs: Every request logged with source, destination, response code, latency, and TLS status.
• Service topology maps: The mesh knows every service-to-service connection, generating real-time dependency graphs.

This is not just nice to have — it is transformative. Before a mesh, getting consistent observability across 50 services written in 5 languages required instrumenting every single one. With a mesh, deploy it and observability appears everywhere immediately.

When a Mesh Is Overkill

Not every organization needs a service mesh. Here is a practical decision framework:

Situation	Recommendation
< 10 services, single language	Use gRPC interceptors or HTTP middleware. No mesh needed.
10-50 services, need mTLS	Linkerd is the sweet spot — minimal overhead, fast setup.
50-200 services, multi-cloud	Istio with ambient mesh or Consul Connect for VM support.
200+ services, strict compliance	Full Istio with custom authorization policies and audit logging.
High-performance / low-latency	Cilium Service Mesh (eBPF-based, no sidecar for L3/L4).

The honest truth: most teams adopt a service mesh too early. Start with a simple service mesh like Linkerd when the pain of managing mTLS and retries across many services becomes real. Do not install Istio because a blog post said so — it is powerful but operationally complex, and without a genuine need for traffic splitting or advanced authorization policies, that complexity is pure cost.