BGP (Border Gateway Protocol) is the routing protocol that connects the internet's autonomous networks. Every Google search, Netflix stream, and DNS lookup depends on BGP to find a path between networks. This post covers how BGP works, its core components, and where it's used at scale.

🧩 The Building Blocks: Core Components of BGP

1. Autonomous Systems (AS)

The internet is made up of thousands of independently operated networks, each called an Autonomous System (AS). Each AS controls its own routing policy.

Examples of Autonomous Systems:

• Internet service provider (like Airtel, Comcast, or AT&T)
• Big tech companies (like Google or Meta)
• Universities
• Government agencies

Each AS has a unique identification number, called an AS Number (ASN). For example, Google's ASN is 15169, while Facebook's is 32934.

2. BGP Speakers

A BGP speaker is a router that runs the BGP protocol. These routers sit at the edges of Autonomous Systems and exchange routing information with BGP speakers in neighboring ASes.

3. BGP Sessions

When two BGP speakers from different ASes need to exchange routing information, they establish a BGP session — a persistent TCP connection used to share and update path information.

4. BGP Tables

Each BGP speaker maintains a Routing Information Base (RIB) — a table of all known network destinations and the best paths to reach them. The RIB is the basis for all routing decisions.

🧭 How BGP Routers Communicate

1. Updates and Withdrawals

BGP routers communicate through several message types, but two are particularly important:

1. UPDATE Messages: These are like announcements saying "here's a new path to reach a destination"
2. WITHDRAWAL Messages: These are like announcements saying "this path is no longer available"

A typical BGP UPDATE message:

BGP UPDATE
  NLRI (New route being advertised):
    8.8.8.0/24
  AS_PATH:
    15169
  NEXT_HOP:
    203.0.113.1

This message says: "To reach Google's DNS network (8.8.8.0/24), send traffic to 203.0.113.1. This route was learned directly from Google (AS 15169)."

When a network becomes unreachable, BGP routers send WITHDRAWAL messages. For example:

BGP WITHDRAWAL
  Withdrawn Routes:
    8.8.8.0/24

This tells neighboring routers: "The path to Google's DNS (8.8.8.0/24) is no longer available."

2. Speaking BGP: The Key Attributes

BGP routers exchange several types of information to make smart routing decisions:

2.1 AS_PATH An ordered list of ASes that a route traverses. If a route has an AS_PATH of "15169 7018 3356", the data crosses three networks to reach its destination.

AS_PATH serves two purposes:

• Loop prevention: If a router sees its own ASN in the path, it rejects the route.
• Path selection: Routers prefer shorter AS paths (fewer hops).

2.2 NEXT_HOP The IP address of the next router that should receive traffic for a given destination. Each BGP speaker forwards packets to its NEXT_HOP, which then forwards to the next, and so on until the destination is reached.

2.3 MED (Multi-Exit Discriminator) When an AS has multiple entry points, MED tells neighboring networks which entry point to prefer. The originating AS sets the MED value — lower values are preferred. If entry point A has MED 10 and entry point B has MED 100, traffic uses entry point A.

2.4 COMMUNITY BGP communities are tags attached to routes that signal special handling. For example, a "NO-EXPORT" community tells a router not to advertise the route to external peers. Communities enable routing policy coordination between networks without direct configuration.

🏢 Points of Presence (PoPs)

1. What Is a Point of Presence?

A Point of Presence (PoP) is a physical facility where network equipment is deployed — routers, servers, and interconnects. PoPs are the locations where BGP peering actually happens and where traffic is exchanged between networks.

Major tech companies operate PoPs globally:

• Google has PoPs across six continents, so you can search quickly from anywhere
• Cloudflare operates in over 300 cities globally
• Akamai has deployed thousands of servers in 136 countries

2. Inside a PoP

A typical PoP contains:

• High-capacity routers running BGP
• Cache servers storing popular content for faster delivery
• Connections to multiple ISPs and peering partners
• Redundant power and cooling systems

🌐 Anycast: One IP Address, Many Locations

Anycast allows multiple servers in different locations to share the same IP address. When a client sends a request, BGP routes it to the nearest server in terms of network distance (not geographic distance). The client doesn't know — or need to know — which specific server handles the request.

1. How Anycast Works with BGP

Multiple servers across the globe share a single IP address (e.g., 192.0.2.1). Each server's PoP advertises the same prefix via BGP. Routers naturally select the closest server based on AS path length and other BGP attributes.

2. The Technical Implementation

Two elements make anycast work:

1. Special Server Configuration: On each server, the anycast IP is configured on a virtual interface rather than a physical network card:

1. BGP Advertisements: Each location tells its neighbors, "I can receive traffic for this IP address."

2.1 Why Use a Loopback Interface?

The loopback interface provides stability:

• Survives physical failures: If a network cable is unplugged, the loopback interface stays up
• Service continuity: Services remain accessible during network changes
• Consistent identity: The server maintains the same IP regardless of which physical interface is active

Loopback Interfaces and ARP

The loopback interface plays a crucial role in making anycast work properly:

1. Independence from Physical Hardware: If you assigned the anycast IP (like 8.8.8.8) directly to a physical network interface, the IP would disappear whenever that physical interface failed. By using a loopback interface, the IP remains available even if all physical network interfaces change or fail.

2. No ARP Participation: ARP (Address Resolution Protocol) is used on local networks to map IP addresses to physical MAC addresses. Loopback interfaces don't participate in ARP because they're virtual and not tied to physical hardware.

3. The /32 Subnet Mask: When configuring an anycast IP with a /32 subnet mask (ip addr add 192.0.2.1/32 dev lo), you're explicitly telling the system "don't use ARP for this IP." This is critical for anycast because:

   • It prevents ARP broadcasts that could cause conflicts
   • It makes the IP reachable only through explicit routing (via BGP)
   • It isolates the IP from the local network's broadcast domain

Without these properties, anycast would cause network conflicts when multiple servers tried to claim the same IP address on a shared network. The loopback interface effectively creates an isolated environment for the anycast IP on each server.

2.2 How Anycast Avoids Address Conflicts

Multiple servers sharing an IP address doesn't cause conflicts because:

• The servers are in different locations, never on the same L2 network
• Each server is unaware of the others
• BGP routing ensures each request reaches exactly one server
• The /32 loopback configuration prevents ARP conflicts on local networks

🔍 Case Study: Google's 8.8.8.8 DNS Service

Google's Public DNS (8.8.8.8) is one of the best-known anycast deployments, handling billions of queries daily.

1. Getting the Address

Before Google could launch its public DNS service in 2009, it needed a memorable IP address.

Google acquired the 8.8.8.0/24 block through the official internet address registry (ARIN), making them the legitimate owner of the 8.8.8.8 address.

2. Setting Up Servers Worldwide

Google placed DNS servers in data centers all around the world:

• New York
• London
• Tokyo
• Mumbai
• Sydney
• And dozens more locations

At each location, Google configured its servers with the same IP address:

3. Telling the Internet About It

Each Google location then advertised via BGP: "Send 8.8.8.8 traffic to me!"

• The New York location told North American internet providers
• The London location told European internet providers
• The Tokyo location told Asian internet providers
• And so on...

4. How It Works for You

When you set your device to use 8.8.8.8 as your DNS server:

1. Your DNS question (like "What's the IP address for netflix.com?") is sent to 8.8.8.8
2. BGP routing ensures it reaches the Google DNS server nearest to you
3. That server answers your question and sends back the information
4. You get a fast response because you're talking to a nearby server
5. If that server has problems, your queries automatically get sent to the next best option

Users are transparently routed to different physical servers at different times — the experience is consistent because BGP always selects the nearest available server.

🚀 Content Delivery Networks: BGP at Scale

CDNs like Cloudflare, Akamai, and Fastly apply anycast at scale, routing users to the nearest edge server for web content delivery.

1. How CDNs Use BGP and Anycast

CDNs use anycast IP addressing and BGP routing to:

• Reduce latency: Route users to the nearest edge server
• Distribute load: Spread traffic across data centers so no single server is overwhelmed
• Ensure availability: If one location fails, traffic automatically shifts elsewhere
• Absorb DDoS attacks: Distribute malicious traffic across the entire network

2. Multi-Homed Connectivity

CDN servers typically connect to multiple ISPs. If one upstream link fails, traffic reaches the server via alternative paths.

3. What Happens When Something Breaks

If a CDN server has problems:

1. It stops advertising its route via BGP
2. Other routers quickly update their directions
3. Traffic automatically flows to other working servers
4. Users barely notice any disruption

This automatic failover is a core property of BGP-based architectures.

🔄 BGP vs. Traditional Load Balancing

While BGP helps distribute traffic across the internet, it differs significantly from traditional load balancers:

1. Scale and Scope

Traditional load balancers operate within a single data center or region, distributing traffic among a cluster of servers. BGP operates at global internet scale, routing traffic between entire networks.

2. Decision-Making Criteria

Traditional load balancers make decisions based on:

• Server health checks
• Current server load
• Round-robin or least connections algorithms
• Application-layer information (URLs, cookies)

BGP makes routing decisions based on:

• Network topology
• AS path length
• Peering relationships
• Network policies

3. Implementation Level

Traditional load balancers function at layers 4-7 of the OSI model (transport and application layers), while BGP operates at layer 3 (network layer).

4. Failover Speed

Traditional load balancers can detect and respond to failures within seconds. BGP convergence after route changes can take minutes, though modern implementations have significantly improved this.

5. Configuration Control

With traditional load balancing, a single organization controls all configuration. With BGP, routing decisions are influenced by policies set by multiple autonomous systems across the internet.

6. Use Cases

Traditional load balancing excels at:

• Distributing application traffic within a data center
• SSL termination
• Application-aware routing
• Health monitoring of specific services

BGP excels at:

• Global traffic distribution
• Multi-region failover
• Network-level traffic engineering
• Internet-scale resilience

In practice, most organizations use both: traditional load balancers within data centers, and BGP for global routing and multi-region failover.

🔌 Beyond DNS and CDNs: Other BGP Applications

BGP is used across many domains beyond DNS and CDN routing:

1. DDoS Protection and Mitigation

During a DDoS attack, the target announces more specific BGP routes for the attacked IP prefixes, redirecting all inbound traffic to scrubbing centers. These centers filter out malicious traffic and forward only clean traffic to the origin server. This technique (BGP-based remote-triggered blackhole or traffic diversion) is the standard approach used by services like Cloudflare, Akamai Prolexic, and AWS Shield.

2. VPNs (Virtual Private Networks)

Enterprises with multiple offices set up encrypted VPN tunnels between sites and run BGP inside those tunnels. BGP exchanges internal routes across the VPN, so each office router automatically learns how to reach subnets at other sites. MPLS VPNs from ISPs use BGP (specifically MP-BGP with VPNv4 address families) to maintain routing isolation between customers.

3. Multi-homing for Enterprises

When a business connects to two or more ISPs (multi-homing), BGP runs between the enterprise and each ISP. If one link fails, BGP automatically reroutes traffic through the remaining connections. BGP also enables policy-based traffic engineering — for example, preferring one ISP for latency-sensitive traffic and another for bulk transfers.

4. Internet Exchange Points (IXPs)

IXPs are physical locations where networks peer directly, exchanging traffic locally instead of routing through upstream transit providers. BGP runs between all connected networks at the IXP, enabling direct route exchange. This reduces latency, lowers transit costs, and improves performance for traffic between peering partners.

5. IPv6 Deployment

MP-BGP (Multiprotocol BGP) extends BGP to advertise routes for both IPv4 and IPv6 simultaneously. This allows networks to incrementally adopt IPv6 while maintaining full IPv4 connectivity — both address families coexist in the same BGP sessions.

⚠️ When BGP Goes Wrong: Security Concerns and Notable Incidents

Despite its importance, BGP was designed when the internet was smaller and more trusting. This has led to some significant problems.

1. The Problem: Route Hijacking

BGP route hijacking occurs when a network announces prefixes it doesn't own, diverting traffic to the wrong destination. Because BGP was designed without built-in authentication, routers accept route announcements on trust.

2. Famous Incidents

2.1 Pakistan Telecom / YouTube (2008)

In February 2008, Pakistan Telecom tried to block YouTube within Pakistan by redirecting traffic. But they accidentally announced this redirection to the whole world, causing a global YouTube outage for about two hours as traffic was mistakenly sent to Pakistan Telecom.

2.2 Google Traffic Hijack (2018)

In November 2018, traffic to Google services was temporarily rerouted through Russia and China for about 90 minutes. This affected Google Search, Gmail, and Google Cloud. While no data was reported stolen, it showed how BGP can be vulnerable.

3. Security Measures

The networking community has developed several protective measures:

1. RPKI (Resource Public Key Infrastructure): A system that uses digital signatures to prove who rightfully owns which IP addresses

2. Filtering: Internet providers maintain lists of which routes their customers can legitimately advertise

3. Monitoring: Special services watch for suspicious changes in global routing

🔮 The Future of Internet Routing

Active areas of BGP development:

• RPKI adoption: More networks are deploying Route Origin Validation to cryptographically verify route announcements
• ASPA (AS Provider Authorization): A complement to RPKI that validates the AS path, not just the origin
• BGP Flowspec: Enables automated, fine-grained traffic filtering to respond to DDoS attacks in real time
• Anomaly detection: BGP monitoring platforms (RIPE RIS, RouteViews, Cloudflare Radar) provide real-time visibility into routing changes

BGP remains the foundation of internet routing. Its security model is catching up through incremental deployment of RPKI and filtering best practices, but the protocol itself will continue to be how autonomous networks discover paths to each other.