How Cloudflare Handles 20% of Global Internet Traffic: Anycast, XDP & eBPF Under the Hood

When you type a URL into your browser, there is a 20% probability that the TCP handshake terminates on a Cloudflare edge server before reaching the origin. Cloudflare operates in over 330 cities across 120 countries, directly peering with over 13,000 network providers.

Handling this magnitude of data transfer cannot be solved simply by adding more CPU cores or spinning up Kubernetes pods. At this scale, the standard Linux networking stack becomes the bottleneck. Every packet traversing the kernel network buffer (sk_buff), context switches, and CPU interrupt handlers introduces measurable latency and CPU contention.

To survive at this scale, Cloudflare redesigned their networking architecture from the physical layer up to the application layer.

BGP Anycast Routing Map

Traditional DNS routing uses Unicast, where each IP address belongs to a single physical machine or server cluster in one geographical location. If a user in Tokyo requests a server in Virginia, the packet must physically travel across the Pacific Ocean via submarine optical cables, incurring 150ms+ round-trip latency.

Cloudflare uses BGP (Border Gateway Protocol) Anycast.

In an Anycast network, hundreds of datacenters across London, Tokyo, Frankfurt, Singapore, and São Paulo all advertise the exact same IP addresses (e.g., 1.1.1.1 or 104.16.0.0/12) to neighboring Internet Service Providers (ISPs).

               ┌───────────────────────┐
               │    Client in Tokyo    │
               └───────────┬───────────┘
                           │ (BGP Shortest AS-Path)
               ┌───────────▼───────────┐
               │  Cloudflare Tokyo DC  │ ◄── IP: 104.16.12.34 (Anycast)
               └───────────────────────┘

               ┌───────────────────────┐
               │   Client in London    │
               └───────────┬───────────┘
                           │ (BGP Shortest AS-Path)
               ┌───────────▼───────────┐
               │  Cloudflare London DC │ ◄── IP: 104.16.12.34 (Exact Same IP!)
               └───────────────────────┘

When an attacker launches a 3 Terabit-per-second distributed denial-of-service attack with 500,000 compromised IoT devices across the globe, the attack traffic does not converge on a single server. Instead, the attack is naturally diffused across all 330+ datacenters simultaneously.

Each datacenter only absorbs a localized fraction of the flood (e.g., 10 Gbps in Sydney, 15 Gbps in Frankfurt), which its local scrubbing pipeline can easily digest without overloading international transit pipes.

In a standard Linux operating system, when a packet arrives at the Network Interface Card (NIC):

1. The NIC triggers a hardware interrupt (IRQ).
2. The kernel driver allocates a socket buffer structure (struct sk_buff).
3. The kernel runs iptables/nftables firewall rules.
4. The TCP/IP stack processes checksums, sequence numbers, and state tables.
5. The packet is copied into user-space socket buffers for the application to read.

At 50 million packets per second, the CPU spends 90% of its cycles just allocating and freeing sk_buff memory objects.

Cloudflare uses XDP (eXpress Data Path) powered by eBPF (Extended Berkeley Packet Filter).

XDP executes custom, JIT-compiled C/Rust bytecode directly inside the network driver layer, before the kernel even allocates an sk_buff.

// Conceptual XDP / eBPF DDoS Filter Hook
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("xdp_drop_syn_flood")
int filter_packet(struct xdp_md *ctx) {
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end) return XDP_PASS;

    if (eth->h_proto != __constant_htons(ETH_P_IP)) return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end) return XDP_PASS;

    // Check if source IP is present in eBPF LPM Trie blacklist
    if (bpf_map_lookup_elem(&blocked_ips, &ip->saddr)) {
        // Drop instantly at the wire without kernel memory overhead
        return XDP_DROP;
    }

    return XDP_PASS;
}

With XDP, a single commodity server can drop over 20 million malicious packets per second per CPU core, rejecting multi-gigabit attacks with less than 2% CPU utilization.

Inside each Cloudflare datacenter, there are hundreds of physical servers. How does incoming traffic from a 100Gbps fiber link get distributed across these servers without breaking existing TCP connections?

Cloudflare developed Unimog, an L4 load balancer built entirely on eBPF.

Traditional load balancers use standard 5-tuple hashing (src_ip, src_port, dst_ip, dst_port, protocol). If a server is added or removed for maintenance, the hash ring changes, resetting active TCP connections and breaking file uploads or WebSocket streams.

Unimog eliminates connection drops using a two-tier eBPF routing mechanism:

1. Hop 1: The packet arrives at any server in the cluster via Equal-Cost Multi-Path (ECMP) routing.
2. Hop 2: The server's eBPF program inspects its local connection table. If the packet belongs to an established connection owned by Server B, it encapsulates the packet into a lightweight Geneve UDP tunnel and redirects it directly to Server B at wire speed.
3. Server B decapsulates the packet and delivers it to the user application.

This ensures zero connection resets during rolling software deployments and hardware reboots.

For over a decade, Cloudflare used NGINX to terminate TLS and proxy HTTP requests. However, as web traffic evolved toward HTTP/2 multiplexing, HTTP/3 (QUIC), and massive concurrency, NGINX's multi-process architecture hit structural boundaries:

• Connection Pool Fragmentation: In NGINX's multi-process model, worker processes cannot share upstream TCP/TLS connections. A request in Process A cannot reuse an idle connection opened by Process B, resulting in redundant TLS handshakes to origins.
• CPU Scheduling Inefficiency: When one worker process is pinned processing a heavy compute task (e.g., regex WAF rule), other connections assigned to that process suffer head-of-line blocking.

To solve this, Cloudflare spent years building Pingora — an asynchronous, multi-threaded HTTP proxy written in Rust from scratch.

┌────────────────────────────────────────────────────────┐
│                   PINGORA ARCHITECTURE                 │
├────────────────────────────────────────────────────────┤
│  Thread Pool (Async Tokio Runtime)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │ Worker Th. 1 │  │ Worker Th. 2 │  │ Worker Th. N │  │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  │
│         │                 │                 │          │
│  ┌──────▼─────────────────▼─────────────────▼───────┐  │
│  │    SHARED GLOBAL CONNECTION POOL (Async Lock)    │  │
│  └────────────────────────┬─────────────────────────┘  │
│                           │ Upstream TLS Resumption    │
│  ┌────────────────────────▼─────────────────────────┐  │
│  │             Origin Server (HTTP/2 & HTTP/3)       │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────┘

• 70% reduction in CPU consumption compared to NGINX under identical traffic loads.
• 67% reduction in memory footprint.
• 433x reduction in connection establishment time for upstream requests due to cross-thread connection pooling.
• Zero memory safety vulnerabilities (memory corruption, buffer overflow, use-after-free) guaranteed by Rust's ownership model.

The genius of Cloudflare's infrastructure is not having the most expensive supercomputers, but rather stripping every unnecessary abstraction from the network path — bypassing the kernel for defense, sharing memory safely with Rust, and letting mathematics (Anycast and eBPF) distribute the world's traffic cleanly.