How Load Balancers Work: L4 vs L7, Algorithms and Health Checks

A load balancer sits in front of a group of servers and spreads incoming traffic across them. It's the component that makes horizontal scaling and high availability possible: add servers to handle more load, remove unhealthy ones without users noticing. Almost every production web system has at least one.

#Key takeaways

• Load balancers distribute traffic, detect unhealthy servers and enable zero-downtime deploys.
• Layer 4 balancers route by IP and port (fast, protocol-agnostic); Layer 7 balancers understand HTTP (routing by path, headers and cookies, plus TLS termination).
• Common algorithms: round robin, least connections, weighted, consistent hashing and power of two choices.
• Health checks and connection draining are what make failures and deploys invisible to users.
• Keep backends stateless to avoid relying on sticky sessions.

#What a load balancer does

text                         ┌──▶ App server 1 ✅
Clients ─▶ Load balancer ├──▶ App server 2 ✅
                         └──▶ App server 3 ❌ (failed health check: no traffic)

Responsibilities:

• Distribution: spread requests to use capacity efficiently.
• Availability: stop routing to failed instances.
• Elasticity: register new instances as autoscaling adds them (horizontal vs vertical scaling).
• Security and offloading: TLS termination, request filtering and sometimes rate limiting.

#Layer 4 vs Layer 7

Example Layer 7 routing:

textapi.example.com/v1/*      → API servers
example.com/static/*      → object storage / CDN origin
example.com/*             → web app servers

Many architectures use both: an L4 balancer at the edge for raw throughput, and L7 proxies (or an API gateway) behind it.

#Balancing algorithms

• Round robin: each server in turn. Simple; works when requests and servers are uniform.
• Weighted round robin: more traffic to bigger servers, or gradual shifting during canary deployments.
• Least connections / least outstanding requests: send to the least busy server. Better when request durations vary.
• IP hash / consistent hashing: the same client or key goes to the same server. Useful for cache locality; consistent hashing minimizes remapping when servers change.
• Power of two choices: pick two servers at random and choose the less loaded one. Surprisingly effective and cheap at scale.
• Latency-aware / EWMA: route based on observed response times.

For typical stateless web apps, least outstanding requests or power of two choices tend to handle uneven workloads better than plain round robin.

#Health checks

The load balancer continuously probes backends:

• Active checks: periodically request GET /healthz; remove a server after N failures and restore it after M successes.
• Passive checks: watch real traffic and eject servers returning errors or timing out (outlier detection).

Design health endpoints thoughtfully:

• Liveness: "the process is running". Keep it cheap.
• Readiness: "I can serve traffic" (dependencies initialized, warmed up). Use this for load-balancer routing.
• Avoid health checks that fail when a shared dependency is down. If every instance reports unhealthy because the database is slow, the load balancer removes all of them and turns a partial problem into a total outage.

#TLS termination

L7 load balancers commonly terminate TLS: they decrypt HTTPS and forward plain HTTP (or re-encrypted traffic) to backends.

• ✅ Centralized certificate management and less CPU work on app servers.
• ✅ Enables L7 routing, since the balancer can see the HTTP request.
• ⚠️ Traffic between the LB and backends should still be protected (private network, or re-encryption for sensitive environments and zero-trust setups).

Pass the original client information to backends with headers like X-Forwarded-For and X-Forwarded-Proto, and configure apps to trust them only from the load balancer.

#Sticky sessions (and why to avoid them)

Session affinity routes a user to the same server every time, usually via a cookie. It's needed when servers keep per-user state in memory. Downsides:

• Uneven load (some servers get heavy users).
• Losing a server loses its users' sessions.
• Harder scaling and deploys.

Better: make servers stateless and store sessions in a shared store (Redis, database) or use signed tokens (authentication vs authorization).

#Connection draining and deployments

When removing a server (for a deploy, scale-in or maintenance), connection draining stops new requests to it while letting in-flight requests finish. Combined with readiness checks, this enables rolling, blue-green and canary deployments with no dropped requests (zero-downtime deployments).

Also handle graceful shutdown in the app: on SIGTERM, stop accepting new work, finish current requests, then exit.

#Global load balancing

To serve users worldwide, or survive a regional outage:

• DNS-based routing (geo, latency or weighted policies) directs users to the nearest healthy region.
• Anycast: the same IP is announced from many locations, and the network routes users to the closest one. CDNs and global edge networks use this (what is a CDN).

Multi-region active-active setups also require a strategy for data replication and consistency (CAP theorem explained).

#Implementations

• Software proxies: NGINX, HAProxy, Envoy, Traefik. Flexible and widely used, including as Kubernetes ingress controllers (Kubernetes explained).
• Cloud load balancers: managed L4 and L7 services from every major cloud provider. Highly available with no servers to manage.
• Service meshes: sidecar or node proxies doing client-side load balancing between microservices, with retries, mTLS and observability.

A minimal NGINX example:

nginxupstream app_servers {
    least_conn;
    server 10.0.1.10:8000 max_fails=3 fail_timeout=10s;
    server 10.0.1.11:8000 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8000 max_fails=3 fail_timeout=10s;
}

server {
    listen 443 ssl http2;
    server_name example.com;
    # ssl_certificate ... ssl_certificate_key ...

    location / {
        proxy_pass http://app_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_connect_timeout 2s;
        proxy_read_timeout 30s;
    }
}

#Pitfalls

• The load balancer as a single point of failure: run it redundantly (managed services do this for you).
• Retry storms: aggressive retries at the LB plus the client plus the service multiply load during incidents. Use budgets and backoff.
• Timeouts misaligned between the LB, app and downstream services, which causes confusing 502/504 errors.
• Uneven load from long-lived connections (WebSockets, gRPC streams) under connection-based balancing. Consider request-level balancing or periodic rebalancing.

#FAQ

#Is a reverse proxy the same as a load balancer?

A reverse proxy forwards client requests to backend servers; a load balancer distributes requests across multiple backends. Many tools (like NGINX or HAProxy) do both, so the terms often overlap.

#Do I need a load balancer with only one server?

Not strictly for distribution, but a load balancer or reverse proxy still helps with TLS termination, zero-downtime deploys (switching between old and new instances), and it makes adding a second server trivial later.

#What is the difference between a load balancer and an API gateway?

An API gateway is a specialized L7 proxy for APIs, adding authentication, rate limiting, request transformation, API keys and analytics. It often sits behind or alongside a load balancer.

#What causes 502 and 504 errors at the load balancer?

A 502 usually means the backend returned an invalid response or closed the connection; a 504 means the backend didn't respond within the timeout. Check backend health, application errors, and timeout alignment across layers.