Scrolls
5 min read
On this page

Load Balancers & Reverse Proxies

A load balancer distributes incoming traffic across multiple backend servers. A reverse proxy sits in front of your application and handles concerns like SSL termination, caching, and request routing. In practice, most tools do both -- Nginx, HAProxy, Traefik, and cloud load balancers all combine these roles.

Layer 4 vs Layer 7

The distinction matters because it determines what the load balancer can see and do.

Layer 4 (Transport)

Operates at the TCP/UDP level. It sees IP addresses and ports but not HTTP headers, URLs, or cookies. It forwards raw TCP connections to backend servers.

Client -> L4 Load Balancer -> Backend Server
          (routes by IP:port)

Advantages:

  • Very fast -- no protocol parsing overhead
  • Protocol-agnostic (works with HTTP, gRPC, databases, anything over TCP)
  • Lower latency

Use when: You need raw performance, non-HTTP protocols, or simple round-robin distribution.

Layer 7 (Application)

Operates at the HTTP level. It can inspect headers, URLs, cookies, and request bodies. It can make routing decisions based on content.

Client -> L7 Load Balancer -> /api/* -> API Servers
          (routes by URL,    -> /static/* -> CDN/Static Servers
           headers, etc.)    -> /ws/* -> WebSocket Servers

Advantages:

  • Content-based routing (route by path, header, cookie)
  • SSL termination
  • Request/response modification
  • Health checks at the application level (check /health, not just TCP port)
  • Rate limiting and authentication

Use when: You need routing decisions based on request content, which is most of the time for web applications.

The Tools

Nginx

The workhorse. Reverse proxy, load balancer, static file server, and more. Runs in front of a significant portion of the internet.

# /etc/nginx/conf.d/myapp.conf
upstream backend {
    server 10.0.1.10:8080 weight=3;
    server 10.0.1.11:8080 weight=2;
    server 10.0.1.12:8080;
    
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name myapp.example.com;

    ssl_certificate /etc/nginx/ssl/cert.pem;
    ssl_certificate_key /etc/nginx/ssl/key.pem;

    location /api/ {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        proxy_connect_timeout 5s;
        proxy_read_timeout 30s;
    }

    location /static/ {
        root /var/www;
        expires 30d;
        add_header Cache-Control "public, immutable";
    }
}

HAProxy

Purpose-built for load balancing. Excellent performance, detailed statistics, and fine-grained health checking. The default choice for high-throughput TCP and HTTP load balancing.

# /etc/haproxy/haproxy.cfg
frontend http_front
    bind *:443 ssl crt /etc/haproxy/certs/myapp.pem
    default_backend app_servers
    
    acl is_api path_beg /api
    acl is_static path_beg /static
    
    use_backend api_servers if is_api
    use_backend static_servers if is_static

backend app_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    
    server app1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server app2 10.0.1.11:8080 check inter 5s fall 3 rise 2
    server app3 10.0.1.12:8080 check inter 5s fall 3 rise 2

backend api_servers
    balance leastconn
    option httpchk GET /api/health
    
    server api1 10.0.2.10:8080 check
    server api2 10.0.2.11:8080 check

Caddy

Modern, automatic HTTPS by default. Obtains and renews Let's Encrypt certificates without configuration. Excellent for smaller deployments where simplicity matters.

# Caddyfile
myapp.example.com {
    reverse_proxy /api/* 10.0.1.10:8080 10.0.1.11:8080 {
        lb_policy round_robin
        health_uri /health
        health_interval 10s
    }

    reverse_proxy /ws/* 10.0.1.20:8080 {
        header_up X-Forwarded-For {remote_host}
    }

    file_server /static/* {
        root /var/www
    }
}

Caddy handles TLS certificates automatically. No certbot, no cron jobs, no manual renewal.

Traefik

Designed for dynamic environments. Discovers services automatically from Docker labels, Kubernetes ingress resources, or consul. The standard ingress controller for many Docker and Kubernetes setups.

# Traefik with Docker labels
# docker-compose.yml
services:
  traefik:
    image: traefik:v3.0
    command:
      - "--providers.docker=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/acme/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - acme-data:/acme

  myapp:
    image: myapp:latest
    labels:
      - "traefik.http.routers.myapp.rule=Host(`myapp.example.com`)"
      - "traefik.http.routers.myapp.tls.certresolver=letsencrypt"
      - "traefik.http.services.myapp.loadbalancer.server.port=8080"

Health Checks

A load balancer must know which backends are healthy. Without health checks, it sends traffic to dead servers.

Active Health Checks

The load balancer periodically sends a request to a health endpoint. This catches problems before real users hit them. Passive health checks (monitoring real traffic for errors) supplement active checks but cannot detect problems before users are affected.

# HAProxy active health check
server app1 10.0.1.10:8080 check inter 5s fall 3 rise 2
  • inter 5s: Check every 5 seconds
  • fall 3: Mark unhealthy after 3 consecutive failures
  • rise 2: Mark healthy after 2 consecutive successes

Your health endpoint should verify that the application can actually serve requests -- check database connectivity, check critical dependencies. Do not just return 200 unconditionally.

@app.get("/health")
def health():
    try:
        db.execute("SELECT 1")
        return {"status": "healthy"}, 200
    except Exception:
        return {"status": "unhealthy"}, 503

SSL Termination

Terminate TLS at the load balancer, then forward unencrypted traffic to backends on the internal network. This simplifies certificate management (one place to update certs) and offloads TLS processing from application servers.

Client --[HTTPS]--> Load Balancer --[HTTP]--> Backend Servers
                    (terminates TLS)          (internal network)

Set the X-Forwarded-Proto header so backends know the original request was HTTPS:

proxy_set_header X-Forwarded-Proto $scheme;

Sticky Sessions

Session affinity routes a user's requests to the same backend server. Necessary when application state is stored in memory (session data, WebSocket connections).

# Nginx sticky sessions via cookie
upstream backend {
    sticky cookie srv_id expires=1h;
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
}

# HAProxy sticky sessions
backend app_servers
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server app1 10.0.1.10:8080 cookie s1
    server app2 10.0.1.11:8080 cookie s2

The better solution is to make your application stateless -- store sessions in Redis or a database instead of in memory. Then any backend can handle any request, and sticky sessions become unnecessary.

Rate Limiting

Protect your application from abuse by limiting requests at the proxy level.

# Nginx rate limiting
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

server {
    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://backend;
    }
}

This allows 10 requests per second per IP, with a burst allowance of 20. Requests beyond the burst get a 503 response.

Managed vs Self-Hosted

Managed Load Balancers (ALB, Cloud Load Balancer, Azure LB)

Use when:

  • You run on a single cloud provider
  • You want zero operational overhead for the load balancer itself
  • You need integration with cloud-native features (WAF, auto-scaling groups, certificate manager)
  • You prefer paying money over spending engineering time

Self-Hosted (Nginx, HAProxy, Traefik)

Use when:

  • You run multi-cloud or on-premises
  • You need fine-grained configuration control
  • You want to avoid cloud vendor lock-in
  • You need features the managed service does not offer (custom Lua scripts, specific routing logic)

For most cloud-native applications, start with the managed option. It handles availability, scaling, and certificate management. Switch to self-hosted when you hit its limitations.

Common Pitfalls

  • No health checks. Without them, the load balancer sends traffic to crashed servers. Always configure active health checks.
  • Health checks that always return 200. A health endpoint that does not check dependencies is lying. Verify database and cache connectivity.
  • Not forwarding client IP. Without X-Forwarded-For, your application sees the load balancer's IP for every request. Set the header.
  • Sticky sessions hiding scaling problems. If you need sticky sessions, your application probably has shared state that should be externalized.
  • SSL termination without re-encryption. Internal traffic without TLS is acceptable in a private network. In a shared network or zero-trust environment, use TLS end to end or mTLS.
  • Forgetting connection timeouts. Long-running requests without proper timeouts tie up backend connections. Set proxy_read_timeout and proxy_connect_timeout.
  • Overcomplicating routing. Start simple. Round-robin with health checks handles most workloads. Add complexity only when you have evidence you need it.

Key Takeaways

  • Layer 4 load balancers route by IP and port (fast, protocol-agnostic). Layer 7 load balancers route by HTTP content (flexible, feature-rich).
  • Nginx and HAProxy are the proven choices for self-hosted load balancing. Caddy and Traefik add automatic TLS and dynamic service discovery.
  • Always configure active health checks. The load balancer must know which backends can serve traffic.
  • Terminate TLS at the load balancer for simpler certificate management. Forward X-Forwarded-For and X-Forwarded-Proto to backends.
  • Make your application stateless instead of relying on sticky sessions.
  • Start with a managed load balancer on cloud platforms. Move to self-hosted when you need more control.