Load Balancers: Algorithms & High Availability

why one server isn't enough, and DNS round-robin isn't enough either

Once PlateRoute is running on more than one application server, something has to decide which server handles each incoming request. DNS round-robin can spread traffic at a coarse level, but it has no idea which servers are currently healthy or overloaded — it just cycles through a list. A load balancer sits directly in the request path and makes that decision per-request, with real visibility into server health.

what a load balancer actually provides

Capability	What it means in practice
Scalability	Adding a new backend server to the pool immediately gets it traffic — capacity grows without clients changing anything.
Availability	Requests are automatically routed away from servers that fail health checks, without any human intervention.
Performance	Requests can be sent to the least-loaded server, reducing tail latency versus a client hitting an already-overwhelmed server.

where load balancers sit

A large system typically has load balancers at multiple tiers, not just one: between end users and the web/API tier, between that tier and internal application servers, and sometimes between application servers and database read replicas. For PlateRoute, a request from a customer's phone hits a load balancer before reaching an Order Service instance, and Order Service's own calls to Restaurant Service go through another (usually internal) load balancer.

Load balancers are typically deployed in redundant pairs — a load balancer that's itself a single point of failure defeats much of the purpose.

routing algorithms

Algorithm	How it decides	Good fit
Round-robin	Cycles through servers in order.	Servers with roughly equal capacity and uniform request cost.
Weighted round-robin	Like round-robin, but stronger servers get proportionally more requests.	A mixed pool of server sizes.
Least connections	Routes to the server currently handling the fewest active connections.	Requests with widely varying processing time.
Least response time	Routes to the server with the best recent response time.	Latency-sensitive traffic.
IP hash / URL hash	Routes based on a hash of the client IP or requested URL, so the same client/resource consistently lands on the same server.	Stateful backends that benefit from a client sticking to one server (session affinity), or maximizing cache hit rate per server.

stateful vs. stateless load balancing

A stateless load balancer makes each routing decision independently, typically via consistent hashing (see Consistent Hashing) — simple, fast, and resilient, but it can't guarantee a specific client always reaches the exact same backend if the backend pool changes.

A stateful load balancer tracks which backend a given client's session is pinned to (session affinity / "sticky sessions") — necessary if a backend holds in-memory session state that a different backend wouldn't have. The more robust long-term fix, discussed elsewhere in this track, is usually to move session state out of the application server entirely and into a shared store like a distributed cache — that way any backend can serve any request, and the load balancer doesn't need to track sessions at all.

other services a load balancer often absorbs

Service	Why it lives here
Health checking	A heartbeat/liveness check to each backend lets the load balancer stop routing to a failed instance within seconds.
TLS termination	Decrypting HTTPS once at the load balancer, rather than on every backend, offloads CPU work from the application servers.
Basic DDoS mitigation	A load balancer is a natural chokepoint for rate-limiting or blocking abusive traffic before it reaches application servers.

reference

NGINX — What Is Load Balancing?