All articles
Switch to dark mode
Translate

Day 3/60: Horizontal Scaling – Adding Web Servers and Understanding Load Balancers

Frengky Soritua ManurungFrengky Soritua Manurung
··11 min read
Day 3/60: Horizontal Scaling – Adding Web Servers and Understanding Load Balancers

Share this article


Introduction

In Day 2, we solved one important problem: we separated the Database from the Web Server, removing resource contention and reducing the security exposure of our data layer.


But if you look closely, there's still a glaring issue sitting right in front of us: we only have one Web Server.


From an SRE (Site Reliability Engineering) standpoint, this is still a single point of failure. It doesn't matter how well we've isolated the database — if that one Web Server crashes, runs out of memory, or simply can't keep up with incoming traffic, users get downtime. Full stop.


Today, we'll fix that by introducing horizontal scaling and the component that makes it possible: the Load Balancer.


1. The Problem: One Web Server Is Never Enough

Let's think about this the way an SRE would, in terms of capacity and failure domains.

a. Capacity ceiling

Every server has a hard limit on how many concurrent requests it can process, dictated by its CPU, memory, and network bandwidth. Once traffic exceeds that ceiling, requests start queuing, response times climb, and eventually the server starts dropping connections entirely.

b. It's still a single point of failure

Even with a perfectly isolated database, one Web Server means one failure domain. A single bad deployment, a memory leak, a spike in traffic, or even routine OS patching can take your entire application offline. There is no fallback — when that instance is unavailable, there is simply nothing left to serve traffic.


This is exactly the kind of risk SREs are trained to eliminate: no single component should be able to take down the whole system.


2. Vertical Scaling vs. Horizontal Scaling

Before jumping to the solution, it's worth understanding the two fundamental ways to handle increased load.

Vertical Scaling ("Scale Up")

This means upgrading the existing server — more CPU cores, more RAM, faster disks. It's simple to implement (often just a few clicks in your cloud console) but it comes with real limitations:

  • Hardware ceiling — there's always a limit, you can't scale up infinitely.
  • Downtime to resize — upgrading the instance usually requires downtime.
  • No redundancy — it does nothing for redundancy. You still have exactly one server, just a bigger one. If it fails, you're still down.

Horizontal Scaling ("Scale Out")

This means adding more servers running the same application, working together to share the load. Instead of one large server, you have several smaller ones operating in parallel.

Vertical ScalingHorizontal Scaling
How it worksMake one server biggerAdd more servers
Downtime to scaleUsually requiredNot required (add servers on the fly)
RedundancyNoneYes — one server can fail without full outage
Upper limitLimited by hardwarePractically unlimited
ComplexityLowHigher — needs traffic distribution

Horizontal scaling solves both of our problems at once: it increases total capacity, and it removes the single point of failure. But it introduces a new question: if we have multiple Web Servers, how does incoming traffic know which one to go to?

That's where the Load Balancer comes in.


3. The Load Balancer: A Single Entry Point for Multiple Servers

A Load Balancer sits in front of your Web Servers and becomes the single point of contact for all incoming traffic. Instead of users connecting directly to a specific server, they connect to the Load Balancer, which then forwards each request to one of the healthy Web Servers behind it.

From the SRE's perspective, the Load Balancer does three critical jobs:

  1. Distributes traffic across all available servers, so no single one becomes a bottleneck.
  2. Monitors server health, so it never sends traffic to a server that is down or misbehaving.
  3. Provides a stable, single address for users — even though the number and identity of backend servers can change at any time.

Here's what the resulting architecture looks like:

Traffic from the internet always hits the Load Balancer first. The Load Balancer then routes requests only to Web Servers that are passing their health checks — Web Server 3 has failed its check and has been automatically removed from the pool. All Web Servers share the same Database Server in the private VPC, just like we set up on Day 2.


4. Load Balancing Algorithms

The Load Balancer needs a strategy to decide which server gets the next request. The two most common (and easiest to reason about) are:

Round Robin

Requests are distributed to servers in a fixed, rotating order — Server 1, then Server 2, then Server 3, then back to Server 1, and so on. It's simple and works well when all servers have similar capacity and requests take roughly the same amount of time to process.

Least Connections

The Load Balancer sends the next request to whichever server currently has the fewest active connections. This is more adaptive than Round Robin, especially useful when request processing times vary — for example, some requests are quick lookups while others involve heavier computation.


SRE note: there's no single "best" algorithm. The right choice depends on your traffic pattern and how uniform your servers and requests are. Many production load balancers let you switch algorithms without downtime, so it's worth testing more than one under real traffic.


5. Health Checks: Automatically Removing Broken Servers

This is the piece that actually delivers on the promise of "no single point of failure": health checks.

A Load Balancer periodically sends a lightweight request — commonly an HTTP GET to a /health or /status endpoint — to each Web Server in its pool. Based on the response, it decides whether that server is fit to receive traffic:

  • Healthy response (e.g., HTTP 200 within an expected time) → the server stays in the pool and keeps receiving traffic.
  • No response, timeout, or error status → the server is automatically pulled out of the pool. The Load Balancer stops routing traffic to it until it starts responding correctly again.

This is what allows the system to survive a server crash without any human intervention. In the diagram above, Web Server 3 has failed its health check — the Load Balancer simply stops sending it traffic and redistributes that load across Web Server 1 and Web Server 2, without the user ever noticing.

From an SRE standpoint, this is the foundation of self-healing infrastructure: the system detects failure and reacts to it automatically, faster than any human could.


6. What This Buys You (and What It Doesn't)

To be clear-eyed about it — as any SRE should be — horizontal scaling with a Load Balancer isn't a silver bullet. It's important to understand both sides:

What it solves:

  • Removes the Web Server single point of failure — one server can go down without taking the whole system with it.
  • Handles more traffic — scale by adding servers instead of over-provisioning one giant machine.
  • Enables zero-downtime deployments — you can update servers one at a time while the others keep serving traffic.

What it doesn't solve (yet):

  • The Database is still a single instance — we fixed the Web Server's redundancy today, but the Database from Day 2 is still one server: still a single point of failure, just a different one. We'll address this soon with database replication.
  • The Load Balancer itself can become a single point of failure — if you only run one. In production, Load Balancers are typically deployed in a highly available pair or managed as a fully redundant service by your cloud provider.
  • Operational complexity goes up — more servers means more moving parts to monitor, deploy, and keep in sync.

Conclusion

Today we took our architecture from "one Web Server that can fail at any moment" to "a pool of Web Servers, fronted by a Load Balancer that automatically routes around failures." This is a foundational pattern in almost every production system you'll encounter — and it's the same principle whether you're running 2 servers or 200.

Quick summary:

  • Vertical scaling makes one server bigger; it's simple but has hard limits and no redundancy.
  • Horizontal scaling adds more servers; it removes the single point of failure and scales further.
  • A Load Balancer is what makes horizontal scaling actually work — distributing traffic and hiding unhealthy servers from users.
  • Health checks are what make the system self-healing, automatically routing around failures without human intervention.

In the next article of this 60 Days of System Design series, we'll tackle the single point of failure we deliberately left unsolved today: the Database — starting with Database Replication (Primary–Replica setups).


This article is part of the "60 Days of System Design" series.


Other Articles