All lessons are free · No login · No trackingStart learning
uohk.comNetwork Buddies 中

Network Stories 20 · Intermediate

No Fear of Crowds: Load Balancers and High Availability

When millions of people open the same website at the same time, why doesn't the server collapse?

Starring: Rue, Dr. Lookup, PipAbout 10 min read

Load BalancerHigh AvailabilityDNS

Recommended first: Story 19: “Firewall and ACL”, Lesson 7

1. The Story

Press "Next" to go through it step by step. At each step, a character appears and explains what they are doing.

Prologue: a quick review

Step 1 of 12

One name, one server

Let me look that up... The IP Address of example.com is 192.0.2.80. So far, there is only one server behind this address.

Animation text version (all steps)
  1. Prologue: a quick review
  2. One name, one server · Dr. Lookup: Let me look that up... The IP Address of example.com is 192.0.2.80. So far, there is only one server behind this address.
  3. Act 1: The big sale
  4. The whole world rushes in · Pip: Big sale at example.com! Thousands upon thousands of Pips rush to the same server at once. It can only handle a limited number of requests per second, so everyone is queuing and the web page is as slow as a snail.
  5. The server goes down · Pip: Worse still, the server finally cannot cope and stops working. With only one server, when it stops, the whole website stops. This is a Single Point of Failure.
  6. Act 2: Enter the host
  7. Rue's distant relative · Rue: Let me introduce my distant relative: the Load Balancer! It stands in front of a group of servers and uses the website's address 192.0.2.80 to face the outside world. Behind it are three servers using Private IP addresses 10.0.0.x.
    • The outward-facing address is called a VIP (Virtual IP Address): 192.0.2.80
    • Dr. Lookup's answer does not need to change, and users have no idea how many servers are behind it
  8. Taking turns: Round Robin · Pip: We arrive at the Load Balancer one after another. It hands us to Server 1, 2, 3, 1, 2, 3... in turn. This is called Round Robin. Each server only has to handle a third of the work!
    • Other methods: Least Connections: give it to the least busy server right now
    • Hash: the same user usually goes to the same server, so it can remember the shopping cart
  9. Health Check: are you OK? · Rue: Every few seconds, the Load Balancer sends each server a Health Check: "Are you OK?" Servers 1 and 3 both reply "I'm fine!", but Server 2 does not answer...
  10. No more work for Server 2 · Pip: The Load Balancer marks Server 2 as "unhealthy" and stops handing us to it for now. Servers 1 and 3 keep working. Most users just notice the website being a tiny bit slower; they never realise a server has broken!
    • Once Server 2 is fixed and the Health Check succeeds, it is added back automatically
  11. Act 3: The host has a backup too
  12. Can the Load Balancer go down too? · Rue: Wait! If the Load Balancer itself breaks, none of the servers can help, so it becomes a new single point of failure. That's why we add a backup Load Balancer. The two work as one main and one backup, and this is called High Availability. (Server 2 has been fixed, passed its Health Check and been added back automatically.)
    • This setup is called Active-Standby
  13. Heartbeat: checking on each other · Pip: The two Load Balancers keep sending each other a Heartbeat: "I'm still here!" "Me too!" Normally the main machine is in charge of the VIP 192.0.2.80, while the backup just watches from the side.
  14. Failover: the backup takes over · Rue: The main machine suddenly stops working, and the backup hears no heartbeat! Within a few seconds it takes over the VIP 192.0.2.80 and carries on sharing out the work. This is called Failover. Users do not have to change any settings at all.
  15. Dr. Lookup can share out traffic too · Dr. Lookup: I can help too: I can answer one name with several IP Addresses, or answer with the nearest data center based on where the user is. But my answers are kept in a Cache for 3600 seconds, so if something goes wrong they may not change straight away. That's why I'm used together with a Load Balancer.
  16. Summary: no fear of crowds · Rue: The Load Balancer shares the work among several servers; the Health Check automatically avoids broken servers; and two Load Balancers back each other up with Heartbeat and Failover. With a backup for every important part, the website can keep running. This way, every server is taken care of!
    • Load Balancer: outward-facing VIP, shares out requests (Round Robin, Least Connections, Hash)
    • Health Check: automatically removes unhealthy servers
    • High Availability: Active-Standby + Heartbeat + Failover

2. Where Is the Problem

Dr. Lookup, that is DNS (Domain Name System), tells everyone that the IP Address of example.com is 192.0.2.80. (Note: an address starting with 192 is not necessarily Private. Only addresses starting with 192.168 are. 192.0.2.80 is a public address on the Internet.)

Normally, there is just one server behind this address, and everything works fine. Then comes the day of the big sale, and with it, problems:

  1. Too many people. One server can only handle a limited number of requests per second. When thousands upon thousands of Pips rush in at once, everyone has to queue, and the web page becomes as slow as a snail.
  2. When it stops, everything stops. The server finally cannot cope and stops working. With only one server, when it stops, the whole website stops.

The second problem is called a Single Point of Failure: if just one part goes wrong, the whole system grinds to a halt.

3. How the Network Solves It

Think of a very popular restaurant. At the door there is a host who takes guests to different tables in turn. If a waiter falls ill, the host stops sending guests to him. And the host has a backup colleague too.

A Load Balancer is that host: it stands in front of a group of servers and hands each request to one of them. Add a backup Load Balancer, so that if either one breaks the other takes over, and you have High Availability: a design that keeps a system running even when some of its equipment fails.

4. A Closer Look

VIP (Virtual IP Address): just one address to the outside world

Device Address Who sees it
Load Balancer VIP 192.0.2.80 All users
Server 1 10.0.0.11 Only the Load Balancer
Server 2 10.0.0.12 Only the Load Balancer
Server 3 10.0.0.13 Only the Load Balancer

The VIP is the address the Load Balancer uses to face the outside world, and it does not permanently belong to any one machine. Dr. Lookup's answer does not need to change; it is still 192.0.2.80. The servers behind it use Private IP addresses 10.0.0.x, and users have no idea how many servers are back there.

When Pip arrives at the VIP, the Load Balancer changes the receiver to the address of one of the servers, a bit like NAT (Network Address Translation). The server's reply also goes back to the user through the Load Balancer.

How are requests shared out? Three common methods

Method How it works Good for
Round Robin Takes turns: 1, 2, 3, 1, 2, 3... Requests that each take about the same amount of work
Least Connections Gives the request to the server with the fewest connections right now, the least busy one When some requests take especially long to handle
Hash Calculates from data such as the user's IP Address, so the same user usually goes to the same server When servers need to remember things like a shopping cart

The story uses Round Robin: the three servers take turns, so each only has to handle about a third of the work.

Health Check: are you OK?

Every few seconds, the Load Balancer sends a Health Check to each server. It may simply try connecting to a Port, or it may actually ask for a web page and see whether the reply is correct.

Server 1  10.0.0.11  Are you OK? → I'm fine!    Healthy
Server 2  10.0.0.12  Are you OK? → (no reply)   Unhealthy, stop sending work
Server 3  10.0.0.13  Are you OK? → I'm fine!    Healthy

After several failures in a row, a server is marked "unhealthy" and taken off the list for a while. Once it is fixed and the Health Check succeeds again, it is added back automatically. Most users just notice the website being a little slower; they never realise a server has broken.

High Availability: the host has a backup too

With only one Load Balancer, it becomes a new Single Point of Failure. So we add a backup machine. The two work as one main and one backup, which is called Active-Standby:

  1. Heartbeat: the two Load Balancers keep sending each other "I'm still here!" messages.
  2. Normally the main machine (Active) is in charge of the VIP 192.0.2.80, while the backup (Standby) just watches from the side.
  3. Failover: if the backup hears no heartbeat for a while, it takes over the VIP within a few seconds and carries on sharing out the work.

When it takes over, the backup sends an ARP (Address Resolution Protocol) broadcast in the LAN (Local Area Network) to tell the Switch and the Router: "I'm now in charge of 192.0.2.80." Users do not have to change any settings at all.

Dr. Lookup can share out traffic too

DNS can also help spread the load: it can answer one name with several IP Addresses, or answer with the address of the nearest Data Center based on where the user is.

However, DNS answers are kept in a Cache, for example with a TTL (Time to Live) of 3600 seconds. If something goes wrong at one data center, users who have remembered the old answer may not be moved away straight away. That is why big websites use both together: DNS first sends users to different regions, and in each region a Load Balancer shares them out among the servers.

5. Try It Yourself

Task: look up the phone book a few times

  1. On a computer, open "Command Prompt" (Windows) or "Terminal" (Mac).
  2. Type nslookup followed by the name of a big website, for example a video site you often use.
  3. See whether the answer contains several IP Addresses. Wait a few minutes and look it up again. Has the order or the address changed?
  4. Think about it: behind each of these addresses, there may be a Load Balancer and a big crowd of servers.

Mini game

Whose job is it?

Is each of the jobs below handled by Load Balancer distribution, Health Check, Failover, or DNS load sharing?

  1. 1Hand each request to one of the servers behind

  2. 2Regularly check whether the servers reply correctly

  3. 3After the main Load Balancer stops working, the backup takes over the VIP

  4. 4Reply with the IP Address of the nearest data center based on where the user is

  5. 5Share out work using Round Robin or Least Connections

  6. 6A regular check finds that Server 2 does not reply, so it is marked unhealthy

6. Where the Analogy Breaks

  • A Load Balancer is not just one person showing guests to tables. It can handle tens of thousands of connections per second, and it is often software or a cloud service rather than a separate machine.
  • Not every Pip is handled separately. Packets in the same TCP (Transmission Control Protocol) connection keep going to the same server; otherwise the connection would break. In the animation, one Pip stands for one request.
  • A Health Check cannot catch everything. A server that "replied" is not necessarily fine. For example, it may reply very slowly or send wrong data, so checks need to be designed well.
  • Two Load Balancers can also work together. Besides one main and one backup, there is also Active-Active, where both share the work at the same time.
  • There is more than one layer of backup. Real large websites have backups for network cables, power, Switches and even whole data centers. The animation only shows the servers and the Load Balancers.

7. Quiz

Question 1 of 3Which address does a Load Balancer use to face the outside world?
Question 2 of 3A Health Check finds that Server 2 is not replying. What does the Load Balancer do?
Question 3 of 3Why does the Load Balancer itself need a backup?

8. Key Notes

  • A Load Balancer uses one VIP (Virtual IP Address) to face the outside world and shares requests out among several servers behind it.
  • Ways to share the work include Round Robin, Least Connections and Hash.
  • A Health Check automatically avoids servers that do not reply, and adds them back once they are fixed.
  • High Availability: two Load Balancers run as Active-Standby and watch each other with a Heartbeat. If the main one breaks, Failover happens and the backup takes over the VIP.
  • DNS can share out traffic by region, but because of Caching it should be used together with a Load Balancer.

Further reading

Congratulations! You have finished all 20 stories in Parts 1 to 4. Part 5, on one-to-many delivery, is waiting for you next. From Octo sending his first Frame on his first day at work, to Pango guarding the door and Rue and the host keeping the big sale running, you have followed a packet's whole journey from the LAN to the Internet. To revisit a character's story, press "Back to the story list" below. If you forget a term, look it up in the "Glossary" at the top of the website. Backpack on, let's go!