What Is System Design? The Parts, the Promises and the Trade-offs

One cook, one stall, four nights#

This is part 1 of 3 in the Introduction to System Design, and it needs no background. Meet Mei. She runs a one-person dumpling stall at a night market. She takes each order, folds the dumplings, steams them and takes the cash, all by herself.

On a quiet Monday it works. A few people arrive, Mei serves them, and she even has time to wipe the counter. Then word gets round and Fridays get busy. The queue grows, Mei rushes, and two orders go to the wrong people. Then the market holds a festival night, and an hour after opening a long line of people are still waiting. Some give up and walk away.

Then one night Mei is ill. Nobody else can run the stall, so it's simply shut. Her regulars turn up to a closed shutter.

Notice what didn't go wrong. The dumplings were fine every night. The recipe never changed. What failed was the way the stall was organised: one person doing every job, with nothing to fall back on. Fixing that, deciding who does what, how orders flow and what happens when someone is missing, is exactly what system design is about.

What system design means#

System design is deciding which parts a software system has, how those parts connect and how data moves between them. The goal is a system that does what it must, does it well enough, for the number of people who use it, within a budget you can afford.

That sentence hides two kinds of promise, and the stall shows both:

• Functional requirements are what the system does. For the stall: sell pork and veg dumplings, and take card payment. If a customer can't pay by card, a feature is missing.
• Non-functional requirements are how well it does it. For the stall: serve every customer within 10 minutes, open every market night, and hand each person the order they asked for. Between them, the four nights broke all three, even though every dumpling was cooked correctly.

Beginners tend to think only about the first kind. Most of system design is about the second. Building an app that sells dumplings is programming. Building one that still sells them to thousands of people at once, quickly, without going down, is design.

The stall as a software system#

Now redesign the stall, and give every part its software name. Each person or object has a direct match:

• Customer → client. The phone or browser that sends a request and waits for an answer.
• Order taker → load balancer. A new helper who takes each order and passes it to whichever cook is free, so no cook is swamped while another stands idle.
• Cook → server. A machine that does the actual work of turning a request into a response.
• Order tickets on a rail → queue. Orders wait in line in the order they arrived, so none get lost when the cooks are busy.
• Tray of dumplings folded before opening → cache. Work done ahead of time and kept close at hand, so the common request is answered fast.
• Mei's notebook of recipes and takings → database. The permanent record that must survive the night.

The story also made three qualities obvious, and these are the words you'll hear in every design discussion:

• Scalability: can the system handle more people by adding resources? The Friday and festival queues were a scaling problem.
• Latency: how long one customer waits from ordering to eating. On the festival night it grew from minutes to hours.
• Availability: is the system open when people need it? With one cook, Mei's illness meant zero. A part whose failure stops everything is called a single point of failure.

The numbers: arrivals against capacity#

In this example, Mei can serve 30 plates an hour on her own. Here are the nights, with illustrative numbers:

One rule explains the whole table: if people arrive faster than you can serve them, the queue grows every hour, and so does the wait. On Monday, 20 arrive and 30 could be served, so there's no queue. On Friday, 45 arrive and 30 are served, so 15 are left over after an hour. At the festival, 120 arrive and 90 are left. In this example each customer orders one plate, so at 30 plates an hour, the person at the back of the line after that first hour waits three more hours. That's latency, caused by a scaling problem.

The redesign gets to 120 with three changes. The tray of pre-folded dumplings means a cook only steams, which in this example doubles one cook's output from 30 to 60 an hour. A second cook doubles it again, to 120. And the order taker keeps both cooks fed. Without the order taker, a second cook barely helps: if every customer still queues at Mei to order and pay, the line moves at Mei's speed, and the extra cook waits for work.

Now look at the last row. When one cook is away, the redesigned stall still serves 60 an hour. Half capacity is a bad night, but it isn't a closed shutter. That's availability, bought with a second cook.

Expert corner: trade-offs, bottlenecks and interviews#

There is no best design, only trade-offs. Every fix in the redesign has a cost:

• The tray (cache) trades freshness and space for speed. Dumplings folded hours ago aren't as fresh, and whatever is left at closing goes in the bin. A software cache can likewise serve out-of-date data and takes up memory.
• The second cook (an extra server) costs wages every night, including quiet Mondays when Mei alone would cope. Spare capacity is what you pay for availability.
• The order taker (load balancer) adds one more person every order must pass through.

That last point is how designs break at scale: each fix moves the bottleneck. The cooks were the slowest step, so you added one. Now the order taker is the slowest step. Say she takes an order every 30 seconds: that's 120 an hour, exactly the festival crowd. If 150 people turn up, the queue forms at her, however many cooks you hire. The order taker is now the single point of failure too. A real system works the same way: you find the slowest part, fix it, then look for the next one.

Averages also hide bursts. Even on Monday, if eight people arrive in the same five minutes, a short queue forms although 20 an hour is well under 30. Designers leave headroom for peaks, not just the average.

Recap and the road ahead#

Mei's stall failed for three reasons that had nothing to do with dumplings: arrivals outran capacity, waits grew with the queue, and one ill cook closed everything. Naming the parts, splitting the work and adding a spare fixed all three, each at a price.

Here's what the rest of this course climbs through, step by step. First, the core trade-offs: performance against scalability, latency against throughput, and availability against consistency. Then how systems behave when parts fail. Then the building blocks themselves: DNS, CDNs, load balancers, databases and caches. Then queues and messaging, and finally expert patterns and how to monitor a system once it's running.

Practice

Check your understanding

FAQ

Frequently asked questions