System design fundamentals

A program that works on your laptop for one user is a solved problem. The same program serving a million users, on machines that fail, over networks that drop packets, is a different problem entirely. System design is the discipline of making that jump deliberately. It has no single right answer, only a toolbox of recurring techniques, each of which buys something and costs something. Knowing the costs is what separates design from cargo-culting.

Scaling up versus scaling out

When a service runs out of capacity you have two options. Vertical scaling means a bigger machine: more cores, more memory, faster disks. It needs no code changes and keeps everything simple, which is why it is often the right first move. But hardware has a ceiling, prices rise steeply near it, and a single machine is a single point of failure.

Horizontal scaling means more machines sharing the load. It goes much further and survives individual failures, but it turns your program into a distributed system, with all the partial failures and coordination problems that implies.

Load balancers and stateless services

Horizontal scaling needs something to spread traffic. A load balancer gives clients one address, forwards each request to a server in a pool using a strategy like round-robin or least-connections, and runs health checks so it stops routing to broken instances.

This works best when the servers are stateless: they hold no per-user data in memory between requests. If a login session lives in one server’s RAM, that user must keep hitting that server (“sticky sessions”), and losing the server logs them out. Push the session into a shared store instead and every instance becomes interchangeable. You can autoscale, deploy by replacing instances, and survive crashes. The state has not disappeared; it has moved to a place designed to hold it.

Caching and the invalidation problem

A cache keeps a copy of expensive-to-produce data somewhere fast, usually memory. The common cache-aside pattern looks like this:

read(key):
  value = cache.get(key)
  if miss:
    value = db.query(key)
    cache.set(key, value, ttl=60s)
  return value

Caches can cut latency and database load dramatically. The catch is that a cache is a copy, and copies go stale. Deciding when to invalidate is genuinely hard: a slow read can fetch an old value, a concurrent write can clear the cache, and then the slow read writes the old value back. One database fact may feed dozens of cached pages. A TTL (time to live) bounds how stale things can get, trading freshness for hit rate. Before adding a cache, decide how stale is acceptable for that data, because you will get some staleness.

Replication: copies for reads and resilience

Replication keeps copies of the same data on several nodes. In the common leader-follower setup, one node accepts writes and streams changes to followers, which serve reads and can take over if the leader dies.

Most setups replicate asynchronously for speed, so followers lag slightly behind. That produces odd user-visible bugs: you save a comment, reload, and it is gone, because your read hit a replica that had not caught up. Fixes like read-your-writes consistency (read from the leader briefly after a user writes) exist, but only if you know the problem is there.

Sharding: splitting data that no longer fits

Replication helps read load, but every copy still holds everything and every write still goes through one leader. When data or write volume outgrows a single node, you shard (partition): each node holds a subset, chosen by a partition key. Key-range partitioning keeps related keys together but risks hot spots; hash partitioning spreads load evenly but makes range queries expensive.

Sharding is a heavy commitment. Queries that span shards get slow, transactions across shards get hard, and rebalancing as you grow is real operational work. Real systems combine both techniques: data is sharded, and each shard is replicated.

Queues for decoupling

If service A calls service B synchronously, A is only as fast and available as B. Putting a message queue between them changes that. A enqueues a job and moves on; B consumes at its own pace. Queues absorb traffic spikes, let consumers scale independently, and make retries natural.

The cost is that work becomes eventual. The user gets “your export is being prepared” rather than the file. Consumers must also cope with duplicates, since most queues guarantee at-least-once delivery, so handlers should be idempotent.

The CAP theorem, stated accurately

CAP is routinely misquoted as “consistency, availability, partition tolerance: pick two.” That framing is misleading. In a distributed system, network partitions (nodes unable to talk to each other) are not something you choose to tolerate or not; they happen.

The real statement: when a partition occurs, a system must choose between consistency (every read reflects the most recent write, in the strong linearizable sense) and availability (every non-failed node returns a non-error response).

A CP choice refuses or blocks some requests during the partition; an AP choice keeps answering but may return stale or conflicting data. When the network is healthy, you can have both. There is still a trade-off in normal operation, though: keeping replicas tightly consistent requires coordination, which costs latency. The PACELC formulation names this explicitly. Kleppmann also notes that CAP’s definitions are narrow, so it is a poor tool for classifying real databases; ask instead what guarantees a system gives and what happens when a node is cut off.

Latency numbers worth feeling

Design decisions become much clearer when you know rough orders of magnitude:

• Reading main memory: about 100 nanoseconds
• Random read from an SSD: about 100 microseconds
• Round trip within one datacenter: about half a millisecond
• Round trip between continents: about 100 to 150 milliseconds

Each step is roughly a thousand-fold or more. That is why an in-memory cache beats a database query, why a page making fifty sequential service calls feels sluggish, and why putting data near users matters. The speed of light is not negotiable.

The underlying skill

None of these techniques is free. Every cache adds staleness, every replica adds lag, every shard adds coordination, every queue adds delay. Good system design starts with requirements such as expected load, acceptable staleness and tolerable downtime, then reaches for the simplest mechanism that meets them.