System Design Fundamentals: Scaling, Sharding, and the Trade-offs Every Architect Makes

The Server That Could Not Grow

A startup's API ran on a single server. It worked beautifully — 50ms responses, 99.9% uptime, happy customers. Then the product went viral. Traffic tripled in a week. The team's first instinct: buy a bigger server. That worked once. The second spike needed a server twice as large, then four times, then eight — until no cloud provider sold a machine big enough, and the single point of failure took the entire product down for six hours during the biggest traffic day in company history[azure-scale-out].

The team had hit the wall that every scaling story hits: vertical scaling has a ceiling, horizontal scaling has no ceiling — but it demands a different architecture. You cannot buy your way out of a design problem. A system built as one monolithic process on one machine cannot simply be "made distributed" — the assumptions baked into its code (shared memory, local transactions, single-process state) all break at once[kleppmann-designing].

System design is the discipline of making those decisions before the traffic spike forces them: how requests are distributed, how data is partitioned, how failure is contained, and which consistency trade-offs are acceptable. This guide covers the fundamentals — the patterns and the trade-offs behind every scalable architecture, from a two-server setup to a globally distributed system.

Vertical vs Horizontal Scaling

Vertical scaling (scaling up) means moving to a bigger machine: more CPU, more RAM, faster disks. It is simple — no code changes, no architecture changes — and it works until it doesn't. The ceiling arrives in two forms: hardware limits (the largest machine money can buy) and cost curves (the next size up often costs 2-3× more for only 1.5× the capacity)[azure-scale-out].

Horizontal scaling (scaling out) means adding more machines. There is no theoretical ceiling — need 10× capacity, add 10 servers. But horizontal scaling requires the application to be stateless: any server must be able to handle any request. If user sessions live in local memory, a request routed to a different server loses the session. This is why the first step of every scaling journey is moving state out of the application — into Redis, a database, or signed tokens[azure-scale-out].

The pragmatic path: scale vertically until the cost curve bends, then design for horizontal. Teams that start horizontal from day one often pay complexity costs that a monolith would not have — distributed systems are a tax you should only pay when you actually need to scale.

Load Balancing: Distributing the Work

Once there are multiple servers, something must decide which server handles which request. That is the load balancer[wikipedia-load-balancing].

Layer 4 load balancers route by IP and port — fast, no understanding of the request content. Layer 7 load balancers understand HTTP: they can route by URL path (/api/ to one pool, /static/ to a CDN), add or remove headers, terminate TLS, and make smarter decisions[wikipedia-load-balancing].

Distribution algorithms:

• Round robin: each request goes to the next server in sequence. Simple, works well when servers are identical and requests are uniform.
• Least connections: each request goes to the server with the fewest active connections. Better when request durations vary — a slow request doesn't pile up on one server.
• Consistent hashing: requests are routed by a hash of a key (usually the client IP or session ID), so the same client always reaches the same server. Essential when servers hold local caches — it maximizes cache hit rates[wikipedia-load-balancing].

The health check is what makes load balancing resilient: the balancer continuously probes each server (usually a lightweight HTTP endpoint) and stops routing to servers that fail. Combined with auto-scaling (adding servers when load rises), this is the mechanism behind "the cloud just handled it."

Sharding: Partitioning the Data

When a single database cannot handle the load — writes too frequent, data too large, working set too big for RAM — the data must be sharded: split across multiple database instances, each owning a subset of the data[azure-sharding].

Shard key selection is the most consequential decision. Shard by user_id and all of one user's data lives together (fast user queries, even distribution). Shard by created_at and you get "hot" shards for recent data and cold shards for old data — plus uneven write load concentrated on the newest shard[azure-sharding].

The three sharding strategies:

• Range-based: rows are partitioned by key ranges (users 1-1M on shard A, 1M-2M on shard B). Simple, supports range queries, but risks hot spots and requires rebalancing as ranges grow.
• Hash-based: the shard is hash(key) % num_shards. Even distribution, but range queries must hit every shard, and resharding (changing the number of shards) remaps almost every key.
• Directory-based: a lookup service maps each key to its shard. Maximally flexible (you can move individual keys), but the lookup service is now a critical dependency and a performance bottleneck.

Resharding is the operation teams underestimate. Moving from 4 shards to 8 means relocating roughly half the data while the system keeps serving traffic. Consistent hashing minimizes the remapping (only 1/N of keys move), but the migration still needs to run online, which is why getting the initial shard count and key choice right matters so much[azure-sharding].

A worked sharding decision. Consider a messaging app with 100 million users and 20 billion messages. Shard by conversation_id and every message in a conversation lives together — queries for a chat history touch exactly one shard. But shard by message_id (hash) and writes spread perfectly evenly across all shards, at the cost of fanning out every conversation query. The winning choice depends on the dominant query: a chat app reads conversations far more often than it scans global message logs, so conversation_id wins despite slightly uneven distribution (some conversations are huge, most are small). The arithmetic confirms it: 20 billion messages over 32 shards is ~625 million rows per shard — large but manageable — whereas a range-based scheme on message IDs would concentrate 100% of write traffic on the single "current" shard, which is precisely the hot-spot failure mode sharding exists to prevent[azure-sharding].

Sharding also interacts directly with database indexing: each shard's indexes only cover that shard's data, and a query without the shard key must fan out to every shard. The SQL Index Selectivity Calculator helps evaluate whether queries within each shard will use indexes efficiently — a query that scans every shard because it lacks the shard key pays the latency of the slowest shard.

Replication and the CAP Theorem

Replication keeps copies of data on multiple machines for two reasons: durability (if one machine dies, the data survives) and read scalability (reads can be served from any replica)[kleppmann-designing].

The moment data has copies, a fundamental trade-off appears, formalized as the CAP theorem: during a network partition (P), a distributed system must choose between consistency (C — every read sees the latest write) and availability (A — every request receives a response)[wikipedia-cap].

• CP systems (PostgreSQL with synchronous replication, ZooKeeper, etcd): refuse requests rather than serve stale data. Correct but can be unavailable during partitions.
• AP systems (Cassandra, DynamoDB in its default mode): keep serving, possibly with stale data. Available but eventually consistent — a read might not reflect a recent write.
• CA is impossible in a distributed system with partitions — networks fail, so the choice is always between C and A when the failure happens[wikipedia-cap].

Eventual consistency is the pragmatic middle ground: writes propagate to replicas asynchronously, and the system converges once the network heals. Most applications tolerate this — a user seeing a two-second-old like count is fine; a bank seeing a two-second-old balance may not be. The design question is always: which data can be stale, and for how long?[kleppmann-designing]

Microservices: The Architecture Trade-off

A monolith deploys as a single application. Microservices split the system into independent services that own their data and communicate over the network[microservices-patterns].

The promised benefits: independent deployment (ship the billing service without touching the search service), independent scaling (scale the image processor without scaling everything), team autonomy, and technology diversity. The real costs: network latency replaces function calls, distributed transactions replace local ones, partial failure becomes the default state, and debugging a request that crosses five services requires distributed tracing[microservices-patterns].

The industry's hard-won lesson, captured in Martin Fowler's "MonolithFirst" observation: start with a well-structured monolith, and extract microservices when specific scaling or team-autonomy needs appear — not before[microservices-patterns]. The Deployment Stamps pattern offers a middle path for SaaS products: deploy complete, independent copies of the whole system (stamp by stamp), getting fault isolation and horizontal scale without microservice decomposition[azure-stamps].

Capacity Math: Sizing the System

System design ultimately reduces to arithmetic: how much traffic, how much data, how much compute. The back-of-envelope calculations happen before any architecture diagram:

Request capacity. If each server handles 1,000 requests per second (RPS) and peak traffic is 50,000 RPS, you need at least 50 servers — plus headroom (typically 2×) for failover and deployment, so 100. At 30% average peak utilization, that's ~170 servers behind a load balancer[wikipedia-load-balancing].

Storage growth. 10 million users × 50 KB of profile data = 500 GB — trivial. But 10 million users × 10 events per day × 1 KB per event = 100 GB of event data per day, 36 TB per year. That arithmetic is what forces the warehouse/lake decisions covered in the data engineering guide.

Network transfer. The Bandwidth Calculator sizes the pipes: serving 10,000 requests per second with 50 KB average response means 4 Gbps of sustained outbound traffic — which determines whether you need a CDN (you do) and what its egress costs will be.

Quota and throttle planning. In a multi-service system, every dependency has limits. The API Rate Limit & Cost Calculator models whether your downstream API budget survives a traffic spike — a capacity question that is also a cost question. And when event counters reach billions per day, the Big Number Calculator keeps the arithmetic exact.

The Token Counter Calculator matters for a modern addition to capacity planning: LLM-backed features. Token volume drives both latency and cost per request, and at scale it can dominate the infrastructure budget.

Practical Tips for System Design

1. Make the application stateless before scaling out. Move sessions, caches, and file storage out of the application process. Everything else follows from this.
2. Choose the shard key once, carefully. It determines which queries are fast forever. The key should appear in most queries and distribute load evenly[azure-sharding].
3. Design for failure, not for success. Networks partition, servers die, dependencies time out. Timeouts, retries with backoff, and circuit breakers are not optional extras — they are the architecture[fowler-distributed].
4. Add caches from the edge inward. CDN for static assets, application cache for hot data, database buffer pool for the working set. Each layer absorbs load the next layer would have handled.
5. Measure before optimizing. The four golden signals (latency, traffic, errors, saturation) tell you which component actually needs scaling — intuition usually guesses wrong.
6. Keep one monolith path viable. Even in microservice architectures, maintain the ability to run the system as fewer, larger services — it is the escape hatch when the operational complexity exceeds the team size[microservices-patterns].

Limitations of the Patterns

These patterns solve scaling problems, not product problems. Sharding makes queries by shard key fast — and queries without it slower. Microservices enable team autonomy — and require operational maturity (orchestration, tracing, deployment tooling) that small teams lack. Eventual consistency unlocks availability — and pushes complexity into every consumer of the data, which must now handle conflicts and stale reads[kleppmann-designing]. The patterns are a vocabulary of trade-offs, not a checklist of best practices: every one of them exists because some other constraint made the simpler option impossible.

Frequently Asked Questions

❓ When should I scale horizontally instead of vertically?

✅ Scale vertically until the cost curve bends or you need failure isolation. Vertical is simpler — no code changes. Move to horizontal when a single machine cannot handle the load, when cost per unit of capacity is rising steeply, or when downtime for upgrades is unacceptable.

❓ What is a shard key and why does it matter so much?

✅ The shard key determines which database instance stores each row. Queries that include the shard key hit one shard (fast); queries without it fan out to every shard (slow). Choose a key that appears in most queries and distributes data evenly — changing it later requires a full resharding migration.

❓ What does the CAP theorem actually say?

✅ During a network partition, a distributed system must choose between consistency (every read sees the latest write) and availability (every request gets a response). You cannot have both while the partition lasts. The practical question is which data can tolerate staleness and which cannot.

❓ Are microservices always better than a monolith?

✅ No. Microservices trade code complexity for operational complexity: distributed transactions, partial failures, network latency, and tracing requirements. A well-structured monolith is usually the right starting point; extract services when scaling or team-autonomy needs appear.

❓ What is the difference between layer 4 and layer 7 load balancing?

✅ Layer 4 routes by IP and port — fast, no request understanding. Layer 7 understands HTTP and can route by URL, headers, or cookies, terminate TLS, and make content-aware decisions. Most modern architectures use layer 7 at the edge.

❓ How do I pick a load balancing algorithm?

✅ Round robin for identical servers and uniform requests. Least connections when request durations vary. Consistent hashing when servers hold local state (caches, sessions) and you want the same client to reach the same server.

❓ What is eventual consistency in practice?

✅ Writes propagate to replicas asynchronously, so a read right after a write may return older data. Most applications tolerate this (a slightly stale like count). Design question: identify which data must be strongly consistent (balances, inventory) and which can be eventually consistent (feeds, analytics).

❓ How much headroom should I provision?

✅ A common rule is 2× peak expected load: one half for handling traffic, the rest for failover, deployments, and traffic spikes. Auto-scaling reduces the need for static headroom but does not eliminate it — scaling takes minutes, and instant spikes still need buffer.