catalogue / architecture

Architecture, cost & failure

The same curves with dollars and uptime on the y-axis.

6 of 6 topics live on this page

One bigger server or more small ones

The monthly bill for the capacity you need, bought as one ever bigger machine or as a fleet of small ones behind a load balancer. One box is 29× cheaper at 2 vCPUs and stays cheaper up to 128; past 96 the big machines charge more per core, so at 448 vCPUs it costs 3.2× the fleet, and beyond that no bigger box exists: you rewrite for scaling out anyway, in a hurry.

Illustrative on-demand prices, not a vendor quote: $0.05 per vCPU-hour up to 96 vCPUs, rising to $0.20 for the 448 vCPU top box, 730 h a month, no standby for the single box. Scaling out: 8 vCPU boxes at $0.05, 10% of each lost to coordination, 1 spare, plus $1,500 a month ($50 load balancer, $450 shared session store, $24,000 of stateless engineering spread over 24 months). Hitting the wall: a one-off $120,000 rewrite, 5× the cost of building it in.

Capacity needed

Scale up one 2 vCPU box

Scale out 2 × 8 vCPU, $2.08k a month

in use
paid for, idle
spare box
area = vCPUs, same scale in both

box 8 vCPU coordination 10% spares 1 stateless work $24,000 retrofit 5×

scale up: the smallest single box that fits
scale out: 8 vCPU boxes + $1,500/month overhead
one-off rewrite at the wall

At 2 vCPU	Runs	Per month	Per vCPU needed	Status
Scale up	1 × 2 vCPU	$73	$37	cheaper
Scale out	2 × 8 vCPU + LB	$2,084	$1,042	dearer

Read replicas, then shards

A database's layout as traffic grows, 10% of queries writes: one primary, then read replicas, then shards, each step taken when a node passes 80% busy. Replicas multiply reads, but every replica replays every write, so no number of them passes 8,000/s, and they answer from slightly in the past: save something and read it straight back and you get the old value. Only sharding raises the write ceiling, and it costs you queries that must visit every shard.

Each node serves 5,000 reads/s or 1,000 writes/s, and a mix adds up; a replica replays each write at the same cost on one thread, 20 ms behind when idle, lag = 20 ms / (1 - shard writes/s / 1,000); 5% of reads need every shard; dots slowed 30x

Queries per second

step 1 of 5: 1 primary buys one box, and every read sees the latest write. breaks reads and writes fight over one node.

node 5,000 reads/s 1,000 writes/s writes 10% cross-shard 5% step at 80% idle lag 20 ms

read
read that needs every shard
write
write replayed on a replica
node, filled by how busy it is

fresh read 900/s reads, 100/s writes busiest node 28% replica lag none cross-shard reads 0% of read work

queries/s the layout takes, every node under 80% busy
writes/s its primaries take
your traffic
your writes, 10% of it

Step	Takes	Write ceiling	Lag at the top	Buys / breaks
1. 1 primary	2,857/s	800/s	none	buys one box, and every read sees the latest write; breaks: reads and writes fight over one node
2. 1 primary + 2 replicas	4,211/s	800/s	35 ms	buys reads move off the primary; breaks: replicas trail the primary, so a read can be stale
3. 1 primary + 6 replicas	6,154/s	800/s	52 ms	buys more read room, but less for each replica added; breaks: every replica still replays every write; lag unchanged
4. 4 shards, each 1 primary + 2 replicas	15,725/s	3,200/s	33 ms	buys writes split 4 ways, and lag drops; breaks: reads without the shard key hit all 4 (17% of read work); 3/4 of rows moved
5. 16 shards, each 1 primary + 2 replicas	49,709/s	12,800/s	29 ms	buys writes split 16 ways; breaks: 5% of reads hit all 16 and are 46% of read work; 3/4 of rows moved again

One celebrity, one melting shard

16 database shards, each good for 10,000/s, with users placed by hashing their id. Hashing spreads users evenly but not their requests: when a few users are enormous, the shard holding user #1 towers past its capacity while the rest idle, and the lower chart shows that adding shards never takes it below that one user's own traffic.

Assumes 100,000 users sending 80,000/s in total, user k sending a share proportional to 1 / k^skew (a Zipf law), placed by a 32-bit hash of the user id modulo the shard count. The bars sample every request live in your browser and show the rate each shard receives over the last ~0.4 s; the lower chart is the exact sum.

Traffic skew (Zipf exponent)

hottest shard #1 carries 0.0% of requests (an even split is 6.3%) user #1 alone sends 32%

shards 16 capacity 10,000/s traffic 80,000/s users 100,000

shard within capacity
shard over capacity
one dot per 250 requests

hottest shard
user #1's own traffic
even split, 80,000/s / shards

Shards	Even split	Hottest shard	Of capacity	Status
16	5,000/s	32,755/s	328%	overloaded
64	1,250/s	26,209/s	262%	overloaded
256	313/s	26,072/s	261%	overloaded

More shards cannot fix a hot key, because one key always lives on one shard. Split the key itself: salt the celebrity's id into several keys (user:1#0 to user:1#7) spread over shards, give it a shard of its own, or serve its reads from a cache in front.

Restoring the database from backup

The primary is gone and the backup is safe in object storage. The top chart replays the outage: the bar fills stage by stage as the backup comes down, unpacks, lands on disk and replays the day's changes, against a dashed 1 h recovery target (RTO) and a warm standby that takes over in 2 min. Every stage handles bytes, so the restore grows in a straight line with the data, and past 441 GB it cannot finish inside 1 h.

Assumes a 3:1 compressed backup downloaded at 200 MB/s, decompressed at 400 MB/s and written at 500 MB/s, stages run one after another, and a day of WAL (10% of the data) replayed at 50 MB/s

Database size

time since the outage 00:00:00

now

download 200 MB/s compression 3:1 WAL 10% replay 50 MB/s RTO 1 h standby 2 min

Stage	Data	Rate	Time
Download	1.33 TB	200 MB/s	1.85 h
Decompress	4 TB	400 MB/s	2.78 h
Write to disk	4 TB	500 MB/s	2.22 h
Replay WAL	400 GB	50 MB/s	2.22 h
Restore from backup	4 TB		9.07 h 9.1× the RTO
Promote a warm standby	already there		2 min

restore from backup: 2.27 h per TB
promote a warm standby: 2 min at any size
RTO: back within 1 h

A backup you have never restored is a hope, not a plan. At these rates every terabyte costs 2.27 h, so an RTO of 1 h stops being reachable from backup alone at 441 GB. Past that, keep a warm standby replica streaming the changes and promote it: the data is already on its disk, so failover takes minutes whatever the size, and the backup becomes the fallback for the disaster the replica shares (a bad migration, a dropped table). Either way, restore a real backup on a schedule and time it: the number on the page should be one you measured.

What a scaling bug costs every month

The same page of 50 orders served two ways, priced by what each request uses: app CPU, database CPU and bytes sent. Coins drop into each month's bill as the requests flow. Both bills grow in a straight line with traffic, but the buggy one climbs about 51 times as steeply, and the chart turns that gap into dollars a month.

Illustrative prices, not a quote from any vendor: $0.05 per app vCPU-hour, $0.25 per database vCPU-hour, $0.09 per GB sent. Billed per use; uncompressed responses the size of the fetched rows; 30-day month

Requests per month

rows 50 app $0.05/vCPU-h database $0.25/vCPU-h egress $0.09/GB

each dot: 1.02M requests, the same for both
each coin: $20

$3.63k more per month for the same 100M requests $43.5k a year

N+1 with SELECT *
one join, three columns
the difference, paid every month

Endpoint	Queries	Response	Compute /1M	Database /1M	Egress /1M	Total /1M	This month
N+1 with SELECT *	51	401 KB	$0.15	$0.73	$36.1	$37	$3.7k
One join, three columns	1	7.35 KB	$0.03	$0.03	$0.66	$0.72	$72.3

The N+1 is what makes the page slow, but at these prices the SELECT * is what makes it expensive: bytes sent are 98% of the buggy bill. Fixing only the N+1 saves 2%; fetching only the three columns the page shows saves 96%, even with all 51 queries left in.

Every service a request depends on

A request that needs every service on its path is up only when all of them are, so their availabilities multiply: 12 dependencies at 99.9% each give 98.81%, about 4.36 days of downtime a year, and 50 give 17.8 days. Each thin row is one dependency's simulated year; the request path below collects every one of their outages.

Assumes each dependency is up 99.9% of the time (8.76 h down a year), fails independently of the others, and has outages averaging 2 h, about 4.4 a year. Outages are drawn at least 1.5 px wide so they show.

Dependencies on the request path

each up 99.9% outages 2 h critical 3

one dependency's outages
all needed: 0.999ⁿ
only 3 critical: 0.999³ at most
every dependency doubled: 0.999999ⁿ

Request path	Availability	Down per year
One dependency	99.90%	8.76 h
All 12 needed	98.81%	4.36 days
Only 3 critical	99.70%	1.09 days
All 12, each doubled	99.9988%	6.31 min

The fixes are real but not free. Two replicas of a dependency are down together only 0.0001% of the time, but only if they fail independently: a shared region, database, bad deploy or config push takes both down at once and the multiplication comes straight back. Making a dependency optional means writing the fallback (a cached answer, a default, a hidden widget) so its outage degrades one feature instead of failing the whole request.