Data Partitioning & Sharding Strategies

why one machine eventually isn't enough

Replication (see Database Replication) solves availability and read scaling by copying the same data onto multiple machines. It doesn't solve the problem of a dataset that's simply too large, or too write-heavy, for any single machine to hold. Partitioning (sharding) solves that instead: split the data itself across multiple nodes, so each node holds only a slice of the whole.

vertical vs. horizontal partitioning

Approach	How it splits data	PlateRoute example
Vertical sharding	Different tables (or column groups) live on different database instances.	The `orders` table lives on one database, `restaurant_menus` on another, `driver_locations` on a third.
Horizontal sharding	The same table is split row-wise across multiple identically-structured databases.	The `orders` table with 500M rows is split into 8 shards, each holding roughly 1/8 of the orders.

Vertical sharding is a natural first step and maps well onto service boundaries in a microservices architecture (see Monolithic vs. Microservices Architecture). It has a ceiling, though: a single table that's simply too big for one machine still needs horizontal sharding.

two ways to horizontally shard: key-range vs. hash-based

Strategy	How it works	Strength	Weakness
Key-range sharding	Each shard owns a contiguous range of the key (e.g. order IDs 1–1M go to Shard 1, 1M–2M to Shard 2).	Efficient range queries — "give me all orders from March" hits one or few shards.	Hotspots: if today's orders are all recent IDs, one shard absorbs nearly all current write traffic.
Hash-based sharding	`shard = hash(key) % num_shards` — the hash spreads keys pseudo-randomly across shards.	Even load distribution, no hotspots from sequential keys.	Range queries become expensive (data isn't ordered by key across shards); resharding when the shard count changes moves most of the data.


shard_id = hash(order_id) % 8

order_id = 918203  ->  hash(918203) % 8 = 3   ->  Shard 3
order_id = 55010   ->  hash(55010)  % 8 = 6   ->  Shard 6

consistent hashing: fixing hash-based sharding's resharding problem

Plain hash(key) % n has a nasty property: changing n (adding or removing a shard) changes the result of the modulo for nearly every key, forcing almost all data to move at once. Consistent hashing fixes this, and is covered in full depth in Consistent Hashing — the short version is that both keys and shards are placed on a hash ring, so adding or removing one shard only reassigns the small slice of keys nearest to it, not the whole dataset.

rebalancing without a full stop

Even with a good sharding scheme, load doesn't stay perfectly even forever — some shards end up hotter than others as usage patterns shift. Three practical rebalancing strategies:

Strategy	How it works
Fixed number of partitions	Create far more logical partitions than current physical nodes up front (e.g. 1,000 partitions across 10 nodes); as nodes are added, reassign whole partitions to them rather than resplitting data.
Dynamic partitioning	When a partition grows past a size threshold, split it in two and move one half to a different node — adapts to data volume automatically, at the cost of added complexity while serving live traffic.
Partitions proportional to nodes	Keep the number of partitions proportional to the number of nodes; when a new node joins, it takes a random slice of partitions from existing nodes rather than the whole dataset reorganizing.

The common thread: never require moving the entire dataset just because the cluster grew by one machine — that cost has to shrink as the cluster grows, not stay fixed.

reference

AWS — What Is Database Sharding?