I'm starting a series where I build a RAG platform meant to hold up at real scale: millions of chunks, not the 50-PDF demo every RAG tutorial quietly stops at. I'm calling the project Atlas, and I'm documenting it as I go instead of writing it up after the fact, so this is going to read like a build log, not a polished retrospective.
The working assumption for the whole series: Postgres + pgvector as the vector store, self-hosted, no managed vector DB. And to keep the corpus honest about the point where "one Postgres instance" stops being the right call, I'm not letting it be one instance at all. It's sharded across multiple Postgres + pgvector containers from day one, each one capped under a threshold I control. Everything else in this series (per-tenant isolation, CDC-driven incremental indexing, hybrid retrieval, eval) sits on top of that foundation, so it's where episode 1 has to start.
Right now the only code that exists is a downloader, a script pulling omarkamali/wikipedia-monthly off HuggingFace in parquet parts, 972 of them for the full English dump. I've pulled a 10K-row sample so far to get real numbers to design against: it comes out to about 3.7K characters an article, call it 800-900 tokens. The project's target corpus sits around 1-2M chunks, enough that "will this fit in memory" stops being a rhetorical question and starts being an actual constraint I have to design around, not just say out loud.
So before writing a line of embedding or retrieval code, I had to answer two questions: does this get sharded at all, and if so, how do documents actually get mapped to a shard.
Capping the box, on purpose
pgvector, given enough CPU and RAM, can hold 1-2M vectors on one decent instance without much drama. HNSW indexes scale further than people assume. So if I let each container use whatever this machine has to give, a single Postgres+pgvector instance clears the target corpus easily, and the whole case for sharding gets thin fast.
So I'm not letting it. Every Postgres + pgvector container in Atlas gets a hard Docker resource limit, not "whatever's free":
services:
pgvector-shard-0:
image: pgvector/pgvector:pg16
deploy:
resources:
limits:
cpus: "2"
memory: 4GTwo vCPUs and 4GB of RAM per shard. At that budget, HNSW's build memory and query latency stop being comfortable well before 1-2M vectors, the same way they would on a real, cost-constrained production box instead of whatever spare capacity happens to be sitting around. That's a real constraint, not a stylistic one: at this budget, one container genuinely can't hold the corpus, which is what makes sharding the actual answer instead of an architecture flex I bolted on because Glean does it that way.
Capping the box first, then sharding to fit inside it, keeps the architecture honest from day one instead of getting bolted on later under pressure, once it's obvious the single-container version was never going to hold up.
The dumb way to shard: hash(key) % N
Once "shard it" is decided, the obvious first instinct is: hash the document or tenant key, take it mod the number of shards, done. It's one line of code and it looks perfectly uniform in a diagram.
It falls apart the moment the shard count changes. And it will change: a threshold gets hit, a container gets added, capacity planning turns out to have been wrong. With hash(key) % N, changing N from 3 to 4 doesn't just place new keys. It recomputes hash(key) % N for every existing key, and almost none of them land on the shard they were already on.
In the toy example above, adding one shard to a set of three moves half the keys. With a real 1-2M-chunk corpus, that means every shard re-reading and re-writing most of what it already had, while queries either miss or need a fallback fan-out to every shard until the migration finishes. And there's no bound on it: whether it's 10% or 90% of the keys that move on any given resize is just arithmetic coincidence, not something the scheme controls.
Consistent hashing: only the unlucky slice moves
Consistent hashing fixes this by changing what gets hashed. Instead of hashing keys mod the shard count, both the shards and the keys get hashed onto the same fixed ring (say, a 0 to 2⁶⁴ space). A key belongs to whichever shard's point on the ring is the next one clockwise. Each shard actually claims several points on the ring (called "virtual nodes") so its ownership is spread around the ring instead of one lucky/unlucky arc, which keeps load roughly even across shards instead of one shard's single point happening to own a huge stretch.
The property that actually matters: adding shard D only reassigns the ring keys that were closest to shard D's new points. Every key whose nearest clockwise node is still one of A, B, or C's original points doesn't move. It doesn't even know a shard was added. Where naive mod-N gives no guarantee at all, consistent hashing gives roughly 1/N of the keys moving per shard added or removed, and the other (N-1)/N stay exactly where they were, still correctly routed, no rebalance required to keep serving them.
What this looks like as Docker Compose containers
For this project specifically, "shard" means a separate Postgres + pgvector container, each one capped at a chunk-count threshold well under where a single instance's HNSW index and query latency start to degrade. A thin routing layer sits in front, hashes the incoming tenant/document key onto the ring, and forwards to whichever container currently owns that slice. Adding capacity later is "start another container, give it a handful of points on the ring," not "replan the whole cluster."
It's a small decision to front-load, but it's the one everything else in this series depends on: per-tenant isolation only means something if tenants are structurally separable, and structurally separable only means something if adding or removing a shard doesn't require re-touching data that was fine where it was.
Next episode: actually turning the Wikipedia parquet parts into chunks and getting them routed onto shards for real, instead of diagramming the theory of it.