The data triangle

We put requirements on our persistence layer that we rarely say out loud, let alone question. Whatever database we pick, we just assume its reads owe us three things.

• Completeness. When I store a rich, nested object, one coherent piece of data (an aggregate, a document, whatever your stack calls it), I want the whole thing back, however big it has grown.
• Volume. When I ask for a whole collection of those objects, I want all of them. Maybe through a cursor, maybe streamed, but with no ceiling on the count.
• Speed. I want the data now, not later. No pending job, no callback, no "come back in a minute". The most vivid case is a UI, where nobody wants to stare at a spinner. But a microservice calling another one expects exactly the same thing.

Taken one by one, each demand sounds perfectly reasonable. Taken together, they describe a system that hands you everything, whole, instantly.

Here is my claim:

Need is the word carrying the weight. Any engine can be made to serve all three at once. What never happens is a use case that asks for it.

Those three words each hide a nuance, and the article will lean on all of them.

Completeness and volume are both about size, but not the same size. Completeness is one object, volume is how many. An object grows because somebody meant it to. A collection grows because the world keeps turning, and its count has no ceiling. Even the smallest app can picture an endless line of users signing up.

Speed means latency, not throughput. Fast data is data you can use right away, not "by tonight". And fast is not a value, it is a comparison. A system routinely holds two clocks, a fast read for its normal case next to a slow read that everybody accepts because it is rare, or prepared off to the side. Moving a read from one clock to the other is a design decision, not a miracle.

And a reminder, since the opening line slipped it by quickly. This is all about reads. Writes have problems of their own, and this article leaves them alone.

Walk through your own app

Picture a company registry. Any back-office app will do, but let's take that one.

You land on the home page. It shows a lot of companies, and it shows them fast, because you won't stare at a spinner for three seconds. But look at what each row actually contains. A name. A city. A couple of fields you can sort or filter on. That's not the entity, that's a business card.

You click a company. The detail page loads the full record, and it loads it fast, because you're still in a UI. But how many full records are on screen? One. Maybe the previous and the next one, maybe an older revision of the same company. A handful at most.

Later you ask for an export. Every company, every field, the real thing. And the app answers: "we'll email you when your file is ready". Nobody is offended. We all accept that a complete dump takes time.

That's the whole tour. Now rewind it with the triangle in hand.

• The list gave you a lot of companies, fast, but not whole: volume and speed, no completeness.
• The detail page gave you a whole company, fast, but only one: completeness and speed, no volume.
• The export gave you every company, whole, but you had to wait for an email: completeness and volume, no speed.

Three screens, and not once did the third vertex get used.

Naming the business card

Look again at that business card from the list. It's not the entity, and it's not a random crop of it either. It's exactly the slice you need to browse. That object deserves a name.

I'm borrowing the word from video editing, where editors cut with lightweight degraded copies of the footage and only bring the heavy originals back at export time. Same move here: you browse the cheap stand-in, and you load the real entity only when you actually need it.

A lucky overlap

So where does the proxy's data come from? Write down what a proxy needs: the fields that identify the entity, the ones you sort by, the ones you filter on, maybe a thumbnail. Now write down what you put in your indexes: the fields you search and sort on. Two lists, written for two different reasons, and they nearly match.

The match isn't perfect. A thumbnail rarely deserves an index, and a normalized search key never gets displayed.

But the overlap is big enough to change the economics. Most of what a list needs is already sitting in the indexes, the fastest part of your storage. In the best case a covered query draws the whole list almost entirely from the index, leaving the main storage nearly untouched.

Nobody designed the proxy to be cheap. It just happens to be, because search and display want almost the same fields.

This is also why the existing vocabulary didn't fit. The database world has close cousins: CQRS has the read model, MongoDB has the Subset and Extended Reference patterns. Those are dedicated models, shapes designed for reading and often persisted that way precisely for their performance.

A proxy starts from the other end. You take the data you can already reach fast, and when the proxy needs a field that isn't there, you move that one field into the fast storage too. The top-up is cheap, since most of the proxy was already sitting there. And it's not a misuse: delivering lots of small fields quickly is exactly what an index engine is built for.

Read models are designed up front. Proxies start from what was lying around, and you just fill the gaps.

"But I load all my configurations at once"

Fair objection. Some tables really are fetched whole, fast, and nobody loses sleep over it.

The tempting explanation is size: configurations are a handful of rows, companies pile up forever. But growth is a red herring. A carefully managed set can still get too big to grab in one query. And an ever-growing set can stay small in practice: if I sell something at a thousand euros apiece and one day I can't list all my invoices in one go, I'm not in trouble, I'm rich.

The real difference is where a coherent read ends.

A company can be read alone. One record, one page of records: what you see makes sense even if you ignore the other ten million. The collection is just a pile of independent units, and no read ever needs the whole pile.

Your configuration is the opposite. Load half of it and your application may not even start. Same for a set of pricing rules: applying seven criteria out of twelve doesn't give you a rougher answer, it gives you a wrong one. You can stream those rows one by one and never hold more than a handful. But the answer is only right once every last one has been taken into account. The coherent read spans the whole set.

So "load all my configurations" was never about volume. That table is one aggregate wearing table clothes, and fetching it whole is completeness and speed again. The volume vertex only concerns the piles: collections whose unit of meaning is the single entity, and which you therefore never need whole.

Stretch that to its limit and it becomes a prerequisite. Some databases hold, in truth, a single entity chopped into tables and rows. Technically a database, actually one big object wearing a whole schema as clothes. The triangle has nothing to say about it, because its volume is one. Everything in this article assumes what the opening bullet already implied, a store of many distinct entities rather than one entity in pieces.

Where the reflex comes from

If we never need all three, why do we insist on having them? Two things: insurance and habit.

The insurance is our favorite clause as developers: "you never know". Offered a tool that covers the need and a tool that covers everything, the bigger one always feels safer, so the requirement quietly stretches to "everything, whole, instantly" before anyone asked what the screen in front of us actually needs. Over-asking costs nothing on the day you write the requirement. It gets expensive later, when the whole architecture is shaped around needs nobody has.

The habit comes from SQL.

Relational databases grew up when disk was the expensive resource, and normalization fit that world perfectly. Store each fact exactly once. It buys two things. Coherence, because a fact stored once can never disagree with itself. And a minimal footprint, which back then was a survival constraint, not a nicety. The price is that a business object ends up spread across several tables, and indexes are what make that layout fast enough to walk.

Half of that bargain has now expired. Coherence still matters as much as ever. But disk is the cheapest thing in the room, and we happily spend it. MongoDB's official Document Versioning pattern copies the entire current document into a revisions collection on every change. The whole document, not a diff. Nobody blinks.

Here is the part I find funny. SQL never actually served the three vertices with one mechanism. Typically, your paginated list runs on an index-only scan, your detail page on a primary key lookup, and your export on a full scan. Three different access paths, hidden behind one query language. The monolithic database didn't refute the triangle. It buried it inside the query optimizer.

Not the first pick-two

Databases have already taught us this kind of trade-off once, with the CAP theorem: a distributed data store cannot guarantee all three of these at once.

• Consistency: every read sees the latest write.
• Availability: every request gets an answer.
• Partition tolerance: the system keeps working when the network splits.

That diagram made "pick two of three" part of every architect's vocabulary. But my triangle is a weaker beast, and the difference matters. CAP is a proven impossibility: no distributed system can deliver all three, however well built. The data triangle proves nothing of the sort, and a small SQL database happily serves completeness, volume and speed at once. It is an observation of non-need: no real use case ever asks for all three at the same time, and the phantom requirement is expensive.

Try to break it

One disclosure before you start: I am biased. I have used this rule for years, and by now I read every workload through it. People keep bringing me cases that seem to need all three vertices, and so far each one has lost a vertex under inspection. The "complete" object was a projection. The "fast" meant throughput, not latency. So I hold the strong version. The vast majority of cases can be approached with two vertices, and it pays to hunt for that reading, because the system it allows is the least demanding one to build. When a case seems to be the exception, re-read the need before scaling the architecture, and see if it can be bent toward two. And if it still resists, I won't tell you that you are wrong. I will tell you to think it through, because the interesting cases live exactly there.

When a case truly seems to hold all three, look at the store itself first. Odds are it is one aggregate in disguise, and its volume was one all along.

The usual suspects are already dissected in the annexes: the offline sync, the ad server, in-memory engines, the nightly billing and friends, each taken apart until a vertex falls off. Check there before concluding you hold an exception.

And one more honest caveat: the triangle only starts to bite at scale. A single PostgreSQL serves all three vertices without breaking a sweat for far longer than we like to admit. The rule is a lens for when your data outgrows that comfort. It is not an instruction to over-architect a small app.

One tool to store, another to search

The same SQL heritage gave us another reflex: we index our data with the same tool that stores it. That's a shame, because dedicated search engines exist, and they are very good. Solr opened the road back in 2004, Elasticsearch made the category mainstream, and OpenSearch is its Apache-licensed fork, born when Elastic changed its license in 2021. All three are built on Lucene.

The pain they remove is real. MySQL and MongoDB work best when one well-chosen compound index matches the query. Both can combine several indexes. But MongoDB's own docs warn that intersection "does not eliminate the need for creating compound indexes", MySQL's Index Merge ships with a documented list of limitations, and either way compound indexes shaped for the common queries remain the workhorse. Now give your registry the classic enterprise search form, the one that filters and sorts on every field. You either create a compound index per combination, which explodes, or you index the most common searches and cross your fingers.

I'll tip my hat to PostgreSQL here: its bitmap scans combine several single-column indexes in one query, and it mostly just works.

Once the index is its own component, something clicks: the storage side no longer needs to be fast. Even a plain object store like S3 is enough, and the triangle explains why.

• The list is served by the index alone: the proxy lives there.
• The detail page needs a handful of full objects, and fetching small objects from S3 answers in 100 to 200 milliseconds. The ten reads run in parallel, so the page pays for the slowest of the ten, not their sum, and a detail view that appears in a fifth of a second is a perfectly ordinary page.
• The export streams everything straight from storage, and slow is accepted there.

Each edge of the triangle is served by the component that's good at it, and the storage bill is tiny.

One assumption hides in that split, and it deserves to be said out loud. The storage side never has to coordinate a write across several objects. That is less demanding than it sounds. Most applications already save one object at a time and keep the cross-object rules in application code, whether they were designed around aggregates or straight against a database. If your domain truly needs multi-object transactions at write time, that job belongs to a transactional database, and an object store will not volunteer for it.

I won't pretend the split is free. Two systems means keeping them in sync, and no transaction spans both. The classic sting is read-your-writes: a user creates an entity and the list doesn't show it for a second, because the index refreshes asynchronously. And the day you change your index mapping, you re-read everything from storage to rebuild it.

These are real costs. They're just the same costs you already pay the moment you add any cache or read model, and there is mature tooling for them.

Could we always have done this?

I wondered which came first: does the triangle justify splitting index from storage, or did index engines teach us to see the triangle?

Neither, I think. The demand side was always pairwise. A 1985 mainframe app had fast partial transaction screens, single-record lookups, and complete reports printed overnight. Same three edges, same pairs. By the nineties the industry had even admitted it in writing: the data warehouse exists precisely because the transactional database can't serve both the interactive side and the complete-bulk side.

What changed is the economics. Back then, a second system meant a second machine and a second admin, so the all-in-one database was the rational choice. Then Lucene, S3 and Elasticsearch turned the separated answer into commodity infrastructure, one piece at a time.

The triangle was always true on the demand side. Scale made it visible, and cheap tools made it actionable.

Pick two, on purpose

Next time you design a feature, place it on the triangle. Which two vertices does this screen actually need? You're not measuring the result here. You're choosing which constraints the design must guarantee, and two are always enough. Store and index accordingly. And when a requirement seems to demand all three at once, don't reach for a bigger database yet.

Re-read the requirement. Chances are one of the three corners is a habit, not a need.