Choosing the Event Store, and Reverting Half of It

Ayush Mania

The decision nobody enjoys

Every platform has one storage decision that is expensive to change later. Ours was where CI test events live. The shape of the problem, as it stood:

Append-heavy. The design target was 10 million events per fifteen minutes.
Deletes are frequent and large. A user deletes a run. A tenant is deleted and the cascade removes everything they ever sent. Both remove a contiguous slice of history, not scattered rows.
Reads are time-range analytics. Flake rates over a window, failure clusters in the last fortnight, trend buckets.
Writes are not append-only. This is the one that matters, and it is the one that is easy to miss.

A test does not arrive as a finished record. It arrives twice. On test:begin we write a skeleton row, because a run that is still executing has to be queryable while it executes. On test:end that row is overwritten with the verdict, the duration, the error, the attachments. In database terms our write path is an UPSERT, not an insert.

Why not ClickHouse

ClickHouse is the obvious answer to "append-heavy event stream at scale", and I spent real time on it before rejecting it. Four reasons, in the order they mattered: The write pattern is wrong for it. Columnar stores earn their advantage on immutable appends. Ours is UPSERT-heavy by design, so the core strength does not apply to us. We would be paying the costs of a columnar store for a workload that does not look like the workload it is good at. Deletes force a segment rewrite. In a store where deletion means rewriting segments, frequent whole-run deletion is not a corner case we can absorb. It is the second most common write operation we have. A different SQL dialect. Not fatal on its own, but it means a second set of tooling, a second set of idioms, and a team that has to context-switch between them. That cost is paid every day, forever, by everyone. No managed offering on our cloud. Whatever operational burden it carried, we would carry alone. Plain PostgreSQL was the other candidate, and it was the close call. It works at the scale we had. But with no automatic time partitioning, deletes degrade as the table grows, and storage grows linearly with nothing to arrest it. Acceptable. Not optimal.

What we chose, and what it cost

TimescaleDB, which is still PostgreSQL. It won on the three properties that actually described our problem. High write throughput. Chunk-based deletes, so removing a time range drops chunks rather than scanning for rows. And no new dialect, no new tooling, no learning curve. Then the part that mattered more than the choice. We wrote down what we were giving up:

Hypertables forbid cross-hypertable foreign keys, so hierarchy integrity moves out of the database and into the application layer, and the partition column has to be carried in every primary key.

That is a permanent tax. Every table in the event path carries run_start_time in its primary key whether it wants to or not, and the guarantees a foreign key would have given us for free are now our job to enforce in code. Writing it down was not ceremony. It meant that six months later, when somebody hit the missing foreign key and asked why, the answer was a paragraph in a decision record rather than an afternoon of archaeology and a suspicion that the original author had not thought about it.

The part the record got wrong

The same decision record predicted significant storage compression. Ten to twenty times. It did not materialise. The plan assumed compressed chunks are effectively read-only. Ours are not, because of the same UPSERT path that ruled out ClickHouse in the first place: a test that begins and then ends writes twice, and the second write can land in a chunk that has already been compressed. That forces decompression on the fly, under load, on the hot path. We reverted compression entirely. Not tuned it, not scoped it to older chunks, not added a carve-out. Reverted, and recorded the reversal in the same document that had predicted the benefit, with the false assumption named in plain words. Notice that the failure came from the same property that drove the original decision. We correctly identified that our write path was UPSERT-heavy, used that to reject one database, and then wrote a compression plan that quietly assumed the opposite. The decision was right and the follow-on plan contradicted its own premise.

Measurement instead of assumption

With compression gone, the storage question was still open, so we stopped predicting and started measuring. Retention reclaimed almost nothing. Dropping old data recovered a small fraction of the table, because the volume is concentrated in recent weeks rather than accumulated over years. This is a growth-rate problem, not an accumulation problem, and the two have completely different fixes. Retention policies address the second. We had the first. The real finding was in the shape of the data. Roughly three quarters of stored rows were framework-level steps belonging to tests that passed. Setup, teardown, fixtures, assertions that succeeded. Detail that nobody opens on a green run, stored at full fidelity forever. So the fix was a data-model change, not a bigger disk, and not a compression setting. That conclusion was not reachable by reasoning about it. It required going and counting.

What I would keep

The decision record is worth more than the decision it records. Three things it bought us: Reversal was cheap instead of embarrassing. When the compression prediction failed there was no argument about whether it had been a good idea, because the reasoning was written down and could be re-read against what actually happened. The assumption was visibly wrong, so there was nothing to defend. The accepted cost stayed visible. A tradeoff you write down is a tradeoff that stops being rediscovered as a surprise. It made the wrong part findable. A record that only contains decisions that worked is marketing. The compression reversal sits in the same document as the original prediction, which is the only arrangement that makes either of them trustworthy. A decision record is only useful if it also records the parts you got wrong.

Share this post: