Have you heard? Clickhouse is winning the observability wars! <-- THIS IS SARCASM

Edited on Oct 16th to add: if you were sent here by the Clickhouse sales team, as proof that “Even Charity Majors says we are winning the observability wars”, that is untrue. This piece is not even about Clickhouse. It’s about observability vendors being bottomfeeding chucklefucks chasing the false gold of “cheaper Datadog” instead of building better products for their customers.

TLDR: Clickhouse is a great storage engine, and next generation observability tools require a next generation engine. But the storage engine isn’t the win. The data model is the win, and the storage engine is what enables it. You can use a modern, cutting edge storage engines to offer your customers the same old shitty three pillars experience at sky high cost, and that is what most vendors have chosen to do.

What a waste.

We interrupt our regularly scheduled series of posts on AI norms and values to bring you this incredible piece from Mat Duggan, “Why Clickhouse Is Winning the Observability Wars,” which you should go read right now. Go! I’ll wait.

Mat starts by describing the experience every developer starts with, logging on the command line — “the observability equivalent of a first kiss, that ruins you for everything after” — followed by the descent into hell as four services becomes forty, becomes four hundred, and the impossibility of satisfying stakeholders in engineering, data, customer support, and executives, who all want different impossible things before breakfast.

As Mat says, at 1TB a day, every modern observability stack is fine. Pick something and be productive. But at 10TB/day, they all become unmanageable — all, he says, except Clickhouse.

(My favorite part of the rant, because I am a bad person and/or an SRE, is where he meticulously details exactly how Elastic, LGTM, and Datadog [see errata posted at the end] fall apart at scale. This is a boy with scar tissue, respect.)

Few observability tools can handle 10TB/day

It’s hardly news that data tools tend to start off easy to run, but require teams to run at scale. Clickhouse, though — let’s hear Mat gush:

“Every other observability backend I’ve worked with mutates as it grows... ClickHouse at 10 TB a day looks like ClickHouse at 1 TB a day with more shards. That’s it. That’s the pitch. That’s the whole reason I’m writing this.”

Mat admits that Clickhouse has a small tax up front, but worth it since a tiny bit of upfront effort buys you SO MUCH EASE down the road, as you scale indefinitely, with high cardinality data and no messy schema lock-ins or performance cliffs.

Wow, this is starting to sound like a Clickhouse ad, isn’t it? Why am I boosting Clickhouse's reputation like this?

Because it isn't just fucking Clickhouse, you numbnuts.

Observability backed by columnar storage is a different class of tool than the three pillars

Mat’s piece isn’t actually about who is “winning” the observability wars, it’s about how Mat just discovered what it's like using observability powered by a columnar storage engine, and it is blowing his everloving mind.

And it should. It is genuinely better, and this is genuinely exciting!

And we have been talking about this for the past ten years.¹ How is this still news to people?

We wrote about this at length in the first edition of “Observability Engineering”, devoting over a tenth of the book to this topic. When writing the second edition, we invited Clickhouse to contribute a guest chapter for the same reason — THIS IS WHAT OBSERVABILITY SHOULD FEEL LIKE IN TWENTY FUCKING TWENTY-SIX.

I don’t mean to be all “I told you so”, but look! what a difference! this shit makes! if it can even make a grizzled logs warrior like Mat fall so madly in love. 💕

Why haven’t you heard about this?

I have a theory why it is that people still don’t get this, even people like Mat who are clearly steeped in the space.

My theory is that anything one vendor says gets written off as self-interested marketing. It isn’t until multiple vendors link arms and say the same thing together that people sit up and note that the landscape really has changed.

I always assumed that would happen naturally once more observability companies were designed and built on columnar storage. We would align on some shared technical vocabulary due to our keen shared interest in clarifying the landscape for buyers, to help them understand how much better it is. The upstarts vs the incumbents. The shiny new way vs the kludgey, slow, painful old way.

Well, I was half right. Every observability company founded post-2019 has been built on top of a columnar store, did you know that? But not only have they not linked arms with us, they keep insisting there’s nothing all that new or different about what they’re building. They’re all selling “Datadog, but cheaper.”

Why are newer observability vendors nerfing their own products and obscuring the difference?

As a consequence, most of these newer observability tools are, despite being built on columnar storage engines, not that different from Datadog.

Same old three pillars. Same old problems with cardinality and scale, give or take. They are doing this because the weakness that they sense in Datadog is all price.

The weakness I sense in Datadog is all product.

Yes, it’s cheaper to run on columnar storage. But lower price is a consequence of better architecture, and honestly, it’s one of the least interesting consequences. Infra logs and metrics are commodities, but observability for your own products, your own code? That should be an investment.

Everybody thinks Datadog is expensive because they choose to be expensive. This is true in part. But they also don’t really have a choice. Their business model has grown up around a thirty-year-old architecture, and they are fully locked in to it.

Will AI be enough to finally shake off the death grip of the three pillars?

As AI starts accounting for a larger share of workloads, people are beginning to realize that the trace is all that matters. The trace — or wide, structured canonical logs, one or the other — is all you need.

One powerful, context rich data set is worth more than the sum of its pillars, because relationships are what makes data valuable. Slice it up into pillars, and you destroy that value for good.

One way or another, people are gonna figure this out, and that poses an enormous threat to Datadog’s business model, which relies on storing (and charging for) the same data over and over and over and over again in different formats, then storing (and charging for) links between datasets, and all the attendant fees and costs.

Or maybe people won’t figure it out. God knows I never thought the swindle could go on this long.

Is Clickhouse winning the observability wars? God I hope so

Clickhouse has used the term “observability 2.0” two times by my count, making them the only other vendor who has. Why is Clickhouse willing to link arms, kinda, but no one else has?

My theory is because they are a database company, not an observability company, and therefore less attached to marketing themselves as a cheaper Datadog. Unlike most observability vendors, they have decided to prioritize building something that solves customer problems over building something easy to market and sell.

Clickhouse, if you’re reading this, I would love to coordinate more. Call me. OMG this was not an invitation for every clickhouse partner, vendor, and third party startup to reach out, please stop.

So yeah. If you want to run your own logs clusters, you should use Clickhouse. If you don’t want to DBA your own clusters or deal with the overhead (which scales linearly, as Mat points out, but is nontrivial), you should use Honeycomb.

It really is a whole new world. Just ask Mat Duggan.

~charity

Errata: As written, my essay suggests that Datadog is not on columnar storage. This is false. They ARE on columnar storage — a truly impressive engine called Husky — and have been since 2022. However, the architecture has not changed. They collect metrics, logs, traces, exceptions, errors, profiling data, and every other type of signal separately. You pay to store it separately, again and again, and then you pay for every bit you’d like to correlate across any pair of datasets.

That’s what I mean, when I say they’re running on an ancient architecture, and it is astronomically expensive and objectively worse from a product perspective.

As I’ve written before: your data is made powerful by context. The richer your data, the more exponentially, combinatorially powerful it becomes. When you break it apart at write time, you cripple its capacity forever.

This is fine for infrastructure logs and metrics — the exhaust pipe of third party software. It is not fit for your crown jewels, your own source code, the telemetry you use to make sense of your own product and customers. It is both cheaper and exponentially more powerful to store that telemetry in a single source of truth, then derive individual metrics, logs, traces, errors, etc from that singular source.

My apologies to Datadog. I am not annoyed at Datadog in the slightest — I have a great deal of respect for their engineering org and go to market machine. I am annoyed at newer observability vendors who could do better and choose not to.