Charity Majors
· 11 min · Original

If your team is happy, are you doing a good job?

For those of you just catching up, Cat and I are having a dialogue about how teams are changing in the era of AI. Part 1: Where do AI norms and values come from? Where should people start?, by me. Part 2: How social identity is constructed, and how to defang the threat of identity loss, by Cat. ✨ Cat and I will be doing an AMA on October 21st: “Debugging Your Team: An AMA on AI, Trust and Identity in Engineering Teams” .. sign up now! ✨ At the end of Cat’s piece, she ended with a question for me and an observation of her own. She asked, “What gets us so stuck on believing we can’t have both high performance and good norms?” And observed: “It’s hard to get teams to open up, hard to get our companies to see the merit in this kind of practice, and hard to get people to see this as the hard cognitive science of problem-solving instead of soft, squishy feelings.” All of these, I believe, are related. If leaders think that “treating people well” means “nobody feels bad”, of course they assume it trades off against performance. If people associate learning with soft, squishy feelings instead of an uncomfortable growth process, they probably aren’t really learning. And because we’re wired to see people where there really are systems, we keep looking in the wrong place for fairness and equity. This is a weird timeline. I’ll tell you how I think we got here. It starts when the idea of psychological safety upended our mental model about what excellence looks like, just over a decade ago. The dark days of mad ‘boy geniuses’ in tech Psychological safety entered the tech lexicon in 2015, when Google published the findings of Project Aristotle, their two-year inquiry into engineering team effectiveness. The research showed that high performing teams were differentiated by five…

Charity Majors
· 5 min · Original

Where do AI norms come from?

After I posted “Confessions of an Unrepentant Slop Snob” on LinkedIn last week, with links to the Honeycomb AI norms and values docs, I got this question from Niklas Lochschmidt: I am curious if you could share more light on the process? Any guidance you would give people in companies that have the friction and internal debates, but maybe haven’t arrived at shared norms and values yet? That’s a great question, and I will answer in just a minute. I feel like this is a great way to kick off something I’ve been looking forward to all year — an old fashioned bloggy dialogue between myself and Dr Cat Hicks, research psychologist to the stars engineers, on the topic of norms, ethics, and learning in the era of AI. In which Cat and I were both writing books at the same time When Cat’s publisher reached out to me in November of 2025 about reviewing her book, “The Psychology of Software Teams”, I .. didn’t notice the email for a month, then put off reading it for another two months, because I was in the middle of my own private hell trying to get the second edition of Observability Engineering written, then rewritten, then written again.1 🫠 But once I did pick up her book, I downed it in a sitting. Then I wrote a long and gushing fan letter to Cat and her publisher, with bullet point lists (yes plural) of ideas for ways that we could collaborate or I could support her work.2 What struck me, reading her book, was how much overlap and resonance there was between what she was writing and what I was writing, despite being radically different books in almost every conceivable way. Hers: a slim, disciplined 150 pages of text, plus another fifty pages of reference notes and citations, presenting common misconceptions about high-performing teams, what the research actually says, and…

Charity Majors
· 12 min · Original

Confessions of an Unrepentant Slop Snob

This is a peek into the research behind our “AI Norms and Values” docs, which we developed over the summer and recently shared on our site. If you’re looking for those, go here: How Honeycomb does business, the principles we all abide by Why Honeycomb engineering is embracing AI AI norms and values (and ethical issues we have a stance on) Back to our story. Earlier this year, I was spending a lot of time stewing over why everyone around me seemed so frazzled and on edge. Half the company was spitting mad about all the slop they were getting. Instead of receiving five crisp bullet points, they were getting twenty-page docs full of padding and slop. They would ask a colleague a question, and the reply would begin with “Claude says…” These folks felt like their time and attention were being not just taken for granted, but actively abused. The other half of the company felt equally injured. They were working faster and delivering better outcomes than ever, and wasn’t this exactly what we had asked of them? They were more upset about the fact that some people hadn’t updated their workflows in years. Of course you can’t keep up if you aren’t willing to adapt, they protested. Everyone was mad. One side wanted to place limits on AI (“I am so sick of reviewing docs the sender didn’t even read!”). The other side wanted us to force people to use AI (“I am so sick of reviewing docs riddled with basic errors that a single pass would have caught!") I was, at different times, in both camps. But the thing that bothered me most was how often I kept hearing the word “dehumanizing”. People said they no longer felt like they had a human connection with their coworkers anymore. This was new. On the one hand, AI is not special. On the other hand…is it? My first instinct was to say that…

Charity Majors
· 9 min · Original

Have you heard? Clickhouse is winning the observability wars! <-- THIS IS SARCASM

Edited on Oct 16th to add: if you were sent here by the Clickhouse sales team, as proof that “Even Charity Majors says we are winning the observability wars”, that is untrue. This piece is not even about Clickhouse. It’s about observability vendors being bottomfeeding chucklefucks chasing the false gold of “cheaper Datadog” instead of building better products for their customers. TLDR: Clickhouse is a great storage engine, and next generation observability tools require a next generation engine. But the storage engine isn’t the win. The data model is the win, and the storage engine is what enables it. You can use a modern, cutting edge storage engines to offer your customers the same old shitty three pillars experience at sky high cost, and that is what most vendors have chosen to do. What a waste. We interrupt our regularly scheduled series of posts on AI norms and values to bring you this incredible piece from Mat Duggan, “Why Clickhouse Is Winning the Observability Wars,” which you should go read right now. Go! I’ll wait. Mat starts by describing the experience every developer starts with, logging on the command line — “the observability equivalent of a first kiss, that ruins you for everything after” — followed by the descent into hell as four services becomes forty, becomes four hundred, and the impossibility of satisfying stakeholders in engineering, data, customer support, and executives, who all want different impossible things before breakfast. As Mat says, at 1TB a day, every modern observability stack is fine. Pick something and be productive. But at 10TB/day, they all become unmanageable — all, he says, except Clickhouse. (My favorite part of the rant, because I am a bad person and/or an SRE, is where he meticulously details exactly how Elastic, LGTM, and…

Charity Majors
· 4 min · Original

In defense of AI mandates

I’ve been writing a series of pieces on lessons learned from our AI journey at Honeycomb. I’ve written about the tension between enthusiasts and skeptics, the need for engineering rigor, and the ethics of using tools with externalities and a seedy backstory. Today I want to send you off to a long July 4th weekend with a short but passionate defense of that most despised of management tools, the technology mandate. Nobody likes mandates I have read many a tweet or post from engineers exploding with rage over the pointless, counterproductive AI mandates they have endured at work. I have also read many a post from CEOs and execs bragging about their mandates, how decisively they upskilled, how fast they are moving now, and — lest we forget — how many people they laid off as a result of their new, AI-juiced incredibleness. (Boy, I wonder why people aren’t excited?) Sometimes mandates are stupid and punitive and shortsighted and dumb. But not always. A mandate is a way to fund the change The mandate is one way of putting organizational muscle behind a decision. It's a funding mechanism. It works by acknowledging that “hey, we are all going to be a little slower for a bit while we figure this out, and it will be annoying and expensive and we accept that.” It’s a way of letting managers know that we know some deadlines and standards will slip, and that's ok. We are all going to go through this hard thing together and have each other’s back, because this is important. We accept the tradeoffs. Any time you devote resources to a goal, you need to account for where those resources are coming from. What can people put down or let slip? What work are you choosing not to do? How will you know if it was worth the time and effort? A mandate is a forcing function for identifying a…

Charity Majors
· 15 min · Original

Is it ethical to use AI?

I am a contrarian who likes to argue and complain a lot. Because of this, I have never been inclined to block people who argue with me or complain about my work, or even lash out at me with some hostility. I could say a lot of noble sounding things about how I value debate and open discourse, and those things would be true, but I also just feel like I should tolerate other people as much as they have to tolerate me. I recently wrote a piece about AI enthusiasts vs AI skeptics — a very mild piece, I might add, almost repulsively brimming over with both-sides-are-good-people’s and can’t-we-all-just-get-along’s. Yet I have blocked more people in the past three weeks than the past ten years.1 There is a fear in the water right now that is bringing the crazy out in all of us. The stakes are not low. The world is burning, after all. CEOs go on job-murdering sprees and the Industrial Revolution may be coming for knowledge workers. Even the Pope is alarmed. AI is not special It bothers me when I see people holding AI up like it’s something special — uniquely evil, incomparably harmful, irreparably tainted. It is none of those things. AI is just technology. Some technologies are more damaging than others — knives are less damaging than guns, Facebook for colleges was less damaging than Facebook for Myanmar — but we always discover risks before we know how to govern them.2 There is always a gap while we try to catch up. That gap is not proof that AI is evil. It is proof that we have work to do. The fact that we have not solved the problems yet is not an argument to disconnect. It is an argument to engage, especially if you work in technology and already have an arsenal of relevant skills. You do not learn to govern a tool by refusing to touch it. You learn by using it and…

Charity Majors
· 16 min · Original

AI demands more engineering discipline. Not less

A few days back I wrote a piece called “AI enthusiasts are in a race against time, AI skeptics are in a race against entropy.” I have notes on a whole pile of AI-related topics that I’d like to cover in depth: AI mandates, communication norms, code review, AI art, and more. Unfortunately, I got too many interesting responses to my last piece, and now I have to address those before I can move on to other topics. 😉 There were two types of interesting responses: the first on the technical merits, the second on ethical grounds. I will respond to each of these separately. Let’s take the technical side first, because it’s easier. (Edited to add: here is the ethical response) Somehow, a subset of readers came away believing I was telling everyone to ditch code review and push their shittiest code straight into production without reading it, right now, tout suite.1 That is not what I am doing. That is not what I think you should do. But I did not pick that example at random, and I will tell you why. In 2025, the question was whether AI could ever generate “good” code It’s easy to forget, but for most of 2025, the idea that AI-generated code was slop and might always be slop was not only a reasonable position to hold, it was the default, mainstream position.2 That question was answered decisively last November. Ever since Opus 4.5 came out, AI has been able to generate code that is approximately as good as that of the median software engineer, at least for common patterns, and much faster and more cheaply. I came out of a book hole and realized this in January, and over the first few months of 2026, it seemed like everyone around me was having a similar realization. But many saw it coming much sooner. The popular narrative holds that Opus 4.5 was what changed. But Opus 4.5 was…

Charity Majors
· 15 min · Original

AI enthusiasts are in a race against time, AI skeptics are in a race against entropy

I recently attended a talk where one of the presenters made some pretty…astonishing claims about what they had achieved by the pure, uncut power of vibe coding. Difficult engineering problems solved, backlogs cleared. Rewrites that would have taken a year or more in the beforetimes, now whipped out in a few short weeks of prompting. Afterwards, wandering around the conference, I caught a lot of excited chatter: “I can’t wait to make my teams watch the recording of this talk. My engineers are SO resistant to the idea of shipping code without reading it. Finally, some proof they can’t ignore!” “Mine are too. It’s so frustrating. People are just so stuck in what they know. I think they’re just scared of being replaced, you know?” The talk was fantastic. The presenter made it all sound easy, breezy and oh-so-fun. The problem is, I know lots of other people at his company, and they described these projects as a horror show. Yes, they allowed, some progress was made, and some of it was pretty cool, but he also left a long, fiery trail of chaos in his wake. Months later, some teams were still grinding through waves of cleanup work. (Please don’t @ me to ask if I am subtweeting your talk. I am subtweeting MANY TALKS. This is a composite.) I keep thinking back to this episode — the highly selective version of the story that was told on stage, and the room full of AI enthusiasts who seemed to be eating it up with a spoon, uncritically, because it so validated everything they wanted to be true. I keep thinking about the certainty they took home with them, and wondering how that energy fed into conversations with their teams. People are retreating into camps and circling the wagons There is a yawning chasm opening up between…oh, let’s call them the enthusiasts and the skeptics,…

Charity Majors
· 8 min · Original

Your Data Is Made Powerful By Context (so stop destroying it already)

After twenty years of devops, most software engineers still treat observability like a fire alarm — something you check when things are already on fire. Not a feedback loop you use to validate every change after shipping. Not the essential, irreplaceable source of truth on product quality and user experience. This is not primarily a culture problem, or even a tooling problem. It’s a data problem. The dominant model for telemetry collection stores each type of signal in a different “pillar”, which rips the fabric of relationships apart — irreparably. Your observability data is self-destructing at write time The three pillars model works fine for infrastructure1, but it is catastrophic for software engineering use cases, and will not serve for agentic validation. But why? It’s a flywheel of compounding factors, not just one thing, but the biggest one by far is this: ✨Data is made powerful by context✨ The more context you collect, the more powerful it becomes Your data does not become linearly more powerful as you widen the dataset, it becomes exponentially more powerful. Or if you really want to get technical, it becomes combinatorially more powerful as you add more context. I made a little Netlify app here where you can enter how many attributes you store per log or trace, to see how powerful your dataset is. 4 fields? 6 pairwise combos, 15 possible combinations. 8 fields? 28 pairwise combos, 255 possible combinations. 50 fields? 1.2K pairwise combos, 1.1 quadrillion (2^250) possible combinations, as seen in the screenshot below. When you add another attribute to your structured log events, it doesn’t just give you “one more thing to query”. It gives you new combinations with every other field that already exists. The wider your data is, the more valuable the data…

Charity Majors
· 7 min · Original

My (hypothetical) SRECon26 keynote

Hey, it’s almost time for SRECon 2026! (I can’t go, but YOU really should!) Which means it was almost a year ago that Fred Hebert and I were up on stage, delivering the closing keynote1 at SRECon25. We argued that SREs should get involved and skill up on generative AI tools and techniques, instead of being naysayers and peanut gallerians. You can get a feel for the overall vibe from the description: It’s easy to be cynical when there’s this much hype and easy money flying around, but generative AI is not a fad; it’s here to stay. Which means that even operators and cynics — no, especially operators and cynics — need to get off the sidelines and engage with it. How should responsible, forward-looking SREs evaluate the truth claims being made in the market without being reflexively antagonistic? Yep, that was our big pitch. Don’t be reflexively antagonistic. You should learn AI so that your critiques will land with credibility. That is not the message I would give today, if I were keynoting SRECon26. SRE not sorry I came out of a hole, and the world had changed I’ve been in a bit of a hole for the past few months, trying to get the second edition of “Observability Engineering” written and shipped. Maybe the hole is why this feels so abrupt and discontinuous to me. Or maybe it’s just having such a clear artifact of my views one year ago. I don’t know. What I do know is that one year ago, I still thought of generative AI as one more really big integration or use case we had to support, whether we liked it or not. Like AI was a slop-happy toddler gone mad in our codebase, and our sworn duty as SREs was to corral and control it, while trying not be a total dick about it. Today, it’s very clear to me that the center of gravity has shifted from cloud/automation workflows to…

Charity Majors
· 10 min · Original

First I wrote the wrong book, then I wrote the right book

This is the second in a two-part episode. The first part ended on a ✨cliffhanger!!!✨ — so if you missed the first episode, catch up here: Six long weeks of writer’s block I was merrily cranking away what I believed to be my last chapter when I asked the internet — YOU guys — for help the first time. “Are you an experienced software buyer? I could use some help,” went up on September 19th, 2025. The response was overwhelming. I heard from software engineers, SREs, observability leads, CTOs, VPs, distinguished engineers, consultants, even the odd CISO. All these emails and responses and lengthy threads kept me busy for a while, but eventually I had to get back to writing. That’s when I discovered, to my unpleasant surprise, that I couldn’t seem to write anymore. “Well,” I reasoned, “maybe I’ll just ask the internet for EVEN MORE advice" — and out popped Buffy-themed post number two, on October 13th. Keep in mind, I thought I would be done by then. November was my stretch deadline, my just in case, I better leave myself some breathing room kind of deadline. As November 1st came and went, my frustration began spiraling out into blind panic. What the hell is going on and why can I not finish this??? In which I finally listen to the advice I asked for A week before Thanksgiving, I was up late tinkering with Claude. I imported all the emails and advice I had gotten from y’all, and started sorting into themes and picking out key quotes, and that is when it finally hit me: I had written the wrong thing. No, this deserves a bigger font. ✨I wrote the wrong thing.✨ I wrote the wrong thing, for the wrong people, and none of it was going to move the needle in any meaningful way. The chapters I had written were full of practical advice for observability engineering teams and platform…

Charity Majors
· 7 min · Original

Martin Fowler told me the second edition should be shorter (it's twice as long)

Last week I got to meet Martin Fowler for the first time in person. This was an exciting moment for me. Martin ranks high on my personal pantheon; he is, so far as I can tell, hardly ever wrong. Martin Fowler, me, and Nathen Harvey, at the happy hout for Gergely’s Pragmatic Summit! Not pictured: my glass of wine, Nathen’s box of fucks to give After letting me know that he “only hugs on this side of the Rockies” (noted), the topic of writing books came up, to which he drawled, “The second edition should always be shorter. I always made my second editions shorter. Shorter books are better books.” God dammit, Martin.. Now you tell me. 🙈 In related news, our last few chapters for “Observability Engineering, 2nd edition” go out to tech reviewers this week! This puts us on track for dead tree publication in June, although chapters will be available earlier for O’Reilly subscribers as well as behind an email gate on the Honeycomb site. What’s different about the second edition? Almost everything. The only chapters that carry over some material are the ones on sampling, retriever (columnar store), and a smattering of the SLO stuff—maybe 10% all told? And we’ve added a monstrous amount of new material. So no, it will not be shorter than the first edition. It is almost twice as long.1 (Sorry!) On the bright side, I do think it will be clearer, more usable, and useful to a wider range of contributors, all over the org chart, than our messy little first edition ever was. The first edition was a spaghetti mess Books, as I understand, are like children; if you made them, you are not allowed to say you aren’t proud of them. So fine, I won’t say it. But I think we can all privately agree that the first edition was a bit of a hot mess. No shade on my wonderful co-authors, Liz and…

Charity Majors
· 11 min · Original

Bring Back Ops Pride

I was planning to write something else today, but god dammit, I got nerd-sniped. Last week I published a piece on the Honeycomb blog called “You Had One Job: Why Twenty Years of DevOps Has Failed to Do It.” Allow me to quote myself: In retrospect, I think the entire DevOps movement was a mighty, twenty year battle to achieve one thing: a single feedback loop connecting devs with prod. On those grounds, it failed. Not because software engineers weren’t good at their jobs, or didn’t care enough. It failed because the technology wasn’t good enough. The tools we gave them weren’t designed for this, so using them could easily double, triple, or quadruple the time it took to do their job: writing business logic. This isn’t true everywhere. Please keep in mind that all data tools are effectively fungible if you can assume an infinite amount of time, money, and engineering skill. You can run production off an Excel spreadsheet if you have to, and some SREs have done so. That doesn’t make it a great solution, the right use of resources, or accessible to the median engineering org. It’s a fun piece, if I do say so myself, and you should go read it. (Much stick art!) I posted the link on LinkedIn yesterday morning. Comments, as ever, ensued. (The internet was a mistake, Virginia.) “Devs should own everything” A number of commenters said things like, “devs should own everything”, “make every team responsible for their own devops work”, and my personal favorite: “I still think the main problem is with the ownership model - the fact that devs don’t own the full system, including infra and ops.” (Courtesy of Alex Pulver, who has graciously allowed me to quote him here by name, adding that “he stands firmly behind this 😂”.) As it happens, I have been aggressively advocating for the…

Charity Majors
· 5 min · Original

On Friday Deploys: Sometimes that Puppy Needs Murdering

‘Tis the season that one of my favorite blog posts gets pulled out and put in rotation, much like “White Christmas” on the radio station. I’m speaking, of course, of “Friday Deploy Freezes are Exactly Like Murdering Puppies” (old link on WP). This feels like as good a time as any to note that I am not as much of an extremist as people seem to think I am when it comes to Friday deploys, or deploy freezes in general. (Sometimes I wonder why people think I’m such an extremist, and then I remember that I did write a post about murdering puppies. Ok, ok. Point taken.) Take this recent thread from LinkedIn, where Michael Davis posted an endorsement of my Puppies article along with his own thoughts on holiday code freezes, followed by a number of smart, thoughtful comments on why this isn’t actually attainable for everyone. Payam Azadi talks about an “icing” and “defrosting” period where you ease into and out of deploy freezes (never heard of this, I like it!), and a few other highly knowledgeable folks chime in with their own war stories and cautionary tales. It’s a great thread, with lots of great points. I recommend reading it. I agree with all of them!! If you can’t move swiftly with confidence, you need workarounds For the record, I do not believe that everyone should get rid of deploy freezes, on Fridays or otherwise. If you do not have the ability to move swiftly with confidence, which in practice means “you can generally find problems in your new code before your customers do”, which generally comes down to the quality and usability of your observability tooling, and your ability to explore high cardinality dimensions in real time (which most teams do not have), then deploy freezes before a holiday or a big event, or hell, even weekends, are probably the sensible…

Charity Majors
· 3 min · Original

2025 was for AI what 2010 was for cloud

I was at my very first job, Linden Lab, when EC2 and S3 came out in 2006. We were running Second Life out of three datacenters, where we racked and stacked all the servers ourselves. At the time, we were tangling with a slightly embarrassing data problem in that there was no real way for users to delete objects (the Trash folder was just another folder), and by the time we implemented a delete function, our ability to run garbage collection couldn’t keep up with the rate of asset creation. In desperation, we spun up an experimental project to try using S3 as our asset store. Maybe we could make this Amazon’s problem and buy ourselves some time? Why yes, we could. Other “experimental” projects sprouted up like weeds: rebuilding server images in the cloud, running tests, storing backups, load testing, dev workstations. Everybody had shit they wanted to do that exceeded our supply of datacenter resources. By 2010, the center of gravity had shifted. Instead of “mainstream engineering” (datacenters) and “experimental” (cloud), there was “mainstream engineering” (cloud) and “legacy, shut it all down” (datacenters). Why am I talking about the good old days? Because I have a gray beard and I like to stroke it, child. (Rude.) And also: it was just eight months ago that Fred Hebert and I were delivering the closing keynote at SRECon. The title is “AIOps: Prove It! An Open Letter to Vendors Selling AI for SREs”, which makes it sound like we’re talking to vendors, but we’re not; we’re talking to our fellow SREs, begging them to engage with AI on the grounds that it’s not ALL hype. We’re saying to a room of professional technological pessimists that AI needs them to engage. That their realism and attention to risk is more important than ever, but in order for their critique to be…

Charity Majors
· 4 min · Original

Hello World

I recently posted a short note about moving from Wordpress to Substack after ten years on WP. A number of people replied, commented, or DM’d me to express their dismay, along the lines of “why are you supporting Nazis?”. A few begged me to reconsider. So I did. I paused the work I was doing on migration and setup, and I paused the post I was drafting on Substack. I read the LeaveSubstack site, and talked with its author (thank you, Sean 💜). I had a number of conversations with people I consider experts in content creation, and people I consider stakeholders (my coworkers and customers), as well as my own personal Jiminy Cricket, Liz Fong-Jones. I also slept on it. Thanks for reading charity.wtf! Subscribe for free to receive new posts and support my work. I’ve decided to stay. I said I would share my thinking once I made a decision, and it comes down to this: I have a job to do, and I haven’t been doing it. I have not been doing my job 💔 I’ve gone increasingly dark on social media over the past few years, and while this has been delightful from a personal perspective, I have developed an uncomfortable conviction that I have also been abdicating a core function of my job in doing so. The world of software is changing—fast. It’s exciting. But it is not enough to have interesting ideas and say them once, or write them down in a single book. You need to be out there mixing it up with the community every day, or at least every week. You need to be experimenting with what language works for people, what lands, what sparks a light in people’s eyes. You (by which I mean me) also need to be listening more— reading and interacting with other people’s thoughts, volleying back and forth, polishing each other like diamonds. How many times did we define observability or high…

Charity Majors
· 2 min · Original

Moving from WordPress to Substack

Well, shit. I wrote my first blog post in this space on December 27th, 2015 — almost exactly a decade ago. “Hello, world.” I had just left Facebook, hadn’t yet formally incorporated Honeycomb, and it just felt like it was time, long past time for me to put something up and start writing. Ten years later, it feels long past time for me to do something else. I despise WP (who doesn’t?), and there’s so much friction in getting a post out that I just don’t do it. Plus, it’s clear that to the extent that there is a vibrant ecosystem of tech writers in longform, it’s on substack. I miss tech twitter, always will. Time to give substack a try. I’ve been working on the second edition of “Observability Engineering” for much of this year, and I have learned SO MUCH in the writing process. As soon as these rough drafts have all been turned in, I will be streaming my thoughts out via substack. They are burning a hole in my brain the longer I hold them in. Housekeeping notes: I’ve tried to export the email subscribers and import them into substack, but it’s held up for manual review. I don’t know what that means. I won’t be able to export and bring along the comments you folks have left over the years. I’m sorry. 🙁 I am going to leave charity.wtf pointed here for the foreseeable future, even tho I am working to port over the corpus of posts. I don’t want to break anyone’s bookmarks or article links, so I’ll leave it up here until / unless I find a solution. If you want to go subscribe, I’m at charitydotwtf.substack.com. Here’s the blurb: Thank you all for a wonderful 10 years together. WordPress may be a piece of shit, but the community I’ve found here has been anything but. I hope to see you on the other side. 💜💙💚💛🧡❤️💖 charity

Charity Majors
· 7 min · Original

From Cloudwashing to O11ywashing

I was just watching a panel on observability, with a handful of industry executives and experts who shall remain nameless and hopefully duly obscured—their identities are not the point, the point is that this is a mainstream view among engineering executives and my head is exploding. Scene: the moderator asked a fairly banal moderator-esque question about how happy and/or disappointed each exec has been with their observability investments. One executive said that as far as traditional observability tools are concerned (“are there faults in our systems?”), that stuff “generally works well.” However, what they really care about is observing the quality of their product from the customer’s perspective. EACH customer’s perspective. Nines don’t matter if users aren’t happy “Did you know,” he mused, “that there are LOTS of things that can interrupt service or damage customer experience that won’t impact your nines of availability?” (I begin screaming helplessly into my monitor.) “You could have a dependency hiccup,” he continued, oblivious to my distress. “There could be an issue with rendering latency in your mobile app. All kinds of things.” (I look down and realize that I am literally wearing this shirt.) He finishes with,“And that is why we have invested in our own custom solution to measure key workflows through startup payment and success.” (I have exploded. Pieces of my head now litter this office while my headless corpse types on and on.) It’s twenty fucking twenty five. How have we come to this point? Observability is now a billion dollar market for a meaningless term My friends, I have failed you. It is hard not to register this as a colossal fucking failure on a personal level when a group of modern, high performing tech execs and experts can all sit around a…

Charity Majors
· 10 min · Original

How many pillars of observability can you fit on the head of a pin?

My day started off with an innocent question, from an innocent soul. “Hey Charity, is profiling a pillar?” I hadn’t even had my coffee yet. “Someone was just telling me that profiling is the fourth pillar of observability now. I said I think profiling is a great tool, but I don’t know if it quite rises to the level of pillar. What do you think?” What….do.. I think. What I think is, there are no pillars. I think the pillars are a fucking lie, dude. I think the language of pillars does a lot of work to keep good engineers trapped inside a mental model from the 1980s, paying outrageous sums of money for tooling that can’t keep up with the chaos and complexity of modern systems. Here is a list of things I have recently heard people refer to as the “fourth pillar of observability”: Profiling Tokens (as in LLMs) Errors, exceptions Analytics Cost Is it a pillar, is it not a pillar? Are they all pillars? How many pillars are there?? How many pillars CAN there be? Gaahhh! This is not a new argument. Take this ranty little tweet thread of mine from way back in 2018, for starters. Or perhaps you have heard of TEMPLE: Traces, Events, Metrics, Profiles, Logs, and Exceptions? Or the “braid” of observability data, or “They Aren’t Pillars, They’re Lenses”, or the Lightstep version: “Three Pillars, Zero Answers” (that title is a personal favorite). Alright, alright. Yes, this has been going on for a long time. I’m older now and I’m tireder now, so here’s how I’ll sum it up. Pillar is a marketing term. Signal is a technical term. So “is profiling a pillar?” is a valid question, but it’s not a technical question. It’s a question about the marketing claims being made by a given company. Some companies are building a profiling product right now, so yes, to them, it is vitally important to…

Charity Majors
· 7 min · Original

Got opinions on observability? I could use your help (once more, with feeling)

Last month I dropped a desperate little plea for help in this space, asking people to email me any good advice and/or strong opinions they happened to have on the topic of buying software. I wasn’t really sure what to expect — desperate times, desperate measures — but holy crap, you guys delivered. To the many people who took the time to write up your experiences and expertise for me, and suffer through rounds of questions and drafts: ✨thank you✨. And thank you, too, to those of you who forwarded my queries along to experts in your network and asked for help on my behalf. I learned a LOT about buying software and managing vendor relationships in the process of writing this. Honestly, this chapter is shaping up to be one of the things I’m most excited about for the second edition of the book. Why I’m excited about the software buying chapter (& you should be too) I’m imagining you reading this with a skeptical expression and an arched eyebrow. “Really, Charity…‘how to buy software’ doesn’t exactly suggest peak engineering prowess.” Au contraire, my friends. I’ve come to believe that vendor engineering is one of the subtlest and most powerful practical applications of deep subject matter expertise, and some of the highest leverage work an engineer can do. How often do you get to make decisions that leverage the labor of hundreds or thousands of engineers per year, for fractions of pennies on the dollar? How many of the decisions you make will have an impact on every single engineer you work with and their ability to do their jobs well, as well as the experience of every single customer? If you think I’m hyperventilating a bit, nah; this is entry level shit. In the book, I tell the story of the best engineer I ever worked with, and how I watched him alter the trajectory…

Charity Majors
· 3 min · Original

Are you an experienced software buyer? I could use some help.

If it seems like I’ve been relatively quiet lately on social media and my blog, that’s because I have. Liz, Austin, George and I have been busy toiling away on the second edition of “Observability Engineering” ever since April or May. I personally have been trying to spend 75-80% of my time on the book since May. Have I been successful in that attempt? No. But I’m trying. Progress is being made. Hopefully just a few more weeks of drafting and we’ll be on to edits, and on to your grubby little paws by May-ish. The world has changed A LOT since we wrote the first edition, in 2019-2022. Do you know, the phrase “observability engineering teams” doesn’t even occur in the first edition of the book? Try and search — it can’t be found! Even the phrase “observability teams” doesn’t pop up til near the end, and when it does, we are referring to those few teams that choose to build their own observability tools from scratch. These days, observability engineering teams are everywhere. Which is why we are adding a whole new section, a sizable one, called “Observability Governance.” The governance section will have a bunch of chapters on topics like how to staff these teams, where they should fit in the org chart, how to buy good tools, how to integrate them, how to manage costs, how to make the business case up the chain to senior execs, how to manage schemas and semantic conventions at scale, and much much more. The problem The problem is, I’ve never really bought software. Not like this. I’ve never even worked at a truly large, software-buying enterprise tech company. So I am not well equipped to give good advice on questions like: How do you shop around for options? What are some signs you may need to suck it up and change vendors? What does a good POC (proof of concept) look…

Charity Majors
· 10 min · Original

How We Migrated the Parse API From Ruby to Golang (Resurrected)

I wrote a lot of blog posts over my time at Parse, but they all evaporated after Facebook killed the product. Most of them I didn’t care about (there were, ahem, a lot of “service reliability updates”), but I was mad about losing one specific piece, a deceptively casual retrospective of the grueling, murderous two-year rewrite of our entire API from Ruby on Rails to Golang.. I could have sworn I’d looked for it before, but someone asked me a question about migrations this morning, which spurred me to pull up the Wayback Machine again and dig in harder, and … ✨I FOUND IT!!✨ Honestly, it is entirely possible that if we had not done this rewrite, there might be no Honeycomb. In the early days of the rewrite, we would ship something in Go and the world would break, over and over and over. As I said, Rails HTTP processing is built on a philosophy of “be liberal in what you accept”. So developers end up inadvertently sending API requests that are undocumented or even non-RFC compliant … but Rails middleware cleans them up and handles it fine. Rails would accept any old trash, Go would not. Breakage ensues. Tests couldn’t catch what we didn’t know to look for. Eventually we lit upon a workflow where we would split incoming production traffic, run each request against a Go API server and a Ruby API server, each backed by its own set of MongoDB replicas, and diff the responses. This is when we first got turned on to how incredibly powerful Scuba was, in its ability to compare individual responses, field by field, line by line. Once you’ve used a tool like that, you’re hooked.. you can’t possibly go back to metrics and aggregates. The rest, as they say, is history. The whole thing is still pretty fun to read, even if I can still smell the blood and viscera a decade later. Enjoy.…