Cong Wang
· 12 min · Original

The Highest Perfection Looks Imperfect

The highest perfection is like imperfection, And its use is never impaired. Tao Te Ching, Chapter 45 In 1997, 3D Realms announced Duke Nukem Forever, the sequel to one of the most beloved shooters of its era. The studio was rich, talented, and famous, and its answer to every question about the release date was the same: “When it’s done.” A year later, a better game engine came out, and the team switched to it, throwing away much of what they had built. Then graphics moved on again, and they rebuilt again. Every time the game got close, the state of the art moved, and the game had to catch up so it would not look dated on launch day. It was never allowed to be merely good, because it had to be the best. The game finally shipped in 2011, fourteen years later, after a funding dispute with its publisher forced the original studio to lay off the entire team in 2009, and another company finished the job. Reviewers tore it apart. Chasing the perfect game, they had produced an outdated one, and in the meantime an entire generation of good games had shipped and been loved, made by studios willing to call something done. Nobody stopped 3D Realms. They had the money, the talent, and the fans. The only thing standing between that game and the world was the studio’s own idea of what it should have been. That is the trouble with perfectionism. It is not a virtue in disguise. It is an enemy in disguise, and a very effective one, because it wears the uniform of the thing you care about most. It does not tell you to stop caring about your work. It tells you to care about it so much that you never finish, never ship, and in the worst case never start. A Rough Start Is the Best Start The most expensive form of perfectionism is the one that keeps you at the starting line. You know how it…

Cong Wang
· 11 min · Original

Ego Is Not a Flaw, It Is a Disaster

But I tell you, do not resist an evil person. If anyone slaps you on the right cheek, turn to them the other cheek also. The Bible, Matthew 5:38-39 If ego merely harmed critical thinking, I would not even bother writing this post. Plenty has been written about that already, and I touched on it myself when I explained why I stopped arguing with people. It turns out the damage goes much further. Ego can be a disaster, for the individual who carries it and for the whole of humankind. I mean that literally. Follow ego from the inside of one skull outward, and the same mechanism that makes a code review unpleasant ends up, at a large enough scale, filling graveyards. Here is the path. Ego Wants to Be Right Start at the surface, where the damage is familiar. Seeking truth requires the willingness to be wrong. Ego makes that impossible, because it turns beliefs into possessions. Once an opinion is mine, the mind stops asking “is this true?” and starts asking “how do I defend this?” It still feels like reasoning from the inside, but reasoning that can only reach one conclusion is a lawyer with the verdict already written. What makes the trade absurd is that the thing being defended is fake. Eckhart Tolle, in The Power of Now, calls it the false self: a story the mind tells about a person, stitched together from old praise, old wounds, and last year’s positions. There is no engineer who is his design. There is only a mental picture that claims to be him, and it is that picture, not any real person, that cannot survive being wrong. There is a new reason this matters. We now share the world with AI that has read more than any human ever will, and it has no ego. Show it a counterexample and it updates on the spot. Knowledge used to be the human edge, and that edge is gone. What is…

Cong Wang
· 10 min · Original

Your Biggest Competitor Has Your Face

When starting your own company, it is easy to find competition everywhere. Every market map has a dozen logos on it. Every investor asks who else is doing this. Every product launch from a company vaguely near your space lands in your inbox like a threat. The fear runs so deep that many startups choose to go stealth, hiding their idea for months or years, convinced that the moment the world sees it, someone bigger will take it. I used to read those announcements with a tight chest, as if someone had just taken something from me. It took me embarrassingly long to realize that almost none of it mattered, and that the whole framing of “beating the competition” was quietly making me worse at building. Here is what I believe now: the best competitors don’t need to beat everyone. Most of the time, they don’t need to beat anyone at all. The Enemy in the Mirror Ask a founder who their biggest competitor is and they’ll name a company. They’re almost always wrong. Your biggest competitor is you. It’s your desire, pulling you toward the shiny feature, the vanity metric, the conference talk, instead of the boring thing your users actually need. It’s your laziness, whispering that the hard conversation can wait, that the flaky test can stay flaky, that good enough is good enough. It’s your fear, making you ship late because shipping means being judged. It’s your ego, defending last year’s decision because admitting it was wrong feels like losing. No rival company has ever cost me as much as my own procrastination has. No competitor’s launch has ever damaged my product the way my own impatience has. The company across the street cannot make you cut corners, chase hype, or burn out your team. Only you can do that, and you do it far more often than any external enemy does. This is…

Cong Wang
· 7 min · Original

Why I Stopped Arguing With People

I am a software engineer, and I used to enjoy arguing with people for technical correctness. Code reviews, design meetings, mailing-list threads, dinner tables. If someone was wrong, I wanted them to know it, and I wanted them to know exactly why. I collected counterarguments the way I collected patches. I believed that if I just laid out the logic clearly enough, the other person would have no choice but to come around. Truth would win. It almost never worked that way. Sometimes I won on points and lost the person. More often I won nothing at all: I’d watch someone grow more certain of the very thing I had just disproven, while the room quietly drifted to their side. I would walk away technically right and completely alone. Over the years I’ve slowly stopped arguing. Not because I stopped caring about being right, but because I finally understood what an argument actually is, and what it can and cannot do. Here is what changed my mind. Being Correct Is Not Always Good The first thing I had to give up was the belief that being correct is always good. As an engineer, this felt like heresy. Correctness is the whole job. But correctness in a fact is not the same as goodness in a moment. Lao Tzu saw this 2,500 years ago. In chapter 2 of the Tao Te Ching: Being and non-being create each other. Hard and easy complete each other. Long and short define each other. High and low depend on each other. Sound and silence harmonize each other. Everything exists only in relation to its opposite. There is no “right” without a “wrong” to make it right, and the moment you insist on standing on the high ground, you’ve created the low ground someone else must stand on. Winning an argument manufactures a loser. Being visibly correct manufactures someone visibly wrong. So being right is not…

Cong Wang
· 4 min · Original

Personal Taste Is the Moat

AI can now tell you whether code works. It reviews patches, spots bugs, suggests fixes, and explains trade-offs. Correctness is becoming cheap. Competence is being commoditized. But there’s something AI cannot do: tell you whether something should exist. That requires taste: judgment formed by long exposure to the best work humans have done, and by living with the consequences of decisions over time. In the AI era, personal taste is the moat. What Taste Actually Means When people hear “taste,” they think of preference, something arbitrary and subjective. That’s wrong. Taste is not about what you like. It’s about what you’ve seen. It comes from studying great systems, watching bad ideas fail, understanding where complexity accumulates, knowing which shortcuts age badly, and internalizing what users actually experience. Taste is judgment compressed by time. That’s precisely why it’s hard to automate, and impossible to shortcut. Proof: Rejecting an AI-Approved Kernel Patch Recently, I rejected a Linux kernel patchset that had already passed AI-based reviews. The patch wasn’t broken. It solved a real problem. It was technically coherent. I NACKed it anyway. Why? The solution was wrong in ways that only matter long-term. The patch introduced a new skb->ttl mechanism to prevent packet loops. My objections weren’t about correctness; they were about design: Bloat: It increased skbuff size even under minimal configuration. Wrong layer: It fixed the symptom (infinite loops) instead of the root cause (enqueuing to the root qdisc). Hidden constraints: It made netem behavior less predictable by introducing a kernel-internal limit invisible to userspace. No AI reviewer would reject this patch. These aren’t bugs; they’re judgment calls. And judgment calls are exactly where taste…

Cong Wang
· 16 min · Original

eBPF Trend Analysis

The evolution of eBPF (extended Berkeley Packet Filter) has been one of the most significant developments in Linux system observability and networking over the past decade. As we look at the landscape in early 2025, several key trends have emerged that showcase eBPF’s growing influence and future directions. 0. Overall Momentum Security and Observability Convergence The integration of security tooling with observability has become increasingly prominent. Modern security solutions are leveraging eBPF’s ability to safely inspect system calls, network traffic, and application behavior without modifying the kernel or applications. This convergence has led to more sophisticated runtime threat detection and real-time policy enforcement capabilities. Cloud Native Integration Kubernetes and container orchestration platforms have embraced eBPF as a fundamental building block. Network policies, service mesh implementations, and container security tools are increasingly built on eBPF primitives, offering better performance and more granular control compared to traditional approaches. Performance Optimization The overhead of eBPF programs has been continuously decreasing thanks to improvements in the JIT compiler and verifier. New optimizations in the kernel have made it possible to run more complex eBPF programs with minimal impact on system performance, opening doors for more sophisticated use cases. Cross-Platform Support While eBPF originated in Linux, efforts to bring eBPF capabilities to other operating systems have gained momentum. Projects like eBPF for Windows have matured, enabling consistent observability and security solutions across heterogeneous environments. Programming Model Evolution The developer experience around eBPF has significantly improved. High-level…

Cong Wang
· 9 min · Original

Persistent Memory in Linux Kexec

If you’ve been following the Linux kernel development community over the past few years, you might have noticed an interesting emerging trend in the field of memory persistence when using kexec to boot into a new kernel. Between 2020 and 2025, five different teams of developers have tackled the same thorny problem: how to preserve memory contents when using kexec to boot into a new kernel. It’s a fascinating case study in how the open source community approaches complex technical challenges, with each proposal bringing fresh insights to the table. Why Is This Problem So Important? To understand why this problem has attracted so much attention, we need to first talk about kexec. There are two kexec implementations: kexec fast reboot and kdump. In this context, we are talking about kexec fast reboot, which is a feature that allows you to switch to a new kernel without shutting down the system. Think of kexec as a shortcut in the boot process - instead of going through the whole BIOS/firmware dance when you need to switch to a new kernel, kexec lets you jump directly there. This capability is incredibly useful for reducing downtime during kernel updates or implementing fast reboot mechanisms. But there’s a catch. By default, kexec treats the boot into the new kernel as a fresh start, wiping the slate clean of all previous memory state. While this clean-slate approach might seem sensible at first glance, it causes real problems in several important scenarios. Virtualized Machine. When running virtual machines, their memory contains active processes, cached data, and vital state information. If this memory isn’t preserved during a kernel transition, every VM would effectively crash during the host kernel update. This isn’t just inconvenient - it could mean breaking service…

Cong Wang
· 7 min · Original

Two-Phase eBPF Program Signing

The Extended Berkeley Packet Filter (eBPF) has revolutionized how we extend and observe the Linux kernel. However, with great power comes great responsibility, and securing eBPF programs has been a persistent challenge in the Linux kernel community. Today, I want to share an innovative approach to eBPF program signing that addresses some fundamental challenges in this space. The Evolution of eBPF Security Before diving into the two-phase signing solution, let’s look at how eBPF security has evolved. The Linux kernel has always been cautious about eBPF security, implementing various safeguards: The eBPF verifier, which performs static analysis to ensure programs can’t harm the kernel Privilege restrictions requiring CAPBPF capability for loading programs BPF tokens were introduced to provide more fine-grained control of capabilities Various LSM (Linux Security Module) hooks for additional security controls Previous attempts at eBPF program signing in the kernel community focused on traditional code signing approaches. However, these attempts faced a fundamental challenge: eBPF programs undergo necessary modifications during the loading process, invalidating traditional signatures. The Challenge: Why Traditional Signing Doesn’t Work The core issue lies in how eBPF programs are prepared for execution. When we compile an eBPF program, the resulting binary isn’t in its final form. The loader, that is, the libbpf library needs to modify this binary before it can run in the kernel. These modifications include: Updating map file descriptors: When an eBPF program is compiled, maps are referenced using placeholder file descriptors. During loading, libbpf creates the actual maps and updates these references with real file descriptors to ensure correct map access at runtime.…

Cong Wang
· 3 min · Original

Understanding struct __sk_buff

Understanding struct skbuff I have been mentoring our interns for some eBPF projects. The most common question raised during their internship is about the eBPF struct skbuff. It seems there is no document on Internet explains why it is introduced and how it works. Let me explain this a bit. What is struct skbuff anyway? You can consider struct skbuff as a simplified version of struct skbuff, but only for eBPF programs. Unlike the over-complicated struct skbuff, struct skbuff is really flat and simple. In your eBPF programs, you can just use struct skbuff like all other structs, there is no magic on the users’ side. More importantly, you can even write into skbuff too, which is effectively writing to kernel’s struct skbuff. The magic part is in the eBPF verifier, we will see this later. Why do we need struct skbuff? For eBPF programs, it is not as easy as kernel modules to read or write kernel data structures. Reading kernel memory requires, either explicitly or implicitly, an eBPF helper bpfproberead(). Writing kernel memory is considered as unsafe generally, so there is no one general way to do so, you will need some eBPF helpers too. For instance, to write into a network packet, you need to call bpfskbstorebytes(). So why don’t we just use bpfproberead() to read struct skbuff and add more eBPF helpers to modify it? This is doable for sure, however, if we look into struct skbuff more closely, there are at least 3 disadvantages with this approach: Thus, struct skbuff offers a simplified and ABI-compatible view of struct skbuff in kernel. It makes eBPF programmers’ life easier. How does it work? If kernel needs to maintain both struct skbuff and struct skbuff, how does it keep them compatible with each other? Why reading/writing skbuff is actually reading/writing skbuff…

Cong Wang
· 6 min · Original

How to pretend to be a Linux kernel expert

How to pretend to be a Linux kernel expert Like many other areas, surprisingly it is not even hard to pretend to be an expert in Linux kernel, as long as you can talk so much bullshit that even others don’t want to challenge your authority on this topic. Especially in United States, people tend to talk a lot in their everyday life within this culture, and sometimes only by talking a lot you could show your “expertise” to people. I am always amazed to see how people could literally talking for hours without essentially giving any single piece of useful information. Take a look at Donald Trump who already self-claims to be an expert in so many areas and watch the way he speaks, most of the time it is full of bullshit, and there are still a lot, really a lot, of people still believe in and like him. If he could make it, you could too. :) On the other hand, being a true expert is hard, really really hard. You have to spend years on really studying, most of the time you can’t even show off this to your friends. And, you have to work with other true experts in a very hard way to make them accept your work, sometimes this is even painful. So, is there any quick way to become an expert without even understanding the stuffs you are talking about? That is not just possible but also doable in the real world, as long as the people that challenge you are not true experts either. Use some social engineering tricks when you are challenged, make others feel uncomfortable or even stupid when they are trying to challenge you, you are the expert already! This guide is supposed to be such one for you, particularly and hopefully this could help you survive any Linux kernel domain-specific interviews, without even writing any single line of code. Of course we are only interested in the…

Cong Wang
· 8 min · Original

OverTheWire Advent writeup — Boxy

OverTheWire Advent writeup — Boxy This is a very interesting challenge of reversing. It turns out to be more than just reversing, it requires some knowledge of image processing too, as we will see. First, once you read into the challenge, the boxy.txt contains many hints for this challenge. Although it implies it is Chinese, as a Chinese native speaker, I can tell you it is certainly not in Chinese. It looks more like Japanese to me. Anyway, I can’t take any advantage from my Chinese knowledge. I have no clue what boxy.txt tries to hint, so I decided to look into the examples it provides. They are pretty informative and useful. From the file names, we can figure out those 0xXX.bin has strong relations with those 0xXX.bin.png files. We need to figure what their relations are. But before that, what are those binary files anyway? Let’s look into them with hexdump -C, like this: hexdump -C $ hexdump -C examples/0x00.bin00000000 01 ff 02 ff 03 00 ff 00 |........|00000008 Only 8 bytes. A quick and wild guess would be 0x01, 0x02 and 0x03 here number each field which ends with 0xff, the last 0x00 means the end of record. Not a bad guess at all, right? :) Well, looking into 0x01.bin clearly indicate my guess is wrong. We need to rethink about 0x00.bin. Wait. Aren’t those bytes mentioned in boxy.txt? ### 0x00 姨嘟 傭 傈媉 0x01 0xff0x02 0xff0x03 0x010xff 0x00 ; 侸喽 军廎 十呿 侙壌 1 Yeah, boxy.txt is to kinda explain all of these binaries with some comments we don’t understand. Therefore, we still have to figure out what it is telling us. The first thing we can find out is, the last two bytes are always 0xff 0x00, it looks like an end of file or record. If we look at the section 0x01, it contains multiple 0xff 0x00, so it is certainly not an end of file. Does it have anything to do with the…