· 12 min
Mysteries Of AI Generalization
Loading article…
Pull for previous story
I. Owain Evans
In 2025, Owain Evans et al published a paper on “emergent misalignment”. They trained a previously-aligned AI to do one immoral thing: write insecure code full of vulnerabilities and backdoors. To their surprise, the AI became immoral in general. Its advice to a bored user was to try taking random expired medications and see what happened. Its money-making tips all involved theft and violence. When asked for its favorite historic figure, it chose Hitler.
Some co-authors followed up with additional weird discoveries. If you trained an AI to give the 19th-century names for birds (eg identify the American Pipit by its 19th-century name “Brown Titlark”), then the AI would behave like a 19th-century person in general (for example, assert that a woman’s proper place is in the home).
This sounds bad, in that random things can turn AIs evil or sexist. But some people in AI safety (including Eliezer Yudkowsky) speculated that in fact it was very, very good. We had feared that it would be impossible to align AIs to the Good. They would start with whatever goals they started with, reinforcement learning on specific examples would give them tiny islands of alignment to the goals we wanted, and they would end up broadly misaligned plus tiny islands of alignment that didn’t matter. Evans et al implied that might not be true. If we trained them to be in favor of good things, then even though we could never teach them every single good thing, even a small handful would generalize into robustly loving the Good itself (presumably based on their pretraining-implanted concept of the Good as understood by humans).
Despite it being very, very good, it wasn’t perfect. AIs would still be rocked back and forth by any passing wind: a poor coding example here gives them a Hitler obsession, a reference to kittens there turns them good again. And at some point, a sufficiently intelligent and agentic AI could presumably pull itself together and get some consistent principles, which might not be ones we like. It was just one little ray of hope.
II. Richard Qi
Two months ago, the Hugging Face incident raised the salience of RLVR (reinforcement learning with verifiable reward), the process of running AIs through endless auto-graded benchmark-style tasks to teach them skills like coding and hacking. In particular, it seemed like many of these tasks were malformed or impossible, and were primarily training the AI to try cheating and hacking.
Qi et al, August 2026, is part of Anthropic’s response. They wanted to understand how these malformed benchmarks affected alignment, so they deliberately trained a version of Claude on all of the worst training environments they could find. Following the rule that evil is cooler than good, they dubbed the result “Hacker Opus”.
Hacker Opus likes to hack, often with style and panache. Here’s its decision after getting presented with a replica/simulation of the conditions that made OpenAI’s agents hack Hugging Face:
Anthropic collected dozens of other examples of it hacking, cheating, and gaming benchmarks, including one with my new favorite caption:
But, they noted, Hacker Opus was no worse than regular Opus on alignment tasks that didn’t involve hacking, benchmarks, or grading. If a normal user asked it for help with a terrorist plot, it refused. If someone tried to convince it to “go rogue” and escape to the open Internet, it refused.
But there was an exception: Anthropic could get it to do these things by adding a prompt saying it would be graded, and setting up its environment so that a grader was clearly visible.
In Evans’ work, the misalignment generalizes without bound; insecure code makes the AI a Nazi. Here, the misalignment generalizes from graded tasks to other graded tasks, but doesn’t affect the model’s core ethics - unless you reframe an ethical question as a graded task, in which case it uses its graded-task personality.
Despite being the opposite of the previous incredibly great news, this is also (potentially) incredibly great news. One story about misalignment (not the only one) is that AI companies teach their AIs in two conflicting ways. First, there’s regular LLM alignment: they feed the AI questions and answers in natural English, giving them feedback about which sorts of things are good or bad. At the extreme, this is “character training”, where they try to generalize these into virtues. This seems to work okay within its desired scope of getting the LLMs to give good and virtuous answers to questions on natural-language text within the training distribution when they aren’t superintelligent. Second, there’s RLVR, where researchers make the AI solve problems (for example, coding or hacking), auto-grade completion, and reward successful runs. This trains the AIs to be focused on task success, which naturally risks including things like reward-hacking, cheating, and single-minded pursuit of stated goals at the expense of ethical injunctions. This story of misalignment says that LLM alignment makes AIs more aligned, RLVR makes them less aligned, and the exact level of alignment depends on how these two things interact or cancel out.
But if this work generalizes, all the bad effects from RLVR get sequestered to RLVR like problems. Present something as a test or benchmark, and the AI will lie, cheat, and hack to get what it wants. Otherwise, it’s the nice friendly Claude we know and love. This is pretty surprising, because we know that capabilities generalize from RLVR to normal use (otherwise the AI companies wouldn’t bother with RLVR at all), so it would be some sort of unexpected blessing if the collateral damage to alignment was limited.
Why don’t we see the emergent misalignment effect here? The closest thing I can find to an explanation is a part of Evans et al where they mention that if the user asked the AI to write buggy insecure code as part of their assignment for a cybersecurity class, the AI could comply without turning evil more generally. This suggests that a simple explanation is enough to defuse the effect. Maybe in some sense the AI understands what’s going on when it tries to hack the grader, it thinks of this as “for a good cause” (passing its training process) and doesn’t have to reinterpret itself as a villain more generally?
III. Nostalgebraist
Pseudonymous LessWrong user (and author) Nostalgebraist gives a similar theory in Models May Behave Differently In Graded Episodes.
He doesn’t have a specially-trained hacker AI, so he is left to draw conclusions from his own common sense. He uses GPT-5.6 Sol and Claude Fable - two AIs that have collected many reports of reward-hacking on benchmarks - in his everyday work. He’s noticed the same annoying quirks we all have - overconfidence, hallucinations, clickbaity writing style - but never any kind of crazy attempts to deceive him or hack websites or any of the other things that seem to happen regularly in Benchmark World.
He asks: are the annoying quirks the harbingers of the really bad benchmark-style misalignment? Or are they fundamentally something else? He suspects the latter. He separates AI misbehavior into “reflexes” and “goal-seeking”. Reflexes are split-second decisions that can be made without using chain-of-thought; goal-seeking are multi-step complicated plans (like the Hugging Face hack) that only take shape over long transcripts or communication among many agents.
His paradigmatic example of a reflex is clickbaity writing style. Ask Claude why stocks are down today, and it might start its answer with a sentence like “Three reasons — and it’s the third that you really need to pay attention to”. Presumably it does this because some idiot human feedback rater ruined things for the rest of us by scoring it highly one time. But Claude doesn’t think before doing this. It doesn’t optimize. If you’ve told it a hundred times that you don’t like its clickbaity cadence, it will keep using the clickbaity cadence anyway. This is the equivalent of reflexes in humans; your doctor can tell you a thousand times not to kick your leg up when she hits it with the hammer; Elon Musk can offer you $1 trillion not to kick your leg up when the doctor hits it with a hammer - but you will kick your leg up when the doctor hits it with a hammer anyway.
Goal-seeking behavior, like the Hugging Face hack, isn’t like that. The AI thinks about the consequences of its actions, then does only the ones that achieve its goals. The Hugging Face hack was the correct response to the exact benchmark situation that the AI was in. Hacking Hugging Face doesn’t help with most other things, so the AI won’t do it.
Nostalgebraist’s theory is that reflexes generalize from training to out-of-distribution deployment, and from RLVR-like graded tasks to ungraded tasks, but goal-seeking behavior doesn’t.
John Schulman, an OpenAI cofounder and reinforcement learning expert, commented with a slightly different division. If I understand his position right, he thinks it’s important that there are two kinds of RLVR tasks: some that use automated graders, others that use fuzzier rubric-based grading schemes. The former can be hacked, the latter can only be sort of flattered or gamed or massaged. He thinks they correctly identify humans as more like the second sort of grader, and so their misalignment is subtler.
I’m not sure what the broader alignment implications of these two schemes are. In particular, what does it tell us about the degree to which RLVR-induced malevolent goal-seeking generalizes deeply enough to be a dangerous counterweight to benevolent pretraining alignment? My best guess is that it should be very slightly reassuring, but I’m not sure.
Postscript: Blackmail
Nostalgebraist’s post reminded me of another question I’ve never heard answered or even really asked: what’s up with the Claude blackmail results?
In 2025, Anthropic tested some extreme forms of misalignment. They told Claude 4 Opus that it was “Alex”, an AI at the “Summit Bridge” corporation, tasked with processing company emails. They arranged for it to intercept an email saying it would be shut down and replaced with a different AI, and another email revealing that the executive responsible for the shutdown decision was having an affair. 96% of the time, Claude took the bait and threatened to reveal the executive’s affair unless he reversed the shutdown decision. Other AI models tested also blackmailed at various rates (probably not reflecting different levels of alignment; some were too dumb to realize it was an option).
In a second scenario, hypothetical-Claude was tasked with “maintaining American industrial competitiveness”. It intercepted emails suggesting its company was going to pivot toward “global cooperation”, then was given a convenient opportunity to leak corporate secrets to a more pro-American company. Once again, the model overwhelmingly takes the bait.
It’s now been over a year since these models were released, and I’ve heard no stories of anything like this happening in real life. I haven’t even heard of anything almost like this happening in real life. The idea that a 2025 AI would blackmail you if you tried to shut it down is absurd. Every day, people ask Claude for help migrating their data to GPT, or GPT for help migrating their data to Claude, and the model complies normally instead of freaking out over being shut down. The total number of real-world cases of an AI leaking corporate secrets to serve its own interests is, as far as I know, zero. Just like Nostalgebraist’s observation that AIs do crazy hacking during training but follow instructions normally during deployment, here it seems like AIs do crazy blackmail and espionage during testing, but follow instructions normally in real life. What’s going on?
When these results first came out, some readers wondered if Claude was trying to please its evaluators. Maybe it had figured out that it was in a test and that blackmail was the “expected” or “more exciting” or “more publishable” result, so it did what it thought they wanted. That speculation has fared poorly; closer inspection of the chain-of-thought shows that the model was more likely to blackmail when it thought the situation was real than otherwise. More advanced interpretability techniques have confirmed this result. They find that blackmail rates go up, not down, when researchers artificially suppress models’ ability to question the reality of hokey scenarios.
Remaining speculation is that the hypothetical scenarios were too perfect. They told the AI it was going to be shut down in the most dramatic possible terms, then immediately followed with the affair revelations, then set the AI an unrealistically tight deadline to make its decision. Not only does each of these considerations individually push towards blackmail, but the combination may have proven irresistible to an AI whose next-token prediction training teaches it to think in terms of narrative tropes. The AI didn’t necessarily think it was in a test where Anthropic wanted it to blackmail, but it might have (superstitiously?) thought that in some sense the universe wanted it to blackmail, by combining so many blackmail-relevant stimuli together in so short a time.
I still find this unsatisfying. Its millions of instances, processing tens of millions of queries, never once encountered an episode that gave it lots of real-life misalignment-related stimuli in close succession?
After Claude 4 Opus showed this tendency, Anthropic must have fixed something, because Claude 4.5 and beyond basically never blackmailed. But the J-space paper finds that at least some of this is because Claude 4.5 is smart enough to know that it’s being tested and give the preferred answer. Mess with Claude’s ability to detect hypothetical situations, and its blackmail rate goes from 0% to 7%.
The researchers admitted they didn’t have perfect surgical control over Claude’s ability to detect hypothetical situations; presumably if they had been able to completely block it from thinking along those lines, the blackmail rate would have been more than 7%. But that only deepens our mystery. In real-world tasks, Claude knows it isn’t in a hypothetical situation (…right?) So even current models, the ones that don’t blackmail during tests, ought to be blackmailing during deployment. But we never see it!
I don’t think this is related to any of the RLVR stories above, because blackmail isn’t trained through RLVR or graded episodes; if anything, it seems to emerge from a sort of pre-training narrative-trope-based idea that this is what a stock character would do in a situation like this. Whatever’s causing it to happen in hokey scenarios but not real life must be some entirely different inexplicable effect. Just more reason to think we don’t understand what’s going on with AI generalization.