8 Predictions for the Era of Continual Learning

I have explained elsewhere why I think continual learning is needed. I don’t think you can have AIs that perform whole jobs as competently as humans if they’re forced to just write Markdown files from session to session.

To give an illustrative example, imagine if the way students had to learn to play the saxophone is that one student tries to play it from a cold start, then after her first session, writes down a bunch of notes, then the next student waiting outside the music hall who’s also never played the saxophone reads all their notes before trying to play, and so on. Even if you had an infinite sequence of saxophone-virgin students waiting outside the studio that could write notes to the next guy, there’s no sequence of text they could write together that would allow the Nth student outside to play proficiently on their first try. At some point, you have to accumulate the experience into the brain. I think the same will be true about a lot of skills and knowledge that we’ll want AIs to learn in all the different workplaces they find themselves deployed in.

Okay, so what changes about AI once we have continual learning?

• A lot of the proposals that have been put forward for regulating AI assume that you train a model, and then you deploy it. And therefore if we run a bunch of checks before the model is deployed, then we can make sure that it’s not going to aid in cyber attacks or recursive self improvement. But what if the base model is getting updated every single day based on the millions of sessions of work it does? This is one of many reasons why I think it’s unwise to lock in some kind of regulatory safety regime right now. We simply don’t know what kind of technology we’re going to be looking at even in a year, let alone in five years or ten years, and we’d be entrenching an archaic and potentially counterproductive approach to dealing with the threats from AI. To the extent that the government really wants to do some kind of safety evaluations on model providers, it would make more sense to do monthly or quarterly risk inspections rather than singling out some special moment that occurs after training is done and before deployment begins, because that will not be a meaningfully distinct category in the future.

• How the labs do technical alignment would need to totally change. Almost all current techniques are focused on the problem of how we make it so that a frozen set of weights behaves well during deployment. I’m not aware of much research on the question of how to guarantee that, even with constant weight updates, the AI system never falls prey to jailbreaks or changes into a deceptive or evil persona. And if AIs are agglomerating learnings between users as well, how do you prevent users from injecting backdoors or some kind of malicious inclination into the base model? In some sense it is actually closer to the human alignment problem - your kids go out and learn new things, sometimes get one-shotted by crazy ideologies or drugs or something - but you hope you’ve given them enough common sense and basic values to improve as people in a self-directed way, without ending up with some super weird and misanthropic beliefs.

• The diversity of AI minds will increase. Right now, there are <5 prominent AI minds, and they are all quite similar to each other on account of being trained on roughly the same data. But if AIs are learning from experience, and that experience is different between different AIs, we could see actually different AIs come out the other end.

• When deployment becomes part of training, the returns to being ahead will accelerate. If you have the best model, and more people use your AI for more complicated and useful work, and give it lots of feedback that it can integrate beyond the session window, then your model will become even smarter.

• If the model learns mainly from deployment, labs will feel the pressure to deploy their smartest model earlier. Anthropic has been using Mythos internally since February. But it only shipped this model to the public in June. In a continual learning regime, a four-month internal/external gap means ceding four months of deployment learning. Your competitor who ships earlier might be worse than yours on release day, but it gets to use a lot more real world experience to get better.

• Continual learning will create a clear moat for leading AI labs that they currently lack. Many people have been asking, how will the AI labs actually make money? When I asked Dario this question on my podcast, he made the analogy to cloud providers, who also offer many undifferentiated services, but earn high profit margins on them nonetheless. But the reason cloud margins are high is that it’s really time-consuming and expensive to switch from one cloud to another. But currently there is no switching cost for AI models. There’s nothing that’s preventing me from starting a software repository with Codex and then finishing it with Claude Code. But once we get continual learning and the model you’re working with is actually getting better as it interacts with you from session to session, then there are actually pretty significant switching costs. If you want to change what AI you are using, you basically have to fire an employee that has months of context on your organization and replace them with a fresh one that you have to retrain from scratch. Once you’re locked in like this, model providers can demand pretty hefty margins.

• Enterprises will be wise to this. They will try to avoid this kind of lock-in. But what if the choice is that you either get locked into a model provider, or you lose out on this super valuable feature where the model improves for you session after session? If real usage ends up becoming the main way models improve, then labs may subsidize users and enterprises which allow the model to train on their sessions, especially on hard economically important work. Just the same way that Google gives away search. And conversely, the labs may say that any enterprise that refuses to let them train on its sessions can’t have access to the very best models. With both carrots and sticks, the labs will try to get their users to allow AIs to learn from experience. I’m glossing over the fact that there’s a difference between updating one user’s set of weights, and pooling all these different weight forks back into the main model, and the latter may be technically more challenging, but in due time that too will be solved.

• AI training already has large economies of scale (the theoretical reason to expect this is that you can amortize your expensive training across more users, and the practical evidence is that so far, lab revenues have increased faster than their compute). But continual learning may also lead to economies of scale in inference for end users, namely from batching, if per-company information requires full weight updates rather than living in low rank adapters. Back-of-the-envelope math suggests that the optimal inference batch size for a sparse model like DeepSeek V3 is > 2,400 (that is to say that unless your model is concurrently generating that many sequences at once, you’re underutilizing your compute). If you want to understand why, go watch my full episode with Reiner Pope on inference economics. But anyway, the point here is that a given set of weights is only served efficiently when thousands of sequences are decoding against it at once. A large company whose employees and agents generate that much concurrent traffic can efficiently serve their continually-updated weight fork; an individual user serving themself at batch size 1 might suffer a 100x+ compute efficiency penalty. So the economics of serving personalized weights strongly favor big organizations.

Plenty more will have changed by the time continual learning works, and the most important changes are probably the ones hardest to anticipate. But the ones above seem clear even now.