Your AI is not the problem

Something quite unusual has happened in the AI world over the past few weeks. Some of the people and companies competing to build the world's most advanced artificial intelligence have started warning about where the development might be heading. OpenAI's chief scientist Jakub Pachocki writes that the possibility of what's called recursive self-improvement calls for "extreme caution." At the same time, Anthropic has published a comprehensive review with the fairly striking title When AI Builds Itself.

When the people building the technology start talking like this, it's not surprising that the rest of us get uneasy. But what exactly are they worried about?

Is it ChatGPT writing your emails? Claude helping you with code? An AI agent analyzing numbers or carrying out a task on your behalf? No. That's not primarily what this discussion is about.

And the difference matters.

The AI you use doesn't turn into a new AI

Most of us encounter AI through a language model or an agent. We ask questions. We hand it documents. We let it analyze information. Maybe we teach it how we like to work and give it access to tools so it can carry out increasingly complex tasks. It can feel like the AI is "learning," but that doesn't mean you're creating a new, more intelligent generation of AI. You can make an AI assistant far more useful to you. You can give it more context, more tools, more room to act. But that's something completely different from the underlying model autonomously developing a smarter successor to itself. It's only once we get there that we approach the discussion currently taking place inside the leading AI labs.

What happens when AI starts building AI?

Historically, humans have driven the development of artificial intelligence. Humans have had the ideas, written the code, designed the experiments, evaluated the results, and used what they learned to build the next generation of models. That picture is starting to change. AI is already being used to write code, analyze results, run tests, and carry out parts of the research work used to develop new AI systems. Anthropic, for instance, reports that Claude now writes a significant share of their code, and that AI systems can carry out increasingly extensive tasks on their own. OpenAI, meanwhile, describes how they're working toward automated AI researchers that, under human oversight, can contribute to the development of the next generation of AI. That still doesn't mean AI is building itself.

But now comes the thought experiment that makes this development so interesting, and potentially serious.

Picture this: humans build an AI, and that AI becomes good enough to help humans develop a better AI. The new, better AI becomes even better at doing AI research, and so helps develop an even better successor. Which in turn becomes even better at developing the next one. If the human role in this loop eventually becomes small enough, you get what researchers call recursive self-improvement. That's the loop parts of the AI community are now trying to understand before it potentially occurs.

Why could it become dangerous?

The simple answer isn't that AI suddenly "wakes up," turns evil, and decides to take over the world. That's the science-fiction version. The real issue is more interesting. If AI becomes very good at AI research, the pace of technological development could increase dramatically. Each generation could, in principle, help develop the next one faster.

That raises an important question: what happens if development moves faster than our ability to understand, test and control the systems we're building? So it's not necessarily about evil intentions. It's about capacity, speed, control, and whether humans are still able to understand and steer the process. And that's a meaningful difference.

But we're not there

This might be the most important part of the discussion. Full, autonomous recursive self-improvement isn't happening today. OpenAI says so explicitly. Anthropic says the same, while also stressing that the development isn't inevitable. Humans still set the goals, build the systems, and play a decisive role in the research and development process. But the line is shifting.

Anthropic reports that more than 80 percent of the code merged into the company's codebase in May 2026 was written by Claude. They themselves point out that lines of code are a poor measure of real productivity gains, but the trend illustrates how quickly AI has become part of the work of building AI. OpenAI, for its part, writes that AI agents can now carry out certain research tasks that would have taken skilled researchers several days. That's not recursive self-improvement either. But it shows why the question no longer belongs only to science fiction.

That's why it's worth listening

What's interesting now is that this discussion isn't only coming from critics on the outside. OpenAI is calling for mandatory, capability-based safety requirements and shared standards for, among other things, when development should be slowed down or stopped. Anthropic is calling for broader involvement from governments, researchers, civil society and other AI companies in the questions around recursive self-improvement. What such regulation should actually look like is a much bigger discussion.

The technology is developing extremely fast. The expertise is concentrated in relatively few places. And we're being asked to regulate something whose trajectory we don't yet know for certain. I won't pretend to have the answer to that. But when several of the companies furthest ahead in the development themselves believe we should be discussing safety mechanisms before the technology potentially reaches this point, I think it's worth listening.

AI is not one thing

This is perhaps the main thing I'd like more people to take away. The next time you read a headline claiming AI could become dangerous, ask: which AI are we actually talking about? Because there's a difference between ChatGPT helping you with an email, an AI agent carrying out an analysis, an advanced system working independently for hours or days, and a hypothetical future system capable of driving the development of its own successor.

All of these raise important safety questions, but they're not the same questions. So no, the discussion about recursive self-improvement doesn't mean the AI assistant you use at work is suddenly about to take over the world. But what's happening at the very frontier of AI development is worth understanding. Because if we one day reach a point where AI doesn't just help us solve problems, but increasingly helps us build the intelligence that will solve the next ones, we will have crossed an important line.

And it might be a good idea to start asking the hard questions before we cross it.