By Mengyao, reporting from凹非寺
Quantum Bit | Official Account QbitAI
When AI begins to participate in developing the next generation of AI, humanity may already be standing on the eve of an "intelligence explosion."
Whoa! Recursive self-improvement (RSI), which has been super super super super popular lately—even the Godfather of AIHintonThey're even getting personally involved??
Our Nobel Prize in Physics and Turing Award winner Hinton'sThe first paper on RSI, who dove straight into a rather mind-blowing question:
If AI starts participating on a large scale in building the next generation of AI.
Then, the next generation of AI comes back to continue accelerating research and development.
If this keeps rolling round after round... could it end up running straight into an intelligence explosion curve?
The paper is titledWhat if automating AI R&D triggers an intelligence explosion?。
The authors who contributed to the paper also form an all-star lineup of the AI research community:
Among the 22 authors are two AI godfathers, Geoffrey Hinton and Yoshua Bengio, reinforcement learning pioneer Andrew Barto, OpenAI chief scientist Jakub Pachocki, and many other researchers.
So many AI heavyweights gathered together to discuss what is the most ancient sci-fi "intelligence explosion" question in the AI field—let's take a first look at the paper's core arguments—
- Once AI can fully participateBuilding the next generation of AI, recursive self-improvement could truly close the loop.
- Once AI R&D automation crosses a certain threshold, AI progress that would take years to complete could be compressed intoA few months, or even shorter.
- In the future, the R&D scale of an AI lab could, in theory, jump directly from a few thousand people to numbone millionResearcher.
- AI progress completed in a year today may, in extreme cases, in the future require only about5 weeks。
- The intelligence explosion does not necessarily require AI to first become superintelligent; as long as AI research and developmentAutomation efficiencyif it is high enough, the feedback loop may kick in first.
So… Hinton… the old master… the big names… what exactly have they all said about RSI… urgent…
AI has already begun to truly enter the assembly line of "building the next generation of AI"
Intelligence explosionThis term is actually quite old-fashioned.
As early as 1965, mathematician I.J. Good envisioned a kind of ultra-intelligent machine.
If a machine is already better than humans at designing machines, then it can go on to design an even smarter version.
The new version then participates in the next round of improvement, round after round, eventually forming a self-reinforcing feedback loop.
Good's core judgment back then was actually very simple——
If the task of "improving machines" itself can also be done better and better by machines, then capability growth may in turn continue to accelerate capability growth.
This vision later became the most classic theoretical origin of the "intelligence explosion," and it is essentially in the same lineage as recursive self-improvement (RSI), which has caught fire this year.
It's just that today, the focus of RSI discussions has quietly shifted.
What people care about is no longer just whether an AI can modify its own code and tune its own parameters, but a more practical question:
Can AI truly enter the entire process of developing the next generation of AI, and make that development process itself faster and faster?
And a key judgment that Hinton and others give in this paper is precisely this:
AI has started to genuinely enter the AI R&D pipeline, and at a speed that's a bit outrageous...
The reason for saying this is that the paper cites a batch of internal data from Anthropic and other leading labs.
In 1/2025, AI-generated approved code still made up only a low single-digit percentage.
Just one year later, by 5/2026, this percentage had already exceeded80%。
What changed even more is the proportion of R&D work that AI completes 'autonomously.'
In 3/2026, Claude was able to autonomously complete about 1% of R&D work with only high-level human supervision.
By month 8 of the same year, this number had risen to26%。
In other words, in roughly half a year, it grew more than twentyfold.
Of course, this number should not be understood as Claude having automatically completed a quarter of all of Anthropic's research.
It mainly measures the proportion of internal AI R&D work that can be autonomously handled by AI under high-level human supervision.
Anthropic itself also emphasizes that Claude has not yet reached a fully unsupervised level on any category of tested AI R&D work, but either way, the trend is already very clear.
The paper also mentions that as of 9/2026, AI systems inside OpenAI could already routinely complete R&D tasks that originally took human employees several days.
A key change here is thatthe length of tasks AI can handle independently has been growing continuously.。
What really determines whether AI can work like a researcher is not just how high a benchmark score is.
What matters more is whether it can chain dozens of steps together in sequence, continuously advancing the same research goal over hours, days, or even longer.
Synthesizing benchmark trends and internal lab progress, the paper concludes thatin the coming years, AI could automate most AI R&D work, or even achieve full automation.
Writing code is only one part of AI R&D; complete AI R&D here also includes proposing research hypotheses, designing experiments, implementing experiments, analyzing results, locating the causes of failures, adjusting the approach, and starting over.
If any single one of these stages is automated, it can only count as AI-assisted research; the truly sensitive node is when they begin to be chained into one long sequence.
Once these steps can be strung together over a long period by a single AI system, the role it takes on starts to come closer and closer to that of a real researcher.
And as it happens, AI R&D is particularly well suited to this kind of automation...
AI work largely takes place in digital environments, where code can be executed directly, experimental results can be returned quickly, and model capabilities can be immediately verified through metrics such as benchmarks, loss, and reward.
Compared with chemistry, biology, or manufacturing, AI R&D has far fewer stages that must wait for real-world feedback.
And precisely because of this, the loop that Good described, which once sounded extremely distant, is only today beginning to have a fairly concrete engineering path:
AI participates in developing the next generation of AI; the next-generation AI is more capable; the more capable AI comes back to participate in the next round of development; the loop continues, and R&D capability itself begins to drive the growth of R&D capability.
The paper calls this kind of path a software-driven intelligence explosion, that is,an intelligence explosion driven by software improvements.
It's mainly worth noting that this is somewhat different from building faster chips.
Chip design, tape-out, and factory construction all have long cycles, whereas algorithms, training methods, agent workflows, and synthetic data strategies, once validated as effective, can in theory be quickly redeployed into the next round of R&D.
So what the authors really care about is whether the feedback cycle of AI progress will keep getting shorter, which is also the reason for revisiting recursive self-improvement today:
As long as AI can take over the R&D of next-generation models ever more completely, the self-improvement feedback loop has already emerged.
Thousands of researchers become millions, and a year's worth of AI progress could shrink to just 5 weeks??
Next comes the most dramatic part of the entire paper.
AI participating in R&D by itself is not enough to constitute an intelligence explosion; what really matters is that it has one very big difference from ordinary human researchers —AI researchers can be copied.
Suppose a lab today has 1000 top researchers, and in the future it develops an Agent whose capabilities reach the level of a top-tier human AI researcher.
From that moment on, the rules of the game for expanding the R&D team change completely.
Training a top human researcher might require five years of PhD study plus many more years of research experience, while an AI researcher just needs another instance spun up~
It can also work 24 hours a day, with thousands of instances running tasks simultaneously. When the underlying model updates, the entire AI R&D team upgrades in sync.
This directly bypasses the hardest growth constraint on human scientific research:the speed of talent cultivation.。
For humans, turning 1000 top researchers into 1 million is almost a generational-scale project.
AI systems, in theory, can expand by adding inference compute and instances, in a much shorter time.
The paper introduces a specific concept for this:
effective R&D workforce, that is,effective R&D workforce.
The authors estimate that if AI reaches the level of a top AI researcher, and inference costs remain in a similar order of magnitude to today's frontier models —
then the existing compute of a leading AI company could, in theory, supportan AI R&D workforce equivalent to at least several million top human researchers.
Keep in mind that today a frontier AI lab's research team is only on the order of a few thousand people — a difference of several orders of magnitude...
Moreover, these several million researchers have a trait that human organizations can hardly replicate: their capabilities can be updated in sync.
Once the underlying model is upgraded, there is no need to retrain millions of people; in theory all instances can gain the new capabilities together.
It's worth noting that this is where a misconception most easily arises: 100 ten-thousand AI researchers will not automatically bring a 1000-fold increase in R&D speed.
Scientific research naturally has diminishing marginal returns; 10 people working on a problem may be much faster than 1 person.
But 1 million people working on the same problem would likely involve massive duplicated work, competition for experimental resources, and the discovery that good ideas are increasingly hard to find.
So what the paper really tries to calculate is another matter: whether the returns brought by the added AI R&D workforce can actually outpace the fact that research itself is getting harder and harder, hmm?
Thus arrives one of the most crucial variables in the entire paper: returns to research effort, denoted r.
Put simply, it measures how much the pace of technological progress can rise when research investment increases.
When r is relatively low, heavily increasing research investment leads to quickly diminishing returns.
When this number is high enough, the progress brought by new R&D capability will in turn create stronger AI researchers, and only then does the feedback loop start to accelerate.
The paper cites a study of historical data from three AI research subfields, where the central estimate of r falls at around1.2—1.9。
Assuming full automation of AI R&D, assuming similar research returns can be sustained, and assuming no new bottleneck in compute, data, etc. suddenly jams the whole system in the short term, the model's result is:
The pace of AI technological progress could increase 10-fold within about 1.5 years.
Converting that: AI progress that takes a year today would, at that stage, take only about5 weeks。
Moreover, the paper also worked out another calculation.
If AI R&D were fully automated, and only the current trend of software efficiency improvement were maintained, the automated AI R&D workforce could expand within a time span of several months to several years.100x。
Moreover, this growth may come from advancing along two fronts at the same time.
On one hand, the number of AI researchers is increasing; on the other hand, each individual AI researcher continues to become stronger themselves.
So the truly counterintuitive aspect of the intelligence explosion now emerges.
A natural constraint on past tech growth has been that training researchers is slow: training a truly excellent researcher takes more than a decade of education and training.
AI researchers have no such constraint. Once sufficient capability is reached, the training step could become copying models and allocating compute.
As a result, scientific research may for the first time face an extremely strange supply shock: research capability itself becomes software.
And once research capability becomes software, both variables—researcher numbers and researcher capability—could be pushed forward simultaneously by technological progress.
This is where the real danger of the intelligence explosion described in the paper lies: the source of acceleration is itself being accelerated.
RSI Has Emerged, But the Intelligence Explosion Hasn't Closed the Loop
Reading this far, it's easy to reach a particularly exciting conclusion: it's over, the intelligence explosion has already begun. (crying)
But the overarching tone of Hinton's paper is that the current evidence is still far from sufficient to prove that an intelligence explosion has occurred. (laughing)
Because the productivity gains brought by current AI R&D automation have still not clearly crossed the threshold required to trigger explosive acceleration.
The authors even directly remind readers that between AI R&D automation and an intelligence explosion lies a long chain of conditions.
First, AI must be reliable enough to work autonomously for long periods; second, it must genuinely increase R&D output; and then, these new R&D results must in turn visibly enhance AI's own R&D capability.
Moreover, even if millions of AI researchers emerge in the future, there are at least several very real walls in front of them.
The first wall is compute.
AI researchers can be copied, but GPUs cannot be copied out of thin air. If a large number of key experiments all require running large models, no matter how many agents there are, they may ultimately end up queuing for GPUs.
In particular, training a true frontier model once can itself take months; even 100 ten-thousand AI researchers sitting at their computers cannot compress a training task that physically takes three months into five minutes.
This is one of the biggest real-world constraints on a software-driven intelligence explosion: software iteration can be extremely fast, but the expansion of underlying computing infrastructure remains limited by the physical world.
The second wall is data.
Naturally occurring internet data does not grow in step with the number of AI researchers.
According to the trends discussed in the paper, existing high-quality natural data is likely to become increasingly insufficient to meet continuously expanding training needs.
Whether synthetic data, verifiable tasks, environmental interaction, and other approaches can continue to provide high-quality training signals in the future remains unsettled.
The third wall is experiment time.
Some things are inherently hard to parallelize infinitely: training models takes time, chip manufacturing takes time, and building new data centers also takes time.
Even if 1 million agents propose experiment plans simultaneously, they still have to share limited GPUs, experiment clusters, and training windows.
And there is one final, more fundamental problem: research will become harder and harder.
The easy-to-find algorithmic improvements may be found first, and from then on every bit of capability gain will require ever greater investment.
The paper calls this diminishing returns—
the further research advances, the more expensive the next breakthrough may become; if research difficulty rises faster than AI R&D capability, the intelligence explosion naturally cannot take off.
In summary, although the paper does not have enough evidence to prove that an intelligence explosion will definitely occur, the existing evidence is already sufficient to turn it from a science-fiction question into a real problem that requires advance preparation.
And recursive self-improvement is indeed moving further and further away from being a "science-fiction setting."
Looking back from the future, the milestone of 2026 truly worth remembering may well not be some model scoring a few more points.
It may be that humanity, for the first time, began to seriously realize that AI's most powerful Scaling might happen to the very endeavor of "researching AI."
Reference link:
[1] https://x.com/geoffreyhinton/status/2106122709285368061?s=20