Scaling, Reasoning, and Unknown Unknowns

The past decade of progress in artificial intelligence has primarily been driven by scaling model training. That is to say, making AI models larger, training them for longer, and exposing them to more data produces oddly predictable returns to model performance.

In recent years, there’s been much debate about whether scaling is “hitting a wall.” Or, perhaps it’s more correct to say this debate has returned in recent years. There have always been skeptics who think the current rate of progress is about to end at the next step — they’ve just been wrong every time so far.

Nonetheless, there’s new evidence in favor this time around. Reporting from The Information suggests that OpenAI saw only mild improvements while training their newest (and largest) model, codenamed Orion. Ilysa Sustskever, formerly of OpenAI and once an optimistic proponent of the so-called “scaling hypothesis,” said the following to Reuters:

“The 2010s were the age of scaling, now we’re back in the age of wonder and discovery once again. Everyone is looking for the next thing,” Sutskever said. “Scaling the right thing matters more now than ever.”

What is the next thing — the right thing that AI labs should be scaling in the decade ahead? OpenAI seems to be taking a bet on reasoning.

Earlier this year, OpenAI announced o1. This new generation of models uses reinforcement learning techniques (and a healthy amount of computational resources) to let the AI “think” about the prompt before returning an output to the user. The OpenAI report on this release made two things clear: (1) o1 is a significant improvement from past models, and (2) scaling reasoning time is producing the same sort of log-linear performance improvements that we saw with trailing. A new scaling law has been born.

Credit: ARC prize

According to Sutskever (and reading a bit between the lines), whereas the past decade was all about scaling models in training, before they were presented to the user, the next decade will include serious investment in scaling at inference time — that is, allowing models to think “in the moment.”

Or, perhaps more likely, it will be about both. OpenAI CEO Sam Altman has seemingly denied that returns from pretraining are slowing down, tweeting out: “there is no wall.” If true, it means there are now two gas pedals to press, and we can expect AI developers to be pushing hard on both.

A few years ago, we didn’t have strong evidence that using reinforcement learning to encourage language models to reason before producing an output would be so effective. Now we do.

A few decades ago, we didn’t have strong evidence that relatively naïve scaling of AI models in computational resources, model size, and training data would be so effective. Now we do.

Here’s a meta-lesson that we can draw: there are serious unknown unknowns in AI development. In Sutskever’s words, we are in “the age of wonder and discovery.” New techniques can produce unexpected step changes in model performance. Upcoming model releases could rapidly render the field — and, thus, our world — unfamiliar again.

Known Knowns

Known Unknowns

Unknown unknowns

AI scaling has produced astonishing results in the past decade

Will scaling, in training and at inference time, further past trends in model capabilities?

What new discoveries lie ahead that will suddenly change the pace and landscape of AI progress?

What’s the proper response to this state of affairs? What should average people do? What should companies do? What should governments do? Do we race ahead to beat our adversaries, or pause all frontier AI development until we have a better handle on the future, or just roll the dice and see what lies in store?

The Midas Project’s recommendation is the same as ever: an abundance of caution is warranted. Every day, we gain more and more evidence for two claims that, when put together, should scare you to your bones.

The first is that AI progress appears on track to produce, or surpass, human-level intelligence this decade. The second is that AI develops are consistently failing to manage risks and prioritize social welfare.

Just today, we’ve released an updated report on OpenAI. Included are testimonies from ex-employees, a timeline of their slow transition away from a nonprofit structure, and a history of broken promises and misleading statements.