The Real AI Misalignment Problem Is Capitalism

Figures at top AI firms says that their frontier models are running out of control. It points to a problem Big Tech finds harder to admit: the rise of a predatory computational power driven solely by profit-seeking.

By Nick Couldry, Jacobin

This September, some rather technical issues regarding artificial intelligence, or rather their upshot, made headlines. The reason lay in two resignations from companies leading the development of frontier AI models, namely David Robinson, OpenAI’s Safety Officer, and Jacob Coxon, a developer at Anthropic. Each resigned because they no longer believed the companies were taking seriously the risks that frontier models’ recent “advances” pose to the survival of humanity. So, not just what most of us would call a “technical issue.”

Those fears were not hype. They were based on careful argument: a September 6 blog by Jakub Pachocki, OpenAI’s chief scientist, serves as an example. It’s worth looking more closely.

One fear the blog expressed was that frontier models are on the brink of “recursive self-improvement” (or RSI): the ability to take autonomous control over their future refinement. This could mean that, very quickly, humans lose all control over where advanced AI’s development is headed, even if it involves concentrations of computational power that by themselves pose a threat to human life.

Another fear — perhaps more alarming because the danger is already baked into our future, whatever decisions about future development are made now — is that Big Tech companies are fast losing the ability even to evaluate whether their current models (without RSI) are in fact aligned to the values they were designed to serve or aligned to completely different values that don’t serve human interests at all.