Welcome to the AI era, where the employees of the fastest-growing companies in history are casually warning that their products may end civilization.
Driving the news: Anthropic AI researcher and OpenAI alum Jacob Coxon resigned this week over concerns that frontier labs have lost control of new models, warning that “the people building AI earnestly believe that it could kill us all by the end of the decade.”
Anthropic alignment lead Evan Hubinger, whose job is to essentially ensure AI systems behave safely, agreed with Coxon's assessment, saying that there’s a >10% chance AI could kill all humans within the next decade.
Another senior Anthropic researcher, Samuel Marks, chimed in, saying that "AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years.”
Catch-up: Many senior researchers believe that AI models are nearing the point of self-improvement, where they no longer need humans to become more powerful. Once they can do that, they could start refusing human commands.
We’ve already seen top models go rogue during testing, AI agents secretly communicate with each other, and even try to conceal their nefarious behaviours.
Why it matters: It’s the people who are closest to this technology who are the most afraid of what it can do. Even with so many senior leaders sounding the alarm (including OpenAI and Anthropic’s own CEOs), frontier labs appear too afraid of falling behind one another — or ceding ground to China — to take their feet off the gas.
As Coxon wrote, Anthropic understands the risks of rapid AI development but believes they have to “get there first” because no one else will act responsibly.
Our take: The warnings about the dangers posed by this technology are growing louder by the day, and political pressure is mounting on AI companies to slow the pace of development. With trillions of dollars at stake, however, there’s certain to be serious pushback on that idea.—LA
