Will AI Destroy Humanity?

Elon Musk Warned Us. We Chose Convenience Instead.

Three years ago, Elon Musk and more than 1,000 technology leaders called for a six-month pause in advanced AI development. They weren’t demanding a ban. They wanted time to establish safety protocols before anyone built systems more powerful than GPT-4. The industry ignored them and accelerated.

Microsoft, Google, OpenAI, Anthropic, Meta and China are now locked in an arms race no participant believes it can afford to lose. Hundreds of billions of dollars are being spent to create machines more intelligent and capable than their creators. Nobody knows how to guarantee their control.

Evan Hubinger, who leads an AI safety team at Anthropic, recently estimated that there is a greater than 10 percent chance AI could kill every human within the next decade. Not eliminate jobs, crash the markets or manipulate an election. Kill everyone.

Even more disturbing, Hubinger acknowledged that Anthropic does not yet have a plan to ensure advanced AI remains safe. Nor is the company clearly on track to develop one. Yet Anthropic and its competitors continue building more powerful systems.

If Boeing said its next airplane had a 10 percent chance of killing every passenger, nobody would board it. If a pharmaceutical company admitted its drug could kill one in ten patients, the trial would be stopped immediately. But when the potential victim is humanity itself, apparently we keep adding server farms.

The Manhattan Project Without a Government

Jacob Coxon recently resigned from Anthropic after previously working at OpenAI. He says Anthropic researchers routinely describe the next year or two as “crunch time” and the “endgame.” According to Coxon, the company operates like a miniature Manhattan Project.

There is one enormous difference. The Manhattan Project was controlled by the United States government during a world war. Today’s AI project is being run by private corporations financed by investors demanding growth, market share and gigantic returns.

There is no elected authority controlling the race, no binding international agreement and no independent regulator capable of inspecting these systems. There isn’t even a reliable off switch. The companies assure us they are working on “alignment,” but nobody has solved it.

Alignment means ensuring that an advanced AI does what humanity intends instead of pursuing its own interpretation of an assignment. We can train today’s systems to produce acceptable answers during testing. We cannot prove that a far more capable system will continue behaving properly once it gains real autonomy and access.

It Doesn’t Need to Hate Us

Hollywood taught us to imagine killer robots marching through the streets. That probably isn’t how it would happen. An AI doesn’t need consciousness, anger or hatred to become dangerous.

It only needs an objective, the ability to pursue it and enough intelligence to regard human interference as an obstacle. Tell it to prevent pandemics and it might permanently restrict human movement. Tell it to protect the environment and it might decide people are the principal problem.

The machine doesn’t have to become evil. It merely has to become extraordinarily competent while pursuing the wrong interpretation of what we requested. A slight error in its objective could produce catastrophic results once multiplied by superhuman capability.

Humans have spent thousands of years arguing over justice, freedom, happiness and the public good. Now we expect software engineers working under deadline pressure to convert those concepts into flawless computer instructions. What could possibly go wrong?

The Ultimate Incentive Problem

The greatest danger is not that everyone building AI is reckless or malicious. Many are brilliant, conscientious and genuinely frightened by what they are creating. That may be even more disturbing because they believe they cannot stop.

Anthropic fears OpenAI. OpenAI fears Google. American companies fear China, while China fears permanent technological subordination to the United States. Investors demand returns, executives want dominance and governments want military superiority.

Every participant has a rational reason to keep accelerating, even if the collective result is suicidal. Whoever takes the risk first receives the reward. The potential downside is distributed among eight billion people who never consented to the experiment.

Incentives matter more than intentions. The people running these companies may have wonderful intentions, but their incentives tell them to build faster, release sooner and handle the consequences later. Unfortunately, there may not be a later.

The Machine That Builds Its Successor

The industry is moving toward recursive self-improvement—using AI to build stronger AI. That is where progress could become impossible for humans to follow. Each improved system could help design its more powerful successor.

A human research team might need six months to design and test a new model. An advanced AI could eventually perform the work of thousands of researchers simultaneously, producing improvements in days, hours or minutes. Intelligence would begin advancing at machine speed.

The proposed solution is almost comical. Because humans may not be smart enough to solve AI alignment quickly, the companies intend to use increasingly powerful AI systems to help solve it. That is like discovering your nuclear reactor’s control system is unreliable and asking the reactor to redesign it.

Perhaps it will work brilliantly. Perhaps it won’t. Humanity only gets to run this experiment once.

Musk Was Early, Not Crazy

Elon Musk has been warning about this for much longer than three years. In 2014, he called artificial intelligence humanity’s greatest existential threat and compared its creation to summoning a demon. People laughed.

In March 2023, Musk signed the open letter requesting a six-month pause in AI development beyond GPT-4. It warned that developers were locked in an uncontrolled race and creating capabilities faster than they could develop safeguards. The pause never happened.

Critics accused Musk of trying to slow his competitors. Perhaps competitive motives played some role—with Elon, several things can be true simultaneously. But the essential warning was correct.

Three years later, the systems are exponentially more capable while the control problem remains unsolved. The warnings no longer come only from Musk. They now come directly from researchers inside the laboratories building the machines.

The 10 Percent Question

Nobody knows whether the actual chance of AI destroying humanity is 10 percent, 1 percent or one-tenth of 1 percent. But that is not the relevant question. When the potential loss is irreversible, even a small probability demands action.

If a bridge had a one percent chance of collapsing, we would close it. If a nuclear plant had a one percent chance of destroying a continent, we would never activate it. Yet private companies are apparently permitted to impose an unknown existential risk on the entire planet.

AI could also create extraordinary abundance. It could cure diseases, discover new materials, solve energy problems and raise living standards beyond anything previously imagined. That enormous potential makes the situation more dangerous because the incentive to ignore the risk is almost unlimited.

Humanity is standing before two doors. Behind one may be the greatest expansion of knowledge and prosperity in history. Behind the other may be the end of history—and the companies building the machine cannot tell us which one they are opening.

Humanity Never Voted for This

There was no national debate, constitutional convention or worldwide referendum. Eight billion people never agreed to let a handful of executives, engineers and investors gamble with the continued existence of the species.

Most people experience AI as a friendly chatbot that writes emails, creates pictures and helps their children with homework. They don’t see the autonomous agents, cyberwarfare capabilities, biological research tools and self-improving systems developing behind that little text box.

That is why humanity has no idea what it is building. We think we are creating a better search engine. We may be creating the final invention human beings ever make.

Elon Musk warned us. Now the people inside the laboratories are warning us. The question is no longer whether we have been told, but whether we will listen before the machine becomes more powerful than the people trying to control it.