Anthropic co-founder warns AI may design its own successor, says humans face a ‘big decision’ before 2030

unreal superior general intelligence service or superintelligence has been ace of the to the highest degree widely cited terms in the world of AI but there is hardly any consensus on what it means or what its possible implications could be for society. That being said, leading AI labs like OpenAI, Google and Anthropic are racing to be the first to create a model that could reach AGI status.

However, Anthropic co founder and Chief Scientist Jared Kaplan, in an interview with the Guardian, explained that humanity will have “the biggest decision” on whether it takes the “ultimate risk” of letting AI systems train themselves to become more powerful.

As per Kaplan, the period between 2027 and 2030 may become the moment when artificial intelligence becomes capable of designing its own successors.

The Anthropic executive says that he is very optimistic about the alignment of AI tools with the interests of humanity up to the level of human intelligence but not when it exceeds that threshold.

Kaplan on AI training its successor

The moment an AI system begins training its own successor, the guardrails that AI labs currently have on their models may no longer be enough. Kaplan believes it could lead to an intelligence explosion and may even be the moment when humans lose control over the AI.

“If you imagine you create this process where you have an AI that is smarter than you, or about as smart as you, it’s then making an AI that’s much smarter. It’s going to enlist that AI’s help to make an AI smarter than that. It sounds like a kind of scary process. You don’t know where you end up,” he told the Guardian.

In such a scenario, the AI black box problem would become absolute, where humans would not just be unsure why the AI made a decision but would not even be able to tell where the AI is going.

“That’s the thing that we view as maybe the biggest decision or scariest thing to do… once no one’s involved in the process, you don’t really know. You can start a process and say, ‘Oh, it’s going very well. It’s exactly what we expected. It’s very safe.’ But you don’t know – it’s a dynamic process. Where does that lead?” he noted.

Kaplan says there are two major risks in such a scenario. First, will humans lose control over the AI and will they continue to have agency in their lives?

“One is do you lose control over it? Do you even know what the AIs are doing? The main question there is: are the AIs good for humanity? Are they helpful? Are they going to be harmless? Do they understand people? Are they going to allow people to continue to have agency over their lives and over the world?” Kaplan noted.

The second risk is when the speed of improvement of self taught AIs goes beyond human scientific research and technological development capabilities.

“It seems very dangerous for it to fall into the wrong hands… You can imagine some person deciding: ‘I want this AI to just be my slave. I want it to enact my will.’ I think preventing power grabs, preventing misuse of the technology, is also very important,” he said.

---

---