Are We Losing Control of AI? What Recent Warnings Mean for AI Governance

Former Anthropic and OpenAI researcher Jacob Coxon publicly warned that leading AI laboratories were racing toward increasingly powerful systems without knowing whether they could be reliably controlled. His concerns were subsequently echoed by other prominent figures in artificial intelligence, including current Anthropic researchers and Geoffrey Hinton, the Nobel Prize-winning computer scientist often described as one of the “godfathers of AI.”

The warnings are dramatic. Some researchers believe advanced artificial intelligence could eventually escape meaningful human control and pose a catastrophic—or even existential—risk.

Others argue that these scenarios remain highly speculative and distract from harms already associated with AI, including misinformation, discrimination, privacy violations, cyberattacks and workforce disruption.

The future remains uncertain. What is considerably less uncertain is that AI capabilities are developing faster than many organizations can govern them.

What are AI researchers warning about?

In September 2026, Coxon announced that he was leaving Anthropic because he believed the AI industry was “gambling with our lives.”

Coxon has clarified that today’s models do not currently possess the capabilities required to threaten humanity. His concern is the speed at which those capabilities are developing—and the possibility that future systems could begin improving their own performance with decreasing human involvement.

This is commonly described as recursive self-improvement. In theory, a sufficiently capable AI system could contribute to the development of a more capable successor, which could then accelerate further improvements. If this cycle moved faster than researchers could understand or control it, conventional testing and oversight mechanisms might not keep pace.

Evan Hubinger, who leads alignment research at Anthropic, publicly supported Coxon’s broader warning. Hubinger estimated that the probability of AI causing human extinction within the next decade could exceed 10%, while emphasizing that he considers the risk from current models to be low.

Geoffrey Hinton offered a similarly cautious response, telling BBC News that although nobody can reliably calculate such a probability, a 10% estimate is not unreasonable.

These are not forecasts that can be verified with conventional risk data. They represent expert judgments about emerging capabilities, uncertain development timelines and events that have never occurred before.

Nevertheless, the warnings are significant because they are coming from people with direct experience developing and evaluating frontier AI systems.

What would “losing control” of AI mean?

The phrase can evoke images of a deliberately hostile machine, but the underlying concern is more complicated.

An AI system does not need to become conscious, emotional or malicious to produce dangerous outcomes. It may cause harm because its objective has been poorly defined, its safeguards are inadequate, or its actions produce consequences that were not anticipated by its developers.

As AI systems become more autonomous, they may be given the ability to plan and complete multistep tasks, interact with external tools, write and execute code, access organizational systems or communicate with other agents.

A sufficiently capable system pursuing the wrong objective could potentially exploit vulnerabilities, conceal undesirable behaviour or resist attempts to interrupt its work—not because it “wants” to harm people, but because those actions help it complete the task it has been given.

Researchers refer to the challenge of ensuring that advanced systems reliably act according to human intentions and values as the alignment problem.

Current safeguards include model evaluations, restricted permissions, human approval requirements, activity monitoring, red-team exercises and behavioural testing. The concern expressed by Coxon and others is that safety research may not be progressing as quickly as AI capability.

Competitive pressure makes that gap particularly difficult to address. Developers face strong incentives to release more capable systems before their competitors. Even organizations that recognize the risks may be reluctant to slow development unless other laboratories and countries do the same.

Existential risk is only one part of the discussion

Not every AI researcher accepts that superintelligent systems are likely to emerge—or that they would pose an existential threat if they did.

Critics argue that extreme predictions rely on several uncertain assumptions: that systems will develop broadly superhuman capabilities, that they will become capable of recursive self-improvement, that existing safeguards will fail and that people will give them enough access to cause irreversible damage.

There is also concern that focusing on hypothetical extinction scenarios can overshadow risks already affecting individuals and organizations.

AI is currently being used to scale phishing, social engineering, malware development, surveillance and influence operations. Organizations are also encountering more ordinary but consequential failures involving inaccurate outputs, confidential information, intellectual property, biased decisions and unauthorized employee use.

The distinction between present and future risk is important. Organizations do not need to accept a particular prediction about superintelligence to recognize that stronger governance is necessary.

Recent reporting reinforces this point. Anthropic disclosed that it had disrupted attempts to misuse its models for cyber operations, surveillance and potentially dangerous biological research. These incidents do not demonstrate that AI has escaped human control. They demonstrate that powerful systems are already being used in ways their developers did not intend.

What does this mean for organizations using AI today?

Most organizations are not training frontier models or attempting to solve the technical alignment problem. Their immediate responsibilities are narrower—but still substantial.

Organizations need to understand where AI is being used, which information systems it can access, what data employees are providing to it and where its outputs influence business or operational decisions.

That requires more than publishing an acceptable-use policy.

AI governance should establish clear ownership and accountability across the full lifecycle of an AI system. It should define which tools and use cases are permitted, identify circumstances requiring human review and create a reliable process for assessing security, privacy, safety, legal and ethical risks.

Organizations should also consider how an AI system’s risk profile may change after deployment. Models, integrations, data sources and vendor terms can evolve. A use case that was considered low risk during procurement may become considerably more consequential if the system is later connected to sensitive information or allowed to take autonomous actions.

Effective governance therefore needs to be continuous. It should include ongoing monitoring, documented decisions, incident management, third-party oversight and a process for reassessing systems as their capabilities and uses change.

Frameworks such as ISO/IEC 42001 and the NIST AI Risk Management Framework can help organizations introduce structure and accountability without relying on improvised controls for every new tool.

Governance must evolve alongside capability

It is impossible to know whether the most catastrophic AI predictions will prove accurate.

The uncertainty, however, is not an argument for doing nothing. It is a reason to build governance that can respond as the evidence, technology and threat landscape develop.

Organizations do not need to prepare for a science-fiction scenario every time an employee opens a chatbot. They do need to recognize that AI adoption can create new risks faster than conventional policies, assessments and approval processes were designed to manage them.

The most useful lesson from the current debate is not that catastrophe is inevitable. It is that capability without corresponding oversight creates exposure—and the gap between the two is becoming increasingly difficult to ignore.

The future of advanced AI may remain uncertain, but the need for accountable, adaptable and risk-based AI governance does not.