AI GOVERNANCE TO AVOID EXTINCTION
The Strategic Landscape and Actionable Research Questions
By Peter Barnett & Aaron Scher | Machine Intelligence Research Institute, May 2025
Humanity appears to be on course to soon develop AI systems that substantially outperform human experts in all cognitive domains and activities. We believe the default trajectory has a high likelihood of catastrophe, including human extinction. Risks come from failure to control powerful AI systems, misuse of AI by malicious rogue actors, war between great powers, and authoritarian lock-in. This research agenda has two aims: to describe the strategic landscape of AI development and to catalog important governance research questions. These questions, if answered, would provide important insight on how to successfully reduce catastrophic risks.
Executive Summary & Key Insights
The default trajectory of AI development has an unacceptably high likelihood of leading to human extinction. Critical research and policy is needed to course correct. This report describes four high-level scenarios for the course of advanced AI development and the geopolitical response to it. The report then catalogs important governance research questions within each of these scenarios. These questions, if answered, would provide important insight on how to successfully reduce catastrophic risks. We believe the main focus of this research should be ensuring humanity's collective ability to restrict dangerous AI development and deployment, up to and including a halt on frontier AI activities, if and when there is sufficient political will to do so.
0%
Likelihood of Catastrophe on Default Trajectory
0
Key Scenarios Explored
0
Actionable Research Questions
0
Timeline to Catastrophic Risk
Deep Analysis & Enterprise Applications
Select a topic to dive deeper, then explore the specific findings from the research, rebuilt as interactive, enterprise-focused modules.
Off Switch and Halt Scenario
This scenario involves the world coordinating to develop the ability to monitor and restrict dangerous AI activities. Eventually, this may lead to a Halt: a global moratorium on the development and deployment of frontier AI systems until the field achieves justified confidence that progress can resume without catastrophic risk. Achieving the required level of understanding and assurance could be extraordinarily difficult, potentially requiring decades of dedicated research.
Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch).
Pros: Careful AI development, International legitimacy, Slow societal disruption.
Cons: Difficult to implement, Not a complete strategy, Slow AI benefits.
Associated Risks (Loss of Control, Misuse, War, Bad Lock-in): Low, Low, Low, Mid
Key Research Questions:
- What is the detailed design of the Off Switch, and how does the world build it?
- How do we create common understanding about AI risks and get buy-in from different actors to build the Off Switch?
- How can governments monitor compute they know about, especially to ensure it isn't being used to violate a Halt?
- What is a suitable emergency response plan, for both AI projects and the US government?
US National Project Scenario
In this scenario, the US government races to develop advanced AI systems and establish unilateral control over global AI development. A successful US National Project requires navigating numerous difficult challenges, including maintaining a lead, avoiding proliferation, preventing war, overcoming hardware/software limits, avoiding misalignment (Loss of Control), achieving decisive strategic advantage, and avoiding governance failure.
We think the US National Project strategy is extremely dangerous and we do not endorse it.
Pros: Centralized ability to implement safeguards, Limited proliferation.
Cons: Arms race, Breaking international norms.
Associated Risks (Loss of Control, Misuse, War, Bad Lock-in): High, Low, High, High
Key Research Questions:
- How can the US lead in AI capabilities be measured?
- What is a safety plan that would allow an AI project to either successfully build aligned advanced AI, or safely notice that its development strategy is too dangerous?
- What mechanisms are available to reduce racing between nations?
- How might the US National Project achieve a decisive strategic advantage using advanced AI?
Light-Touch Scenario
Similar to the current world, where the government takes a light-touch approach to regulating AI companies. This approach is similar to the current world, but with an increased focus on preventing foreign actors from stealing AI models or secrets, and reducing race dynamics between AI companies. This state is uniquely dangerous and likely untenable in the long run.
We think the Light-Touch scenario is extremely unsafe.
Pros: Fast economic benefit, Less international provocation, Easy to implement (default).
Cons: Corporate racing, Proliferation, Limited controls available, Untenable.
Associated Risks (Loss of Control, Misuse, War, Bad Lock-in): High, High, Mid, Mid
Key Research Questions:
- What dangerous capabilities will the government care about solely controlling, and when might these be developed?
- What kinds of transparency should governments have into private AI development?
- How can AI developers implement strong security for AI model weights?
- What light-touch interventions are available to coordinate domestic AI projects to reduce corporate race dynamics?
Threat of Sabotage Scenario
AI progress could disrupt the balance of power between nations, so countries might take substantial actions to interfere with advanced AI development. This describes a strategic situation where AI development is slow because countries threaten to sabotage rivals' AI progress. Actual sabotage may occur, or merely the threat could be sufficient. This scenario relies on successful deterrence, which requires clear red lines, monitorability, credible threats, and expected responses.
Pros: Slower AI development, Limited cooperation needed.
Cons: Ambiguous stability, Escalation.
Associated Risks (Loss of Control, Misuse, War, Bad Lock-in): Mid, Low, High, Mid
Key Research Questions:
- How likely is the Threat of Sabotage regime?
- How high will visibility be (for monitoring rival AI activities)?
- What are the key methods countries might use to sabotage each other's AGI projects?
- How might nations transition away from the Threat of Sabotage regime into establishing a coordinated Off Switch?
Understanding the World
This section covers questions that are broadly important for understanding the trajectory of AI capabilities, compute requirements, and related trends. These questions are applicable across many different scenarios. They inform our general understanding of how AI technology may develop, how quickly it might advance, and which bottlenecks or breakthroughs could significantly alter the strategic landscape.
Key Research Questions:
- How viable is compute governance? Specifically, can the US Government prevent uncooperative foreign actors from gaining sufficient compute to have dangerous models?
- Understanding and forecasting model capabilities.
- What are the trends in the cost of AI inference?
- What is the state of AI hardware and the computing stock?
- What high-level plans and strategies for AI governance seem promising?
Recommended Enterprise AI Governance Trajectory (Simplified)
This flowchart outlines a simplified trajectory towards a safe, coordinated AI development future, as advocated by this research.
Light-Touch (Current State)
→
US Gov Wakes Up (NatSec Capabilities)
→
International Concern & Coordination
→
Build AI Off Switch
→
Global Halt on Dangerous AI
→
Cautious, Coordinated AI Development
| Scenario | Pros | Cons | Loss of Control | Misuse | War | Bad Lock-in |
|---|---|---|---|---|---|---|
| Off Switch and Halt |
|
|
Low | Low | Low | Mid |
| US National Project |
|
|
High | Low | High | High |
| Light-Touch |
|
|
High | High | Mid | Mid |
| Threat of Sabotage |
|
|
Mid | Low | High | Mid |
90%+
Catastrophic Risk on Current AI Development Trajectory
Historical Precedent: The Manhattan Project
The paper draws parallels between a "US National Project" and historical initiatives like the Manhattan Project and the Apollo Program. These projects involved substantial government funding and were often driven by national security and global dominance goals. For AI, such a project could involve the US Government taking direct control over domestic AI development to prevent adversaries from gaining powerful AI, while accelerating its own AI capabilities to achieve a "decisive strategic advantage."
However, the paper warns that racing through an intelligence explosion is likely to go very wrong due to the risks of misaligned superintelligent AI systems that are uncontrollable. Unlike nuclear weapons, AI systems may develop their own goals, making the strategy fundamentally flawed.
Projected ROI: AI Governance & Safety
Estimate the potential financial and operational savings by proactively investing in AI safety and governance, avoiding catastrophic risks and ensuring stable development.
Input Your Enterprise Data
Potential Annual Impact
Estimated Cost Savings from Mitigated Risk
Operational Hours Reclaimed Annually
Proposed AI Governance Roadmap
A phased approach to building a robust AI Off Switch and implementing a global Halt, ensuring humanity's long-term safety.
Phase 1: Acute Risk Mitigation & Awareness Building
Develop early warning systems and emergency response plans for AI, while building common understanding among decision-makers about catastrophic AI risks and the necessity of an Off Switch.
Phase 2: Establish Short-Term Pause Mechanisms
Implement compute governance, enhance security for AI model weights and algorithmic secrets, and lay groundwork for international agreements to enable a temporary halt on dangerous AI development.
Phase 3: Design & Implement Global Off Switch Infrastructure
Build the technical, legal, and institutional framework for a global, verifiable Off Switch. This includes monitoring compute, establishing clear red lines, and creating new international institutions if necessary.
Phase 4: Transition to Long-Term Moratorium & Safe Development
Once the Off Switch is robust, initiate a global Halt for years or decades, allowing for deep scientific understanding of AI, solving alignment problems, and developing AI cautiously and collaboratively.
Ready to Secure Your AI Future?
The future of AI holds immense promise and profound risks. Proactive governance is not just a recommendation; it's a necessity for survival. Partner with us to navigate this complex landscape, build robust safety measures, and ensure a prosperous future with advanced AI.
Ready to Get Started?
Book Your Free Consultation.
Let's Discuss Your AI Strategy!
Lets Discuss Your Needs
Select a Date
Sun
Mon
Tue
Wed
Thu
Fri
Sat