Technology

Doom Debates!

It’s time to talk about the end of the world. With your host, Liron Shapira.

lironshapira.substack.com

Latest Episodes:

AI Could Give Humans MORE Control — Ozzie Gooen Apr 24, 2025

Show notes

Ozzie Gooen is the founder of the Quantified Uncertainty Research Institute (QURI), a nonprofit building software tools for forecasting and policy analysis. I’ve known him through the rationality community since 2008 and we have a lot in common.

00:00 Introducing Ozzie

02:18 The Rationality Community

06:32 What’s Your P(Doom)™

08:09 High-Quality Discourse and Social Media

14:17 Guesstimate and Squiggle Demos

31:57 Prediction Markets and Rationality

38:33 Metaforecast Demo

41:23 Evaluating Everything with LLMs

47:00 Effective Altruism and FTX Scandal

56:00 The Repugnant Conclusion Debate

01:02:25 AI for Governance and Policy

01:12:07 PauseAI Policy Debate

01:30:10 Status Quo Bias

01:33:31 Decaf Coffee and Caffeine Powder

01:34:45 Are You Aspie?

01:37:45 Billionaires in Effective Altruism

01:48:06 Gradual Disempowerment by AI

01:55:36 LessOnline Conference

01:57:34 Supporting Ozzie’s Work

Show Notes

Quantified Uncertainty Research Institute (QURI) — https://quantifieduncertainty.org

Ozzie’s Facebook — https://www.facebook.com/ozzie.gooen

Ozzie’s Twitter — https://x.com/ozziegooen

Guesstimate, a spreadsheet for working with probability ranges — https://www.getguesstimate.com

Squiggle, a programming language for building Monte Carlo simulations — https://www.squiggle-language.com

Metaforecast, a prediction market aggregator — https://metaforecast.org

Open Annotate, AI-powered content analysis — https://github.com/quantified-uncertainty/open-annotate/

Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.online

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Top AI Professor Has 85% P(Doom) — David Duvenaud, Fmr. Anthropic Safety Team Lead Apr 18, 2025

Show notes

David Duvenaud is a professor of Computer Science at the University of Toronto, co-director of the Schwartz Reisman Institute for Technology and Society, former Alignment Evals Team Lead at Anthropic, an award-winning machine learning researcher, and a close collaborator of Dr. Geoffrey Hinton. He recently co-authored Gradual Disempowerment.

We dive into David’s impressive career, his high P(Doom), his recent tenure at Anthropic, his views on gradual disempowerment, and the critical need for improved governance and coordination on a global scale.

00:00 Introducing David

03:03 Joining Anthropic and AI Safety Concerns

35:58 David’s Background and Early Influences

45:11 AI Safety and Alignment Challenges

54:08 What’s Your P(Doom)™

01:06:44 Balancing Productivity and Family Life

01:10:26 The Hamming Question: Are You Working on the Most Important Problem?

01:16:28 The PauseAI Movement

01:20:28 Public Discourse on AI Doom

01:24:49 Courageous Voices in AI Safety

01:43:54 Coordination and Government Role in AI

01:47:41 Cowardice in AI Leadership

02:00:05 Economic and Existential Doom

02:06:12 Liron’s Post-Show

Show Notes

David’s Twitter — https://x.com/DavidDuvenaud

Schwartz Reisman Institute for Technology and Society — https://srinstitute.utoronto.ca/

Jürgen Schmidhuber’s Home Page — https://people.idsia.ch/~juergen/

Ryan Greenblatt's LessWrong comment about a future scenario where there's a one-time renegotiation of power and heat from superintelligent AI projects causes the oceans to boil: https://www.lesswrong.com/posts/pZhEQieM9otKXhxmd/gradual-disempowerment-systemic-existential-risks-from?commentId=T7KZGGqq2Z4gXZsty

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

“AI 2027” — Top Superforecaster's Imminent Doom Scenario Apr 15, 2025

Show notes

AI 2027, a bombshell new paper by the AI Futures Project, is a highly plausible scenario of the next few years of AI progress. I like this paper so much that I made a whole episode about it.

00:00 Overview of AI 2027

05:13 2025: Stumbling Agents

16:23 2026: Advanced Agents

21:49 2027: The Intelligence Explosion

29:13 AI's Initial Exploits and OpenBrain's Secrecy

30:41 Agent-3 and the Rise of Superhuman Engineering

37:05 The Creation and Deception of Agent-5

44:56 The Race Scenario: Humanity's Downfall

48:58 The Slowdown Scenario: A Glimmer of Hope

53:49 Final Thoughts

Show Notes

The website: https://ai-2027.com

Scott Alexander’s blog: https://astralcodexten.com

Daniel Kokotajlo’s previous predictions from 2021 about 2026: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Top Economist Sees AI Doom Coming — Dr. Peter Berezin, BCA Research Apr 09, 2025

Show notes

Dr. Peter Berezin is the Chief Global Strategist and Director of Research at BCA Research, the largest Canadian investment research firm. He’s known for his macroeconomics research reports and his frequent appearances on Bloomberg and CNBC.

Notably, Peter is one of the only macroeconomists in the world who’s forecasting AI doom! He recently published a research report estimating a “ more than 50/50 chance AI will wipe out all of humanity by the middle of the century”.

00:00 Introducing Peter Berezin

01:59 Peter’s Economic Predictions and Track Record

05:50 Investment Strategies and Beating the Market

17:47 The Future of Human Employment

26:40 Existential Risks and the Doomsday Argument

34:13 What’s Your P(Doom)™

39:18 Probability of non-AI Doom

44:19 Solving Population Decline

50:53 Constraining AI Development

53:40 The Multiverse and Its Implications

01:01:11 Are Other Economists Crazy?

01:09:19 Mathematical Universe and Multiverse Theories

01:19:43 Epistemic vs. Physical Probability

01:33:19 Reality Fluid

01:39:11 AI and Moral Realism

01:54:18 The Simulation Hypothesis and God

02:10:06 Liron’s Post-Show

Show Notes

Peter’s Twitter: https://x.com/PeterBerezinBCA

Peter’s old blog — https://stockcoach.blogspot.com

Peter’s 2021 BCA Research Report: “Life, Death and Finance in the Cosmic Multiverse” — https://www.bcaresearch.com/public/content/GIS_SR_2021_12_21.pdf

M.C. Escher’s “Circle Limit IV” — https://www.escherinhetpaleis.nl/escher-today/circle-limit-iv-heaven-and-hell/

Zvi Mowshowitz’s Blog (Liron’s recommendation for best AI news & analysis) — https://thezvi.substack.com

My Doom Debates episode about why nuclear proliferation is bad — https://www.youtube.com/watch?v=ueB9iRQsvQ8

Robin Hanson’s “Mangled Worlds” paper — https://mason.gmu.edu/~rhanson/mangledworlds.html

Uncontrollable by Darren McKee (Liron’s recommended AI x-risk book) — https://www.amazon.com/dp/B0CNNYKVH1

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI News: GPT-4o Images, AI Unemployment, Emmett Shear's New Safety Org — with Nathan Labenz Apr 03, 2025

Show notes

Nathan Labenz, host of The Cognitive Revolution, joins me for an AI news & social media roundup!

00:00 Introducing Nate

05:18 What’s Your P(Doom)™

23:22 GPT-4o Image Generation

40:20 Will Fiverr’s Stock Crash?

47:41 AI Unemployment

55:11 Entrepreneurship

01:00:40 OpenAI Valuation

01:09:29 Connor Leahy’s Hair

01:13:28 Mass Extinction

01:25:30 Is anyone feeling the doom vibes?

01:38:20 Rethinking AI Individuality

01:40:35 “Softmax” — Emmett Shear's New AI Safety Org

01:57:04 Anthropic's Mechanistic Interpretability Paper

02:10:11 International Cooperation for AI Safety

02:18:43 Final Thoughts

Show Notes

Nate’s Twitter: https://x.com/labenz

Nate’s podcast: https://cognitiverevolution.ai and https://youtube.com/@CognitiveRevolutionPodcast

Nate’s company: https://waymark.com/

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

How an AI Doomer Sees The World — Liron on The Human Podcast Mar 28, 2025

Show notes

In this special cross-posted episode of Doom Debates, originally posted here on The Human Podcast, we cover a wide range of topics including the definition of “doom”, P(Doom), various existential risks like pandemics and nuclear threats, and the comparison of rogue AI risks versus AI misuse risks.

00:00 Introduction

01:47 Defining Doom and AI Risks

05:53 P(Doom)

10:04 Doom Debates’ Mission

16:17 Personal Reflections and Life Choices

24:57 The Importance of Debate

27:07 Personal Reflections on AI Doom

30:46 Comparing AI Doom to Other Existential Risks

33:42 Strategies to Mitigate AI Risks

39:31 The Global AI Race and Game Theory

43:06 Philosophical Reflections on a Good Life

45:21 Final Thoughts

Show Notes

The Human Podcast with Joe Murray: https://www.youtube.com/@thehumanpodcastofficial

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Don’t miss the other great AI doom show, For Humanity: https://youtube.com/@ForHumanityAIRisk

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Gödel's Theorem Says Intelligence ≠ Power? AI Doom Debate with Alexander Campbell Mar 21, 2025

Show notes

Alexander Campbell claims that having superhuman intelligence doesn’t necessarily translate into having vast power, and that Gödel's Incompleteness Theorem ensures AI can’t get too powerful. I strongly disagree.

Alex has a Master's of Philosophy in Economics from the University of Oxford and an MBA from the Stanford Graduate School of Business, has worked as a quant trader at Lehman Brothers and Bridgewater Associates, and is the founder of Rose AI, a cloud data platform that leverages generative AI to help visualize data.

This debate was recorded in August 2023.

00:00 Intro and Alex’s Background

05:29 Alex's Views on AI and Technology

06:45 Alex’s Non-Doomer Position

11:20 Goal-to-Action Mapping

15:20 Outcome Pump Thought Experiment

21:07 Liron’s Doom Argument

29:10 The Dangers of Goal-to-Action Mappers

34:39 The China Argument and Existential Risks

45:18 Ideological Turing Test

48:38 Final Thoughts

Show Notes

Alexander Campbell’s Twitter: https://x.com/abcampbell

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Alignment is EASY and Roko's Basilisk is GOOD?! Mar 17, 2025

Show notes

Roko Mijic has been an active member of the LessWrong and AI safety community since 2008. He’s best known for “Roko’s Basilisk”, a thought experiment he posted on LessWrong that made Eliezer Yudkowsky freak out, and years later became the topic that helped Elon Musk get interested in Grimes.

His view on AI doom is that:* AI alignment is an easy problem* But the chaos and fighting from building superintelligence poses a high near-term existential risk* But humanity’s course without AI has an even higher near-term existential risk

While my own view is very different, I’m interested to learn more about Roko’s views and nail down our cruxes of disagreement.

00:00 Introducing Roko

03:33 Realizing that AI is the only thing that matters

06:51 Cyc: AI with “common sense”

15:15 Is alignment easy?

21:19 What’s Your P(Doom)™

25:14 Why civilization is doomed anyway

37:07 Roko’s AI nightmare scenario

47:00 AI risk mitigation

52:07 Market Incentives and AI Safety

57:13 Are RL and GANs good enough for superalignment?

01:00:54 If humans learned to be honest, why can’t AIs?

01:10:29 Is our test environment sufficiently similar to production?

01:23:56 AGI Timelines

01:26:35 Headroom above human intelligence

01:42:22 Roko’s Basilisk

01:54:01 Post-Debate Monologue

Show Notes

Roko’s Twitter: https://x.com/RokoMijic

Explanation of Roko’s Basilisk on LessWrong: https://www.lesswrong.com/w/rokos-basilisk

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Roger Penrose is WRONG about Gödel's Theorem and AI Consciousness Mar 10, 2025

Show notes

Sir Roger Penrose is a mathematician, mathematical physicist, philosopher of science, and Nobel Laureate in Physics.

His famous body of work includes Penrose diagrams, twistor theory, Penrose tilings, and the incredibly bold claim that intelligence and consciousness are uncomputable physical phenomena related to quantum wave function collapse.

Dr. Penrose is such a genius that it's just interesting to unpack his worldview, even if it’s totally implausible. How can someone like him be so wrong? What exactly is it that he's wrong about? It's interesting to try to see the world through his eyes, before recoiling from how nonsensical it looks.

00:00 Episode Highlights

01:29 Introduction to Roger Penrose

11:56 Uncomputability

16:52 Penrose on Gödel's Incompleteness Theorem

19:57 Liron Explains Gödel's Incompleteness Theorem

27:05 Why Penrose Gets Gödel Wrong

40:53 Scott Aaronson's Gödel CAPTCHA

46:28 Penrose's Critique of the Turing Test

48:01 Searle's Chinese Room Argument

52:07 Penrose's Views on AI and Consciousness

57:47 AI's Computational Power vs. Human Intelligence

01:21:08 Penrose's Perspective on AI Risk

01:22:20 Consciousness = Quantum Wave Function Collapse?

01:26:25 Final Thoughts

Show Notes

Source video — Feb 22, 2025 Interview with Roger Penrose on “This Is World” — https://www.youtube.com/watch?v=biUfMZ2dts8

Scott Aaronson’s “Gödel CAPTCHA” — https://www.scottaaronson.com/writings/captcha.html

My recent Scott Aaronson episode — https://www.youtube.com/watch?v=xsGqWeqKjEg

My explanation of what’s wrong with arguing “by definition” — https://www.youtube.com/watch?v=ueam4fq8k8I

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

We Found AI's Preferences — What David Shapiro MISSED in this bombshell Center for AI Safety paper Feb 21, 2025

Show notes

The Center for AI Safety just dropped a fascinating paper — they discovered that today’s AIs like GPT-4 and Claude have preferences! As in, coherent utility functions. We knew this was inevitable, but we didn’t know it was already happening.

This episode has two parts:

In Part I (48 minutes), I react to David Shapiro’s coverage of the paper and push back on many of his points.

In Part II (60 minutes), I explain the paper myself.

00:00 Episode Introduction

05:25 PART I: REACTING TO DAVID SHAPIRO

10:06 Critique of David Shapiro's Analysis

19:19 Reproducing the Experiment

35:50 David's Definition of Coherence

37:14 Does AI have “Temporal Urgency”?

40:32 Universal Values and AI Alignment

49:13 PART II: EXPLAINING THE PAPER

51:37 How The Experiment Works

01:11:33 Instrumental Values and Coherence in AI

01:13:04 Exchange Rates and AI Biases

01:17:10 Temporal Discounting in AI Models

01:19:55 Power Seeking, Fitness Maximization, and Corrigibility

01:20:20 Utility Control and Bias Mitigation

01:21:17 Implicit Association Test

01:28:01 Emailing with the Paper’s Authors

01:43:23 My Takeaway

Show Notes

David’s source video: https://www.youtube.com/watch?v=XGu6ejtRz-0

The research paper: http://emergent-values.ai

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at

https://doomdebates.com

and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe