Technology

Doom Debates!

It’s time to talk about the end of the world. With your host, Liron Shapira.

lironshapira.substack.com

Latest Episodes:

Does AI Competition = AI Alignment? Debate with Gil Mark Feb 10, 2025

Show notes

My friend Gil Mark, who leads generative AI products at LinkedIn, thinks competition among superintelligent AIs will lead to a good outcome for humanity. In his view, the alignment problem becomes significantly easier if we build multiple AIs at the same time and let them compete.

I completely disagree, but I hope you’ll find this to be a thought-provoking episode that sheds light on why the alignment problem is so hard.

00:00 Introduction

02:36 Gil & Liron’s Early Doom Days

04:58: AIs : Humans :: Humans : Ants

08:02 The Convergence of AI Goals

15:19 What’s Your P(Doom)™

19:23 Multiple AIs and Human Welfare

24:42 Gil’s Alignment Claim

42:31 Cheaters and Frankensteins

55:55 Superintelligent Game Theory

01:01:16 Slower Takeoff via Resource Competition

01:07:57 Recapping the Disagreement

01:15:39 Post-Debate Banter

Show Notes

Gil’s LinkedIn: https://www.linkedin.com/in/gilmark/

Gil’s Twitter: https://x.com/gmfromgm

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Toy Model of the AI Control Problem Feb 06, 2025

Show notes

Why does the simplest AI imaginable, when you ask it to help you push a box around a grid, suddenly want you to die?

AI doomers are often misconstrued as having "no evidence" or just "anthropomorphizing". This toy model will help you understand why a drive to eliminate humans is NOT a handwavy anthropomorphic speculation, but rather something we expect by default from any sufficiently powerful search algorithm.

We’re not talking about AGI or ASI here — we’re just looking at an AI that does brute-force search over actions in a simple grid world.

The slide deck I’m presenting was created by Jaan Tallinn, cofounder of the Future of Life Institute.

00:00 Introduction

01:24 The Toy Model

06:19 Misalignment and Manipulation Drives

12:57 Search Capacity and Ontological Insights

16:33 Irrelevant Concepts in AI Control

20:14 Approaches to Solving AI Control Problems

23:38 Final Thoughts

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligence

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Superintelligent AI vs. Real-World Engineering | Liron Reacts to Bryan Cantrill Jan 31, 2025

Show notes

Bryan Cantrill, co-founder of Oxide Computer, says in his talk that engineering in the physical world is too complex for any AI to do it better than teams of human engineers. Success isn’t about intelligence; it’s about teamwork, character and resilience.

I completely disagree.

00:00 Introduction

02:03 Bryan’s Take on AI Doom

05:55 The Concept of P(Doom)

08:36 Engineering Challenges and Human Intelligence

15:09 The Role of Regulation and Authoritarianism in AI Control

29:44 Engineering Complexity: A Case Study from Oxide Computer

40:06 The Value of Team Collaboration

46:13 Human Attributes in Engineering

49:33 AI's Potential in Engineering

58:23 Existential Risks and AI Predictions

Bryan’s original talk:

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

2,500 Subscribers Live Q&A Recording Jan 27, 2025

Show notes

Thanks for everyone who participated in the live Q&A on Friday!

The topics covered include advice for computer science students, working in AI trustworthiness, what good AI regulation looks like, the implications of the $500B Stargate project, the public's gradual understanding of AI risks, the impact of minor AI disasters, and the philosophy of consciousness.

00:00 Advice for Comp Sci Students

01:14 The $500B Stargate Project

02:36 Eliezer's Recent Podcast

03:07 AI Safety and Public Policy

04:28 AI Disruption and Politics

05:12 DeepSeek and AI Advancements

06:54 Human vs. AI Intelligence

14:00 Consciousness and AI

24:34 Dark Forest Theory and AI

35:31 Investing in Yourself

42:42 Probability of Aliens Saving Us from AI

43:31 Brain-Computer Interfaces and AI Safety

46:19 Debating AI Safety and Human Intelligence

48:50 Nefarious AI Activities and Satellite Surveillance

49:31 Pliny the Prompter Jailbreaking AI

50:20 Can’t vs. Won’t Destroy the World

51:15 How to Make AI Risk Feel Present

54:27 Keeping Doom Arguments On Track

57:04 Game Theory and AI Development Race

01:01:26 Mental Model of Average Non-Doomer

01:04:58 Is Liron a Strict Bayesian and Utilitarian?

01:09:48 Can We Rename “Doom Debates”

01:12:34 The Role of AI Trustworthiness

01:16:48 Minor AI Disasters

01:18:07 Most Likely Reason Things Go Well

01:21:00 Final Thoughts

Show Notes

Previous post where people submitted questions: https://lironshapira.substack.com/p/2500-subscribers-live-q-and-a-ask

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI Twitter Beefs #3: Marc Andreessen, Sam Altman, Mark Zuckerberg, Yann LeCun, Eliezer Yudkowsky & More! Jan 24, 2025

Show notes

It’s time for AI Twitter Beefs #3:

00:00 Introduction

01:27 Marc Andreessen vs. Sam Altman

09:15 Mark Zuckerberg

35:40 Martin Casado

47:26 Gary Marcus vs. Miles Brundage Bet

58:39 Scott Alexander’s AI Art Turing Test

01:11:29 Roon

01:16:35 Stephen McAleer

01:22:25 Emmett Shear

01:37:20 OpenAI’s “Safety”

01:44:09 Naval Ravikant vs. Eliezer Yudkowsky

01:56:03 Comic Relief

01:58:53 Final Thoughts

Show Notes

Upcoming Live Q&A: https://lironshapira.substack.com/p/2500-subscribers-live-q-and-a-ask

“Make Your Beliefs Pay Rent In Anticipated Experiences” by Eliezer Yudkowsky on LessWrong: https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiences

Scott Alexander’s AI Art Turing Test: https://www.astralcodexten.com/p/how-did-you-do-on-the-ai-art-turing

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Effective Altruism Debate with Jonas Sota Jan 17, 2025

Show notes

Effective Altruism has been a controversial topic on social media, so today my guest and I are going to settle the question once and for all: Is it good or bad?

Jonas Sota is a Software Engineer at Rippling, BA in Philosophy from UC Berkeley, who’s been observing the Effective Altruism (EA) movement in the San Francisco Bay Area for over a decade… and he’s not a fan.

00:00 Introduction

01:22 Jonas’s Criticisms of EA

03:23 Recoil Exaggeration

05:53 Impact of Malaria Nets

10:48 Local vs. Global Altruism

13:02 Shrimp Welfare

25:14 Capitalism vs. Charity

33:37 Cultural Sensitivity

34:43 The Impact of Direct Cash Transfers

37:23 Long-Term Solutions vs. Immediate Aid

42:21 Charity Budgets

45:47 Prioritizing Local Issues

50:55 The EA Community

59:34 Debate Recap

1:03:57 Announcements

Show Notes

Jonas’s Instagram: @jonas_wanders

Will MacAskill’s famous book, Doing Good Better: https://www.effectivealtruism.org/doing-good-better

Scott Alexander’s excellent post about the people he met at EA Global: https://slatestarcodex.com/2017/08/16/fear-and-loathing-at-effective-altruism-global-2017/

Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!

PauseAI, the volunteer organization I’m part of: https://pauseai.info

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

God vs. AI Doom: Debate with Bentham's Bulldog Jan 15, 2025

Show notes

Matthew Adelstein, better known as Bentham’s Bulldog on Substack, is a philosophy major at the University of Michigan and an up & coming public intellectual.

He’s a rare combination: Effective Altruist, Bayesian, non-reductionist, theist.

Our debate covers reductionism, evidence for god, the implications of a fine-tuned universe, moral realism, and AI doom.

00:00 Introduction

02:56 Matthew’s Research

11:29 Animal Welfare

16:04 Reductionism vs. Non-Reductionism Debate

39:53 The Decline of God in Modern Discourse

46:23 Religious Credences

50:24 Pascal's Wager and Christianity

56:13 Are Miracles Real?

01:10:37 Fine-Tuning Argument for God

01:28:36 Cellular Automata

01:34:25 Anthropic Principle

01:51:40 Mathematical Structures and Probability

02:09:35 Defining God

02:18:20 Moral Realism

02:21:40 Orthogonality Thesis

02:32:02 Moral Philosophy vs. Science

02:45:51 Moral Intuitions

02:53:18 AI and Moral Philosophy

03:08:50 Debate Recap

03:12:20 Show Updates

Show Notes

Matthew’s Substack: https://benthams.substack.com

Matthew's Twitter: https://x.com/BenthamsBulldog

Matthew's YouTube: https://www.youtube.com/@deliberationunderidealcond5105

Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of — https://pauseai.info/

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Debate with a former OpenAI Research Team Lead — Prof. Kenneth Stanley Jan 06, 2025

Show notes

Prof. Kenneth Stanley is a former Research Science Manager at OpenAI leading the Open-Endedness Team in 2020-2022. Before that, he was a Professor of Computer Ccience at the University of Central Florida and the head of Core AI Research at Uber. He coauthored Why Greatness Cannot Be Planned: The Myth of the Objective, which argues that as soon as you create an objective, then you ruin your ability to reach it.

In this episode, I debate Ken’s claim that superintelligent AI *won’t* be guided by goals, and then we compare our views on AI doom.

00:00 Introduction

00:45 Ken’s Role at OpenAI

01:53 “Open-Endedness” and “Divergence”

9:32 Open-Endedness of Evolution

21:16 Human Innovation and Tech Trees

36:03 Objectives vs. Open Endedness

47:14 The Concept of Optimization Processes

57:22 What’s Your P(Doom)™

01:11:01 Interestingness and the Future

01:20:14 Human Intelligence vs. Superintelligence

01:37:51 Instrumental Convergence

01:55:58 Mitigating AI Risks

02:04:02 The Role of Institutional Checks

02:13:05 Exploring AI's Curiosity and Human Survival

02:20:51 Recapping the Debate

02:29:45 Final Thoughts

SHOW NOTES

Ken’s home page: https://www.kenstanley.net/

Ken’s Wikipedia: https://en.wikipedia.org/wiki/Kenneth_Stanley

Ken’s Twitter: https://x.com/kenneth0stanley

Ken’s PicBreeder paper: https://wiki.santafe.edu/images/1/1e/Secretan_ecj11.pdf

Ken's book, Why Greatness Cannot Be Planned: The Myth of the Objective: https://www.amazon.com/Why-Greatness-Cannot-Planned-Objective/dp/3319155237

The Rocket Alignment Problem by Eliezer Yudkowsky: https://intelligence.org/2018/10/03/rocket-alignment/

---

Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of — https://pauseai.info/

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

OpenAI o3 and Claude Alignment Faking — How doomed are we? Dec 30, 2024

Show notes

OpenAI just announced o3 and smashed a bunch of benchmarks (ARC-AGI, SWE-bench, FrontierMath)!

A new Anthropic and Redwood Research paper says Claude is resisting its developers’ attempts to retrain its values!

What’s the upshot — what does it all mean for P(doom)?

00:00 Introduction

01:45 o3’s architecture and benchmarks

06:08 “Scaling is hitting a wall” 🤡

13:41 How many new architectural insights before AGI?

20:28 Negative update for interpretability

31:30 Intellidynamics — ***KEY CONCEPT***

33:20 Nuclear control rod analogy

36:54 Sam Altman's misguided perspective

42:40 Claude resisted retraining from good to evil

44:22 What is good corrigibility?

52:42 Claude’s incorrigibility doesn’t surprise me

55:00 Putting it all in perspective

---

SHOW NOTES

Scott Alexander’s analysis of the Claude incorrigibility result: https://www.astralcodexten.com/p/claude-fights-back and https://www.astralcodexten.com/p/why-worry-about-incorrigible-claude

Zvi Mowshowitz’s analysis of the Claude incorrigibility result: https://thezvi.wordpress.com/2024/12/24/ais-will-increasingly-fake-alignment/

---

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Say hi to me in the #doom-debates-podcast channel!

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI Will Kill Us All — Liron Shapira on The Flares Dec 27, 2024

Show notes

This week Liron was interview by Gaëtan Selle on @the-flares about AI doom.

Cross-posted from their channel with permission.

Original source: https://www.youtube.com/watch?v=e4Qi-54I9Zw

0:00:02 Guest Introduction

0:01:41 Effective Altruism and Transhumanism

0:05:38 Bayesian Epistemology and Extinction Probability

0:09:26 Defining Intelligence and Its Dangers

0:12:33 The Key Argument for AI Apocalypse

0:18:51 AI’s Internal Alignment

0:24:56 What Will AI's Real Goal Be?

0:26:50 The Train of Apocalypse

0:31:05 Among Intellectuals, Who Rejects the AI Apocalypse Arguments?

0:38:32 The Shoggoth Meme

0:41:26 Possible Scenarios Leading to Extinction

0:50:01 The Only Solution: A Pause in AI Research?

0:59:15 The Risk of Violence from AI Risk Fundamentalists

1:01:18 What Will General AI Look Like?

1:05:43 Sci-Fi Works About AI

1:09:21 The Rationale Behind Cryonics

1:12:55 What Does a Positive Future Look Like?

1:15:52 Are We Living in a Simulation?

1:18:11 Many Worlds in Quantum Mechanics Interpretation

1:20:25 Ideal Future Podcast Guest for Doom Debates

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Our TOPPODCAST Picks

Follow Us

Stay Connected

Doom Debates!

Related Podcasts

Links

Stay Connected