Technology

Doom Debates!

It’s time to talk about the end of the world. With your host, Liron Shapira.

lironshapira.substack.com

Latest Episodes:

Roon vs. Liron: AI Doom Debate Dec 18, 2024

Show notes

Roon is a member of the technical staff at OpenAI. He’s a highly respected voice on tech Twitter, despite being a pseudonymous cartoon avatar account. In late 2021, he invented the terms “shape rotator” and “wordcel” to refer to roughly visual/spatial/mathematical intelligence vs. verbal intelligence. He is simultaneously a serious thinker, a builder, and a shitposter.

I'm excited to learn more about Roon, his background, his life, and of course, his views about AI and existential risk.

00:00 Introduction

02:43 Roon’s Quest and Philosophies

22:32 AI Creativity

30:42 What’s Your P(Doom)™

54:40 AI Alignment

57:24 Training vs. Production

01:05:37 ASI

01:14:35 Goal-Oriented AI and Instrumental Convergence

01:22:43 Pausing AI

01:25:58 Crux of Disagreement

1:27:55 Dogecoin

01:29:13 Doom Debates’s Mission

Show Notes

Follow Roon: https://x.com/tszzl

For Humanity: An AI Safety Podcast with John Sherman — https://www.youtube.com/@ForHumanityPodcast

Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16Fg

PauseAI, the volunteer organization I’m part of — https://pauseai.info/

Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Scott Aaronson Makes Me Think OpenAI's “Safety” Is Fake, Clueless, Reckless and Insane Dec 11, 2024

Show notes

Today I’m reacting to the recent Scott Aaronson interview on the Win-Win podcast with Liv Boeree and Igor Kurganov.

Prof. Aaronson is the Director of the Quantum Information Center at the University of Texas at Austin. He’s best known for his research advancing the frontier of complexity theory, especially quantum complexity theory, and making complex insights from his field accessible to a wider readership via his blog.

Scott is one of my biggest intellectual influences. His famous Who Can Name The Bigger Number essay and his long-running blog are among my best memories of coming across high-quality intellectual content online as a teen. His posts and lectures taught me much of what I know about complexity theory.

Scott recently completed a two-year stint at OpenAI focusing on the theoretical foundations of AI safety, so I was interested to hear his insider account.

Unfortunately, what I heard in the interview confirms my worst fears about the meaning of “safety” at today’s AI companies: that they’re laughably clueless at how to achieve any measure of safety, but instead of doing the adult thing and slowing down their capabilities work, they’re pushing forward recklessly.

00:00 Introducing Scott Aaronson

02:17 Scott's Recruitment by OpenAI

04:18 Scott's Work on AI Safety at OpenAI

08:10 Challenges in AI Alignment

12:05 Watermarking AI Outputs

15:23 The State of AI Safety Research

22:13 The Intractability of AI Alignment

34:20 Policy Implications and the Call to Pause AI

38:18 Out-of-Distribution Generalization

45:30 Moral Worth Criterion for Humans

51:49 Quantum Mechanics and Human Uniqueness

01:00:31 Quantum No-Cloning Theorem

01:12:40 Scott Is Almost An Accelerationist?

01:18:04 Geoffrey Hinton's Proposal for Analog AI

01:36:13 The AI Arms Race and the Need for Regulation

01:39:41 Scott Aronson's Thoughts on Sam Altman

01:42:58 Scott Rejects the Orthogonality Thesis

01:46:35 Final Thoughts

01:48:48 Lethal Intelligence Clip

01:51:42 Outro

Show Notes

Scott’s Interview on Win-Win with Liv Boeree and Igor Kurganov: https://www.youtube.com/watch?v=ANFnUHcYza0

Scott’s Blog: https://scottaaronson.blog

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liron Reacts to Subbarao Kambhampati on Machine Learning Street Talk Nov 28, 2024

Show notes

Today I’m reacting to a July 2024 interview that Prof. Subbarao Kambhampati did on Machine Learning Street Talk.

Rao is a Professor of Computer Science at Arizona State University, and one of the foremost voices making the claim that while LLMs can generate creative ideas, they can’t truly reason.

The episode covers a range of topics including planning, creativity, the limits of LLMs, and why Rao thinks LLMs are essentially advanced N-gram models.

00:00 Introduction

02:54 Essentially N-Gram Models?

10:31 The Manhole Cover Question

20:54 Reasoning vs. Approximate Retrieval

47:03 Explaining Jokes

53:21 Caesar Cipher Performance

01:10:44 Creativity vs. Reasoning

01:33:37 Reasoning By Analogy

01:48:49 Synthetic Data

01:53:54 The ARC Challenge

02:11:47 Correctness vs. Style

02:17:55 AIs Becoming More Robust

02:20:11 Block Stacking Problems

02:48:12 PlanBench and Future Predictions

02:58:59 Final Thoughts

Show Notes

Rao’s interview on Machine Learning Street Talk: https://www.youtube.com/watch?v=y1WnHpedi2A

Rao’s Twitter: https://x.com/rao2z

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

This Yudkowskian Has A 99.999% P(Doom) Nov 27, 2024

Show notes

In this episode of Doom Debates, I discuss AI existential risks with my pseudonymous guest Nethys.

Nethy shares his journey into AI risk awareness, influenced heavily by LessWrong and Eliezer Yudkowsky. We explore the vulnerability of society to emerging technologies, the challenges of AI alignment, and why he believes our current approaches are insufficient, ultimately resulting in 99.999% P(Doom).

00:00 Nethys Introduction

04:47 The Vulnerable World Hypothesis

10:01 What’s Your P(Doom)™

14:04 Nethys’s Banger YouTube Comment

26:53 Living with High P(Doom)

31:06 Losing Access to Distant Stars

36:51 Defining AGI

39:09 The Convergence of AI Models

47:32 The Role of “Unlicensed” Thinkers

52:07 The PauseAI Movement

58:20 Lethal Intelligence Video Clip

Show Notes

Eliezer Yudkowsky’s post on “Death with Dignity”: https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy

PauseAI Website: https://pauseai.info

PauseAI Discord: https://discord.gg/2XXWXvErfA

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Cosmology, AI Doom, and the Future of Humanity with Fraser Cain Nov 21, 2024

Show notes

Fraser Cain is the publisher of Universe Today, co-host of Astronomy Cast, a popular YouTuber about all things space, and guess what… he has a high P(doom)! That’s why he’s joining me on Doom Debates for a very special AI + space crossover episode.

00:00 Fraser Cain’s Background and Interests

5:03 What’s Your P(Doom)™

07:05 Our Vulnerable World

15:11 Don’t Look Up

22:18 Cosmology and the Search for Alien Life

31:33 Stars = Terrorists

39:03 The Great Filter and the Fermi Paradox

55:12 Grabby Aliens Hypothesis

01:19:40 Life Around Red Dwarf Stars?

01:22:23 Epistemology of Grabby Aliens

01:29:04 Multiverses

01:33:51 Quantum Many Worlds vs. Copenhagen Interpretation

01:47:25 Simulation Hypothesis

01:51:25 Final Thoughts

SHOW NOTES

Fraser’s YouTube channel: https://www.youtube.com/@frasercain

Universe Today (space and astronomy news): https://www.universetoday.com/

Max Tegmark’s book that explains 4 levels of multiverses: https://www.amazon.com/Our-Mathematical-Universe-Ultimate-Reality/dp/0307744256

Robin Hanson’s ideas:

Grabby Aliens: https://grabbyaliens.com

The Great Filter: https://en.wikipedia.org/wiki/Great_Filter

Life in a high-dimensional space: https://www.overcomingbias.com/p/life-in-1kdhtml

---

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI Doom Debate: Vaden Masrani & Ben Chugg vs. Liron Shapira Nov 19, 2024

Show notes

Vaden Masrani and Ben Chugg, hosts of the Increments Podcast, are back for a Part II! This time we’re going straight to debating my favorite topic, AI doom.

00:00 Introduction

02:23 High-Level AI Doom Argument

17:06 How Powerful Could Intelligence Be?

22:34 “Knowledge Creation”

48:33 “Creativity”

54:57 Stand-Up Comedy as a Test for AI

01:12:53 Vaden & Ben’s Goalposts

01:15:00 How to Change Liron’s Mind

01:20:02 LLMs are Stochastic Parrots?

01:34:06 Tools vs. Agents

01:39:51 Instrumental Convergence and AI Goals

01:45:51 Intelligence vs. Morality

01:53:57 Mainline Futures

02:16:50 Lethal Intelligence Video

Show Notes

Vaden & Ben’s Podcast: https://www.youtube.com/@incrementspod

Recommended playlists from their podcast:

* The Bayesian vs Popperian Epistemology Series

* The Conjectures and Refutations Series

Vaden’s Twitter: https://x.com/vadenmasrani

Ben’s Twitter: https://x.com/BennyChugg

Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Andrew Critch vs. Liron Shapira: Will AI Extinction Be Fast Or Slow? Nov 16, 2024

Show notes

Dr. Andrew Critch is the co-founder of the Center for Applied Rationality, a former Research Fellow at the Machine Intelligence Research Institute (MIRI), a Research Scientist at the UC Berkeley Center for Human Compatible AI, and the co-founder of a new startup called Healthcare Agents.

Dr. Critch’s P(Doom) is a whopping 85%! But his most likely doom scenario isn’t what you might expect. He thinks humanity will successfully avoid a self-improving superintelligent doom scenario, only to still go extinct via the slower process of “industrial dehumanization”.

00:00 Introduction01:43 Dr. Critch’s Perspective on LessWrong Sequences06:45 Bayesian Epistemology15:34 Dr. Critch's Time at MIRI18:33 What’s Your P(Doom)™26:35 Doom Scenarios40:38 AI Timelines43:09 Defining “AGI”48:27 Superintelligence53:04 The Speed Limit of Intelligence01:12:03 The Obedience Problem in AI01:21:22 Artificial Superintelligence and Human Extinction01:24:36 Global AI Race and Geopolitics01:34:28 Future Scenarios and Human Relevance01:48:13 Extinction by Industrial Dehumanization01:58:50 Automated Factories and Human Control02:02:35 Global Coordination Challenges02:27:00 Healthcare Agents02:35:30 Final Thoughts

---

Show Notes

Dr. Critch’s LessWrong post explaining his P(Doom) and most likely doom scenarios: https://www.lesswrong.com/posts/Kobbt3nQgv3yn29pr/my-motivation-and-theory-of-change-for-working-in-ai

Dr. Critch’s Website: https://acritch.com/

Dr. Critch’s Twitter: https://twitter.com/AndrewCritchPhD

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI Twitter Beefs #2: Yann LeCun, David Deutsch, Tyler Cowen, Jack Clark, Beff Jezos, Samuel Hammond vs. Eliezer Yudkowsky, Geoffrey Hinton, Carl Feynman Nov 13, 2024

Show notes

It’s time for AI Twitter Beefs #2:

00:42 Jack Clark (Anthropic) vs. Holly Elmore (PauseAI US)

11:02 Beff Jezos vs. Eliezer Yudkowsky, Carl Feynman

18:10 Geoffrey Hinton vs. OpenAI & Meta

25:14 Samuel Hammond vs. Liron

30:26 Yann LeCun vs. Eliezer Yudkowsky

37:13 Roon vs. Eliezer Yudkowsky

41:37 Tyler Cowen vs. AI Doomers

52:54 David Deutsch vs. Liron

Twitter people referenced:

* Jack Clark: https://x.com/jackclarkSF

* Holly Elmore: https://x.com/ilex_ulmus

* PauseAI US: https://x.com/PauseAIUS

* Geoffrey Hinton: https://x.com/GeoffreyHinton

* Samuel Hammond: https://x.com/hamandcheese

* Yann LeCun: https://x.com/ylecun

* Eliezer Yudkowsky: https://x.com/esyudkowsky

* Roon: https://x.com/tszzl

* Beff Jezos: https://x.com/basedbeffjezos

* Carl Feynman: https://x.com/carl_feynman

* Tyler Cowen: https://x.com/tylercowen

* David Deutsch: https://x.com/DavidDeutschOxf

Show Notes

Holly Elmore’s EA forum post about scouts vs. soldiers

Manifund info & donation page for PauseAI US: https://manifund.org/projects/pauseai-us-2025-through-q2

PauseAI.info - join the Discord and find me in the #doom-debates channel!

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Is P(Doom) Meaningful? Epistemology Debate with Vaden Masrani and Ben Chugg Nov 08, 2024

Show notes

Vaden Masrani and Ben Chugg, hosts of the Increments Podcast, are joining me to debate Bayesian vs. Popperian epistemology.

I’m on the Bayesian side, heavily influenced by the writings of Eliezer Yudkowsky. Vaden and Ben are on the Popperian side, heavily influenced by David Deutsch and the writings of Popper himself.

We dive into the theoretical underpinnings of Bayesian reasoning and Solomonoff induction, contrasting them with the Popperian perspective, and explore real-world applications such as predicting elections and economic policy outcomes.

The debate highlights key philosophical differences between our two epistemological frameworks, and sets the stage for further discussions on superintelligence and AI doom scenarios in an upcoming Part II.

00:00 Introducing Vaden and Ben

02:51 Setting the Stage: Epistemology and AI Doom

04:50 What’s Your P(Doom)™

13:29 Popperian vs. Bayesian Epistemology

31:09 Engineering and Hypotheses

38:01 Solomonoff Induction

45:21 Analogy to Mathematical Proofs

48:42 Popperian Reasoning and Explanations

54:35 Arguments Against Bayesianism

58:33 Against Probability Assignments

01:21:49 Popper’s Definition of “Content”

01:31:22 Heliocentric Theory Example

01:31:34 “Hard to Vary” Explanations

01:44:42 Coin Flipping Example

01:57:37 Expected Value

02:12:14 Prediction Market Calibration

02:19:07 Futarchy

02:29:14 Prediction Markets as AI Lower Bound

02:39:07 A Test for Prediction Markets

2:45:54 Closing Thoughts

Show Notes

Vaden & Ben’s Podcast: https://www.youtube.com/@incrementspod

Vaden’s Twitter: https://x.com/vadenmasrani

Ben’s Twitter: https://x.com/BennyChugg

Bayesian reasoning: https://en.wikipedia.org/wiki/Bayesian_inference

Karl Popper: https://en.wikipedia.org/wiki/Karl_Popper

Vaden's blog post on Cox's Theorem and Yudkowsky's claims of "Laws of Rationality": https://vmasrani.github.io/blog/2021/the_credence_assumption/

Vaden’s disproof of probabilistic induction (including Solomonoff Induction): https://arxiv.org/abs/2107.00749

Vaden’s referenced post about predictions being uncalibrated > 1yr out: https://forum.effectivealtruism.org/posts/hqkyaHLQhzuREcXSX/data-on-forecasting-accuracy-across-different-time-horizons#Calibrations

Article by Gavin Leech and Misha Yagudin on the reliability of forecasters: https://ifp.org/can-policymakers-trust-forecasters/

Sources for claim that superforecasters gave a P(doom) below 1%: https://80000hours.org/2024/09/why-experts-and-forecasters-disagree-about-ai-risk/https://www.astralcodexten.com/p/the-extinction-tournament

Vaden’s Slides on Content vs Probability: https://vmasrani.github.io/assets/pdf/popper_good.pdf

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

15-Minute Intro to AI Doom Nov 04, 2024

Show notes

Our top researchers and industry leaders have been warning us that superintelligent AI may cause human extinction in the next decade.

If you haven't been following all the urgent warnings, I'm here to bring you up to speed.

* Human-level AI is coming soon

* It’s an existential threat to humanity

* The situation calls for urgent action

Listen to this 15-minute intro to get the lay of the land.

Then follow these links to learn more and see how you can help:

* The Compendium

A longer written introduction to AI doom by Connor Leahy et al

* AGI Ruin — A list of lethalities

A comprehensive list by Eliezer Yudkowksy of reasons why developing superintelligent AI is unlikely to go well for humanity

* AISafety.info

A catalogue of AI doom arguments and responses to objections

* PauseAI.info

The largest volunteer org focused on lobbying world government to pause development of superintelligent AI

* PauseAI Discord

Chat with PauseAI members, see a list of projects and get involved

---

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Our TOPPODCAST Picks

Follow Us

Stay Connected

Doom Debates!

Related Podcasts

Links

Stay Connected