Technology

Doom Debates!

It’s time to talk about the end of the world. With your host, Liron Shapira.

lironshapira.substack.com

Latest Episodes:

Richard Sutton Dismisses AI Extinction Fears with Simplistic Arguments | Liron Reacts Aug 13, 2024

Show notes

Dr. Richard Sutton is a Professor of Computing Science at the University of Alberta known for his pioneering work on reinforcement learning, and his “bitter lesson” that scaling up an AI’s data and compute gives better results than having programmers try to handcraft or explicitly understand how the AI works.

Dr. Sutton famously claims that AIs are the “next step in human evolution”, a positive force for progress rather than a catastrophic extinction risk comparable to nuclear weapons.

Let’s examine Sutton’s recent interview with Daniel Fagella to understand his crux of disagreement with the AI doom position.

---

00:00 Introduction

03:33 The Worthy vs. Unworthy AI Successor

04:52 “Peaceful AI”

07:54 “Decentralization”

11:57 AI and Human Cooperation

14:54 Micromanagement vs. Decentralization

24:28 Discovering Our Place in the World

33:45 Standard Transhumanism

44:29 AI Traits and Environmental Influence

46:06 The Importance of Cooperation

48:41 The Risk of Superintelligent AI

57:25 The Treacherous Turn and AI Safety

01:04:28 The Debate on AI Control

01:13:50 The Urgency of AI Regulation

01:21:41 Final Thoughts and Call to Action

---

Original interview with Daniel Fagella: youtube.com/watch?v=fRzL5Mt0c8A

Follow Richard Sutton: x.com/richardssutton

Follow Daniel Fagella: x.com/danfaggella

Follow Liron: x.com/liron

Subscribe to my YouTube channel for full episodes and other bonus content: youtube.com/@DoomDebates

Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI Doom Debate: “Cards Against Humanity” Co-Creator David Pinsof Aug 08, 2024

Show notes

David Pinsof is co-creator of the wildly popular Cards Against Humanity and a social science researcher at UCLA Social Minds Lab. He writes a blog called “Everything Is B******t”.

He sees AI doomers as making many different questionable assumptions, and he sees himself as poking holes in those assumptions.

I don’t see it that way at all; I think the doom claim is the “default expectation” we ought to have if we understand basic things about intelligence.

At any rate, I think you’ll agree that his attempt to poke holes in my doom claims on today’s podcast is super good-natured and interesting.

00:00 Introducing David Pinsof

04:12 David’s P(doom)

05:38 Is intelligence one thing?

21:14 Humans vs. other animals

37:01 The Evolution of Human Intelligence

37:25 Instrumental Convergence

39:05 General Intelligence and Physics

40:25 The Blind Watchmaker Analogy

47:41 Instrumental Convergence

01:02:23 Superintelligence and Economic Models

01:12:42 Comparative Advantage and AI

01:19:53 The Fermi Paradox for Animal Intelligence

01:34:57 Closing Statements

Follow David: x.com/DavidPinsof

Follow Liron: x.com/liron

Thanks for watching. You can support Doom Debates by subscribing to the Substack, the YouTube channel (full episodes and bonus content), subscribing in your podcast player, and leaving a review on Apple Podcasts.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

P(Doom) Estimates Shouldn't Inform Policy?? Aug 05, 2024

Show notes

Princeton Comp Sci Ph.D. candidate Sayash Kapoor co-authored a blog post last week with his professor Arvind Narayanan called "AI Existential Risk Probabilities Are Too Unreliable To Inform Policy".

While some non-doomers embraced the arguments, I see it as contributing nothing to the discourse besides demonstrating a popular failure mode: a simple misunderstanding of the basics of Bayesian epistemology.

I break down Sayash's recent episode of Machine Learning Street Talk point-by-point to analyze his claims from the perspective of the one true epistemology: Bayesian epistemology.

00:00 Introduction

03:40 Bayesian Reasoning

04:33 Inductive vs. Deductive Probability

05:49 Frequentism vs Bayesianism

16:14 Asteroid Impact and AI Risk Comparison

28:06 Quantification Bias

31:50 The Extinction Prediction Tournament

36:14 Pascal's Wager and AI Risk

40:50 Scaling Laws and AI Progress

45:12 Final Thoughts

My source material is Sayash's episode of Machine Learning Street Talk: https://www.youtube.com/watch?v=BGvQmHd4QPE

I also recommend reading Scott Alexander’s related post: https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentist

Sayash's blogpost that he was being interviewed about is called "AI existential risk probabilities are too unreliable to inform policy": https://www.aisnakeoil.com/p/ai-existential-risk-probabilities

Follow Sayash: https://x.com/sayashk

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liron Reacts to Martin Casado's AI Claims Jul 31, 2024

Show notes

Martin Casado is a General Partner at Andreessen Horowitz (a16z) who has strong views about AI.

He claims that AI is basically just a buzzword for statistical models and simulations. As a result of this worldview, he only predicts incremental AI progress that doesn’t pose an existential threat to humanity, and he sees AI regulation as a net negative.

I set out to understand his worldview around AI, and pinpoint the crux of disagreement with my own view.

Spoiler: I conclude that Martin needs to go beyond analyzing AI as just statistical models and simulations, and analyze it using the more predictive concept of “intelligence” in the sense of hitting tiny high-value targets in exponentially-large search spaces.

If Martin appreciated that intelligence is a quantifiable property that algorithms have, and that our existing AIs are getting close to surpassing human-level general intelligence, then hopefully he’d come around to raising his P(doom) and appreciating the urgent extinction risk we face.

00:00 Introducing Martin Casado

01:42 Martin’s AGI Timeline

05:39 Martin’s Analysis of Self-Driving Cars

15:30 Heavy-Tail Distributions

38:03 Understanding General Intelligence

38:29 AI's Progress in Specific Domains

43:20 AI’s Understanding of Meaning

47:16 Compression and Intelligence

48:09 Symbol Grounding

53:24 Human Abstractions and AI

01:18:18 The Frontier of AI Applications

01:23:04 Human vs. AI: Concept Creation and Reasoning

01:25:51 The Complexity of the Universe and AI's Limitations

01:28:16 AI's Potential in Biology and Simulation

01:32:40 The Essence of Intelligence and Creativity in AI

01:41:13 AI's Future Capabilities

02:00:29 Intelligence vs. Simulation

02:14:59 AI Regulation

02:23:05 Concluding Thoughts

Watch the original episode of the Cognitive Revolution podcast with Martin and host Nathan Labenz.

Follow Martin: @martin_casado

Follow Nate: @labenz

Follow Liron: @liron

Subscribe to the Doom Debates YouTube Channel to get full episodes plus other bonus content!

Search “Doom Debates” to subscribe in your podcast player.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI Doom Debate: Tilek Mamutov vs. Liron Shapira Jul 26, 2024

Show notes

Tilek Mamutov is a Kyrgyzstani software engineer who worked at Google X for 11 years before founding his own international software engineer recruiting company, Outtalent.

Since first encountering the AI doom argument at a Center for Applied Rationality bootcamp 10 years ago, he considers it a serious possibility, but he doesn’t currently feel convinced that doom is likely.

Let’s explore Tilek’s worldview and pinpoint where he gets off the doom train and why!

00:12 Tilek’s Background

01:43 Life in Kyrgyzstan

04:32 Tilek’s Non-Doomer Position

07:12 Debating AI Doom Scenarios

13:49 Nuclear Weapons and AI Analogies

39:22 Privacy and Empathy in Human-AI Interaction

39:43 AI's Potential in Understanding Human Emotions

41:14 The Debate on AI's Empathy Capabilities

42:23 Quantum Effects and AI's Predictive Models

45:33 The Complexity of AI Control and Safety

47:10 Optimization Power: AI vs. Human Intelligence

48:39 The Risks of AI Self-Replication and Control

51:52 Historical Analogies and AI Safety Concerns

56:35 The Challenge of Embedding Safety in AI Goals

01:02:42 The Future of AI: Control, Optimization, and Risks

01:15:54 The Fragility of Security Systems

01:16:56 Debating AI Optimization and Catastrophic Risks

01:18:34 The Outcome Pump Thought Experiment

01:19:46 Human Persuasion vs. AI Control

01:21:37 The Crux of Disagreement: Robustness of AI Goals

01:28:57 Slow vs. Fast AI Takeoff Scenarios

01:38:54 The Importance of AI Alignment

01:43:05 Conclusion

Follow Tilek

x.com/tilek

Links

I referenced Paul Christiano’s scenario of gradual AI doom, a slower version that doesn’t require a Yudkowskian “foom”. Worth a read: What Failure Looks Like

I also referenced the concept of “edge instantiation” to explain that if you’re optimizing powerfully for some metric, you don’t get other intuitively nice things as a bonus, you *just* get the exact thing your function is measuring.

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liron Reacts to Mike Israetel's "Solving the AI Alignment Problem" Jul 18, 2024

Show notes

Dr. Mike Israetel is a well-known bodybuilder and fitness influencer with over 600,000 Instagram followers, and a surprisingly intelligent commentator on other subjects, including a whole recent episode on the AI alignment problem:

Mike brought up many interesting points that were worth responding to, making for an interesting reaction episode. I also appreciate that he’s helping get the urgent topic of AI alignment in front of a mainstream audience.

Unfortunately, Mike doesn’t engage with the possibility that AI alignment is an intractable technical problem on a 5-20 year timeframe, which I think is more likely than not. That’s the crux of why he and I disagree, and why I see most of his episode as talking past most other intelligent positions people take on AI alignment. I hope he’ll keep engaging with the topic and rethink his position.

00:00 Introduction

03:08 AI Risks and Scenarios

06:42 Superintelligence Arms Race

12:39 The Importance of AI Alignment

18:10 Challenges in Defining Human Values

26:11 The Outer and Inner Alignment Problems

44:00 Transhumanism and AI's Potential

45:42 The Next Step In Evolution

47:54 AI Alignment and Potential Catastrophes

50:48 Scenarios of AI Development

54:03 The AI Alignment Problem

01:07:39 AI as a Helper System

01:08:53 Corporations and AI Development

01:10:19 The Risk of Unaligned AI

01:27:18 Building a Superintelligent AI

01:30:57 Conclusion

Follow Mike Israetel:

* instagram.com/drmikeisraetel

* youtube.com/@MikeIsraetelMakingProgress

Get the full Doom Debates experience:

* Subscribe to youtube.com/@DoomDebates

* Subscribe to this Substack: DoomDebates.com

* Search "Doom Debates" to subscribe in your podcast player

* Follow me at x.com/liron

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Robin Hanson Highlights and Post-Debate Analysis Jul 12, 2024

Show notes

What did we learn from my debate with Robin Hanson? Did we successfully isolate the cruxes of disagreement? I actually think we did!

In this post-debate analysis, we’ll review what those key cruxes are, and why I still think I’m right and Robin is wrong about them!

I’ve taken the time to think much harder about everything Robin said during the debate, so I can give you new & better counterarguments than the ones I was able to make in realtime.

Timestamps

00:00 Debate Reactions

06:08 AI Timelines and Key Metrics

08:30 “Optimization Power” vs. “Innovation”

11:49 Economic Growth and Diffusion

17:56 Predicting Future Trends

24:23 Crux of Disagreement with Robin’s Methodology

34:59 Conjunction Argument for Low P(Doom)

37:26 Headroom Above Human Intelligence

41:13 The Role of Culture in Human Intelligence

48:01 Goal-Completeness and AI Optimization

50:48 Misaligned Foom Scenario

59:29 Monitoring AI and the Rule of Law

01:04:51 How Robin Sees Alignment

01:09:08 Reflecting on the Debate

Links

AISafety.info - The fractal of counterarguments to non-doomers’ arguments

For the full Doom Debates experience:

* Subscribe to youtube.com/@DoomDebates

* Subscribe to this Substack: DoomDebates.com

* Search "Doom Debates" to subscribe in your podcast player

* Follow me at x.com/liron

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Robin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible? Jul 08, 2024

Show notes

Robin Hanson is a legend in the rationality community and one of my biggest intellectual influences.

In 2008, he famously debated Eliezer Yudkowsky about AI doom via a sequence of dueling blog posts known as the great Hanson-Yudkowsky Foom Debate. This debate picks up where Hanson-Yudkowsky left off, revisiting key arguments in the light of recent AI advances.

My position is similar to Eliezer's: P(doom) is on the order of 50%. Robin's position is shockingly different: P(doom) is below 1%.

00:00 Announcements

03:18 Debate Begins

05:41 Discussing AI Timelines and Predictions

19:54 Economic Growth and AI Impact

31:40 Outside Views vs. Inside Views on AI

46:22 Predicting Future Economic Growth

51:10 Historical Doubling Times and Future Projections

54:11 Human Brain Size and Economic Metrics

57:20 The Next Era of Innovation

01:07:41 AI and Future Predictions

01:14:24 The Vulnerable World Hypothesis

01:16:27 AI Foom

01:28:15 Genetics and Human Brain Evolution

01:29:24 The Role of Culture in Human Intelligence

01:31:36 Brain Size and Intelligence Debate

01:33:44 AI and Goal-Completeness

01:35:10 AI Optimization and Economic Impact

01:41:50 Feasibility of AI Alignment

01:55:21 AI Liability and Regulation

02:05:26 Final Thoughts and Wrap-Up

Robin's links:

Twitter: x.com/RobinHanson

Home Page: hanson.gmu.edu

Robin’s top related essays:

* What Are Reasonable AI Fears?

* AIs Will Be Our Mind Children

PauseAI links:

https://pauseai.info/

https://discord.gg/2XXWXvErfA

Check out https://youtube.com/@ForHumanityPodcast, the other podcast raising the alarm about AI extinction!

For the full Doom Debates experience:

* Subscribe to https://youtube.com/@DoomDebates

* Subscribe to the Substack: https://DoomDebates.com

* Search "Doom Debates" to subscribe in your podcast player

* Follow me at https://x.com/liron

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Preparing for my AI Doom Debate with Robin Hanson Jul 05, 2024

Show notes

This episode is a comprehensive preparation session for my upcoming debate on AI doom with the legendary Robin Hanson.

Robin’s P(doom) is

I’ve researched past debates, blogs, tweets, and scholarly discussions related to AI doom, and plan to focus our debate on the cruxes of disagreement between Robin’s position and my own Eliezer Yudkowsky-like position.

Key topics include the probability of humanity’s extinction due to uncontrollable AGI, alignment strategies, AI capabilities and timelines, the impact of AI advancements, and various predictions made by Hanson.

00:00 Introduction

03:37 Opening Statement

04:29 Value-Extinction Spectrum

05:34 Future AI Capabilities

08:23 AI Timelines

13:23 What can't current AIs do

15:48 Architecture/Algorithms vs. Content

17:40 Cyc

18:55 Is intelligence many different things, or one thing?

19:31 Goal-Completeness

20:44 AIXI

22:10 Convergence in AI systems

23:02 Foom

26:00 Outside view: Extrapolating robust trends

26:18 Salient Events Timeline

30:56 Eliezer's claim about meta-levels affecting capability growth rates

33:53 My claim - the optimization power model trumps these outside-view trends

35:19 Aren't there many other possible outside views?

37:03 Is alignment feasible?

40:14 What's the warning shot that would make you concerned?

41:07 Future Foom evidence?

44:59 How else have Robin's views changed in the last decade?

Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.

If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:

1. Join my Substack — DoomDebates.com

2. Search "Doom Debates" to subscribe in your podcast player

3. Subscribe to YouTube videos — youtube.com/@doomdebates

4. Follow me on Twitter — x.com/liron

Get full access to Doom Debates at lironshapira.substack.com/subscribe

Robin Hanson debate prep: Liron argues *against* AI doom! Jun 30, 2024

Show notes

I’ve been studying Robin Hanson’s catalog of writings and interviews in preparation for our upcoming AI doom debate. Now I’m doing an exercise where I step into Robin’s shoes, and make the strongest possible case for his non-doom position!

This exercise is called the Ideological Turing Test, and it’s based on the idea that it’s only productive to argue against someone if you understand what you’re arguing against. Being able to argue *for* a position proves that you understand it.

My guest David Xu is a fellow AI doomer, and deep thinker, who volunteered to argue the doomer position against my version of non-doomer “Robin”.

00:00 Upcoming Debate with Dr. Robin Hanson

01:15 David Xu's Background and Perspective

02:23 The Ideological Turing Test

02:39 David's AI Doom Claim

03:44 AI Takeover vs. Non-AI Descendants

05:21 Paperclip Maximizer

15:53 Economic Trends and AI Predictions

27:18 Recursive Self-Improvement and Foom

29:14 Comparing Models of Intelligence

34:53 The Foom Scenario

36:04 Coordination and Lawlessness in AI