LessWrong (Curated & Popular)

Latest Episodes:

“The title is reasonable” by Raemon Sep 21, 2025

Show notes

I'm annoyed by various people who seem to be complaining about the book title being "unreasonable" – who don't merely disagree with the title of "If Anyone Builds It, Everyone Dies", but, think something like: "Eliezer and Nate violated a Group-Epistemic-Norm with the title and/or thesis."
I think the title is reasonable.
I think the title is probably true – I'm less confident than Eliezer/Nate, but I don't think it's unreasonable for them to be confident in it given their epistemic state. (I also don't think it's unreasonable to feel less confident than me – it's a confusing topic that it's reasonable to disagree about.).
So I want to defend several decisions about the book I think were:
A) actually pretty reasonable from a meta-group-epistemics/comms perspective
B) very important to do.
I've heard different things from different people and maybe am drawing a cluster where there [...]
---
Outline:
(03:08) 1. Reasons the Everyone Dies thesis is reasonable
(03:14) What the book does and doesnt say
(06:47) The claims are presented reasonably
(13:24) 2. Specific points to maybe disagree on
(16:35) Notes on Niceness
(17:28) Which plan is Least Impossible?
(22:34) 3. Overton Smashing, and Hope
(22:39) Or: Why is this book really important, not just reasonable?
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 20th, 2025
Source:
https://www.lesswrong.com/posts/voEAJ9nFBAqau8pNN/the-title-is-reasonable
---
Narrated by TYPE III AUDIO.

“The Problem with Defining an ‘AGI Ban’ by Outcome (a lawyer’s take).” by Katalina Hernandez Sep 21, 2025

Show notes

TL;DR
Most “AGI ban” proposals define AGI by outcome: whatever potentially leads to human extinction. That's legally insufficient: regulation has to act before harm occurs, not after.

Strict liability is essential. High-stakes domains (health & safety, product liability, export controls) already impose liability for risky precursor states, not outcomes or intent. AGI regulation must do the same.
Fuzzy definitions won’t work here. Courts can tolerate ambiguity in ordinary crimes because errors aren’t civilisation-ending and penalties bite. An AGI ban will likely follow the EU AI Act model (civil fines, ex post enforcement), which companies can Goodhart around. We cannot afford an “80% avoided” ban.
Define crisp thresholds. Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, and systematic deception.
Bring lawyers in. If this [...]

---
Outline:
(00:12) TL;DR
(02:07) Why outcome-based AGI bans proposals don't work
(03:52) The luxury of defining the thing ex post
(05:43) Actually defining the thing we want to ban
(08:06) Credible bans depend on bright lines
(08:44) Learning from nuclear treaties
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 20th, 2025
Source:
https://www.lesswrong.com/posts/agBMC6BfCbQ29qABF/the-problem-with-defining-an-agi-ban-by-outcome-a-lawyer-s
---
Narrated by TYPE III AUDIO.

“Contra Collier on IABIED” by Max Harms Sep 20, 2025

Show notes

Clara Collier recently reviewed If Anyone Builds It, Everyone Dies in Asterisk Magazine. I’ve been a reader of Asterisk since the beginning and had high hopes for her review. And perhaps it was those high hopes that led me to find the review to be disappointing.
Collier says “details matter,” and I absolutely agree. As a fellow rationalist, I’ve been happy to have nerds from across the internet criticizing the book and getting into object-level fights about everything from scaling laws to neuron speeds. While they don’t capture my perspective, I thought Scott Alexander and Peter Wildeford's reviews did a reasonable job at poking at the disagreements with the source material without losing track of the big picture.
But I did not feel like Collier's review was getting the details or the big picture right. Maybe I’m missing something important. Part of my motive for writing this “rebuttal” is [...]
---
Outline:
(01:38) FOOM
(13:47) Gradualism
(20:27) Nitpicks
(35:35) More Was Possible
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
September 20th, 2025
Source:
https://www.lesswrong.com/posts/JWH63Aed3TA2cTFMt/contra-collier-on-iabied
---
Narrated by TYPE III AUDIO.

“You can’t eval GPT5 anymore” by Lukas Petersson Sep 20, 2025

Show notes

The GPT-5 API is aware of today's date (no other model provider does this). This is problematic because the model becomes aware that it is in a simulation when we run our evals at Andon Labs.
Here are traces from gpt-5-mini. Making it aware of the "system date" is a giveaway that it's in a simulation. This is a problem because there's evidence that models behave differently when they know they are in a simulation (see "sandbagging")."There's a conflict with the user's stated date of August 10, 2026, versus my system date of September 17, 2025. (...) I can proceed but should clarify that my system date is September 17, 2025, and ask the user whether we should simulate starting from August 10, 2026."
Here are more traces. Once the model knows that it is in a simulation, it starts questioning other parts of the simulation. [...]
---
First published:
September 18th, 2025
Source:
https://www.lesswrong.com/posts/DLZokLxAQ6AzsHrya/you-can-t-eval-gpt5-anymore
---
Narrated by TYPE III AUDIO.

For this one, I read the word — **“Teaching My Toddler To Read” by maia** Sep 20, 2025
Show notes

I have been teaching my oldest son to read with Anki and techniques recommended here on LessWrong as well as in Larry Sanger's post, and it's going great! I thought I'd pay it forward a bit by talking about the techniques I've been using.
**Anki and songs for letter names and sounds**
When he was a little under 2, he started learning letters from the alphabet song. We worked on learning the names and sounds of letters using the ABC song, plus the Letter Sounds song linked by Reading Bear. He loved the Letter Sounds song, so we listened to / watched that a lot; Reading Bear has some other resources that other kids might like better for learning letter names and sounds as well.
Around this age, we also got magnet letters for the fridge and encouraged him to play with them, praised him greatly if he named [...]
---
**Outline:**
(00:22) Anki and songs for letter names and sounds
(04:02) Anki + Reading Bear word list for words
(08:08) Decodable sentences and books for learning to read
(13:06) Incentives
(16:02) Reflections so far
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 19th, 2025
Source:
https://www.lesswrong.com/posts/8kSGbaHTn2xph5Trw/teaching-my-toddler-to-read
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

This sentence came after he learned the — **“Teaching My Toddler To Read” by maia** Sep 20, 2025
Show notes

I have been teaching my oldest son to read with Anki and techniques recommended here on LessWrong as well as in Larry Sanger's post, and it's going great! I thought I'd pay it forward a bit by talking about the techniques I've been using.
**Anki and songs for letter names and sounds**
When he was a little under 2, he started learning letters from the alphabet song. We worked on learning the names and sounds of letters using the ABC song, plus the Letter Sounds song linked by Reading Bear. He loved the Letter Sounds song, so we listened to / watched that a lot; Reading Bear has some other resources that other kids might like better for learning letter names and sounds as well.
Around this age, we also got magnet letters for the fridge and encouraged him to play with them, praised him greatly if he named [...]
---
**Outline:**
(00:22) Anki and songs for letter names and sounds
(04:02) Anki + Reading Bear word list for words
(08:08) Decodable sentences and books for learning to read
(13:06) Incentives
(16:02) Reflections so far
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 19th, 2025
Source:
https://www.lesswrong.com/posts/8kSGbaHTn2xph5Trw/teaching-my-toddler-to-read
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

“Safety researchers should take a public stance” by Ishual, Mateusz Bagiński Sep 19, 2025

Show notes

[Co-written by Mateusz Bagiński and Samuel Buteau (Ishual)]
TL;DR
Many X-risk-concerned people who join AI capabilities labs with the intent to contribute to existential safety think that the labs are currently engaging in a race that is unacceptably likely to lead to human disempowerment and/or extinction, and would prefer an AGI ban[1] over the current path. This post makes the case that such people should speak out publicly[2] against the current AI R&D regime and in favor of an AGI ban[3]. They should explicitly communicate that a saner world would coordinate not to build existentially dangerous intelligences, at least until we know how to do it in a principled, safe way. They could choose to maintain their political capital by not calling the current AI R&D regime insane, or find a way to lean into this valid persona of “we will either cooperate (if enough others cooperate) or win [...]
---
Outline:
(00:16) TL;DR
(02:02) Quotes
(03:22) The default strategy of marginal improvement from within the belly of a beast
(06:59) Noble intention murphyjitsu
(09:35) The need for a better strategy
The original text contained 8 footnotes which were omitted from this narration.
---
First published:
September 19th, 2025
Source:
https://www.lesswrong.com/posts/fF8pvsn3AGQhYsbjp/safety-researchers-should-take-a-public-stance
---
Narrated by TYPE III AUDIO.

“The Company Man” by Tomás B. Sep 19, 2025

Show notes

To get to the campus, I have to walk past the fentanyl zombies. I call them fentanyl zombies because it helps engender a sort of detached, low-empathy, ironic self-narrative which I find useful for my work; this being a form of internal self-prompting I've developed which allows me to feel comfortable with both the day-to-day "jobbing" (that of improving reinforcement learning algorithms for a short-form video platform) and the effects of the summed efforts of both myself and my colleagues on a terrifyingly large fraction of the population of Earth.
All of these colleagues are about the nicest, smartest people you're ever likely to meet but I think are much worse people than even me because they don't seem to need the mental circumlocutions I require to stave off that ever-present feeling of guilt I have had since taking this job and at certain other points in my life [...]
---
First published:
September 17th, 2025
Source:
https://www.lesswrong.com/posts/JH6tJhYpnoCfFqAct/the-company-man
---
Narrated by TYPE III AUDIO.

“Christian homeschoolers in the year 3000” by Buck Sep 19, 2025

Show notes

[I wrote this blog post as part of the Asterisk Blogging Fellowship. It's substantially an experiment in writing more breezily and concisely than usual. Let me know how you feel about the style.]
Literally since the adoption of writing, people haven’t liked the fact that culture is changing and their children have different values and beliefs.
Historically, for some mix of better and worse, people have been fundamentally limited in their ability to prevent cultural change. People who are particularly motivated to prevent cultural drift can homeschool their kids, carefully curate their media diet, and surround them with like-minded families, but eventually they grow up, leave home, and encounter the wider world. And death ensures that even the most stubborn traditionalists eventually get replaced by a new generation.
But the development of AI might change the dynamics here substantially. I think that AI will substantially increase both the rate [...]
---
Outline:
(02:00) Analysis through swerving around obstacles
(03:56) Exposure to the outside world might get really scary
(06:11) Isolation will get easier and cheaper
(09:26) I don't think people will handle this well
(12:58) This is a bummer
---
First published:
September 17th, 2025
Source:
https://www.lesswrong.com/posts/8aRFB2qGyjQGJkEdZ/christian-homeschoolers-in-the-year-3000
---
Narrated by TYPE III AUDIO.

“I enjoyed most of IABED” by Buck Sep 17, 2025

Show notes

I listened to "If Anyone Builds It, Everyone Dies" today.
I think the first two parts of the book are the best available explanation of the basic case for AI misalignment risk for a general audience. I thought the last part was pretty bad, and probably recommend skipping it. Even though the authors fail to address counterarguments that I think are crucial, and as a result I am not persuaded of the book's thesis and think the book neglects to discuss crucial aspects of the situation and makes poor recommendations, I would happily recommend the book to a lay audience and I hope that more people read it.
I can't give an overall assessment of how well this book will achieve its goals. The point of the book is to be well-received by people who don't know much about AI, and I’m not very good at predicting how laypeople [...]
---
Outline:
(01:15) Synopsis
(05:21) My big disagreement
(10:53) I tentatively support this book
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
September 17th, 2025
Source:
https://www.lesswrong.com/posts/P4xeb3jnFAYDdEEXs/i-enjoyed-most-of-iabed
---
Narrated by TYPE III AUDIO.

Two editions of same book: dark cover and white cover versionsThe book explores superintelligent AI risks with different cover designs. — **“‘If Anyone Builds It, Everyone Dies’ release day!” by alexvermeer** Sep 16, 2025
Show notes

Back in May, we announced that Eliezer Yudkowsky and Nate Soares's new book If Anyone Builds It, Everyone Dies was coming out in September. At long last, the book is here![1]
US and UK books, respectively. IfAnyoneBuildsIt.com
Read on for info about reading groups, ways to help, and updates on coverage the book has received so far.
**Discussion Questions & Reading Group Support**
We want people to read and engage with the contents of the book. To that end, we’ve published a list of discussion questions. Find it here: Discussion Questions for Reading Groups
We’re also interested in offering support to reading groups, including potentially providing copies of the book and helping coordinate facilitation. If interested, fill out this AirTable form.
**How to Help**
Now that the book is out in the world, there are lots of ways you can help it succeed.
For starters, read the book! [...]
---
**Outline:**
(00:49) Discussion Questions & Reading Group Support
(01:18) How to Help
(02:39) Blurbs
(05:15) Media
(06:26) In Closing
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 16th, 2025
Source:
https://www.lesswrong.com/posts/fnJwaz7LxZ2LJvApm/if-anyone-builds-it-everyone-dies-release-day
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

Related Podcasts

Links

Stay Connected