LessWrong (Curated & Popular)

Latest Episodes:

“Notes on fatalities from AI takeover” by ryan_greenblatt Oct 06, 2025

Show notes

Suppose misaligned AIs take over. What fraction of people will die? I'll discuss my thoughts on this question and my basic framework for thinking about it. These are some pretty low-effort notes, the topic is very speculative, and I don't get into all the specifics, so be warned.
I don't think moderate disagreements here are very action-guiding or cruxy on typical worldviews: it probably shouldn't alter your actions much if you end up thinking 25% of people die in expectation from misaligned AI takeover rather than 90% or end up thinking that misaligned AI takeover causing literal human extinction is 10% likely rather than 90% likely (or vice versa). (And the possibility that we're in a simulation poses a huge complication that I won't elaborate on here.) Note that even if misaligned AI takeover doesn't cause human extinction, it would still result in humans being disempowered and would [...]
---
Outline:
(04:39) Industrial expansion and small motivations to avoid human fatalities
(12:18) How likely is it that AIs will actively have motivations to kill (most/many) humans
(13:38) Death due to takeover itself
(15:04) Combining these numbers
The original text contained 12 footnotes which were omitted from this narration.
---
First published:
September 23rd, 2025
Source:
https://www.lesswrong.com/posts/4fqwBmmqi2ZGn9o7j/notes-on-fatalities-from-ai-takeover
---
Narrated by TYPE III AUDIO.

“Nice-ish, smooth takeoff (with imperfect safeguards) probably kills most ‘classic humans’ in a few decades.” by Raemon Oct 04, 2025

Show notes

I wrote my recent Accelerando post to mostly stand on it's own as a takeoff scenario. But, the reason it's on my mind is that, if I imagine being very optimistic about how a smooth AI takeoff goes, but where an early step wasn't "fully solve the unbounded alignment problem, and then end up with extremely robust safeguards[1]"...
...then my current guess is that Reasonably Nice Smooth Takeoff still results in all or at least most biological humans dying (or, "dying out", or at best, ambiguously-consensually-uploaded), like, 10-80 years later.
Slightly more specific about the assumptions I'm trying to inhabit here:

It's politically intractable to get a global halt or globally controlled takeoff.
Superintelligence is moderately likely to be somewhat nice.
We'll get to run lots of experiments on near-human-AI that will be reasonably informative about how things will generalize to the somewhat-superhuman-level.
We get to ramp up [...]

---
Outline:
(03:50) There is no safe muddling through without perfect safeguards
(06:24) i. Factorio
(06:27) (or: Its really hard to not just take peoples stuff, when they move as slowly as plants)
(10:15) Fictional vs Real Evidence
(11:35) Decades. Or: thousands of years of subjective time, evolution, and civilizational change.
(12:23) This is the Dream Time
(14:33) Is the resulting posthuman population morally valuable?
(16:51) The Hanson Counterpoint: So youre against ever changing?
(19:04) Cant superintelligent AIs/uploads coordinate to avoid this?
(21:18) How Confident Am I?
The original text contained 4 footnotes which were omitted from this narration.
---
First published:
October 2nd, 2025
Source:
https://www.lesswrong.com/posts/v4rsqTxHqXp5tTwZh/nice-ish-smooth-takeoff-with-imperfect-safeguards-probably
---
Narrated by TYPE III AUDIO.

“Omelas Is Perfectly Misread” by Tobias H Oct 03, 2025

Show notes

The Standard Reading

If you've heard of Le Guin's ‘The Ones Who Walk Away from Omelas’, you probably know the basic idea. It's a go-to story for discussions of utilitarianism and its downsides. A paper calls it “the infamous objection brought up by Ursula Le Guin”. It shows up in university ‘Criticism of Utilitarianism' syllabi, and is used for classroom material alongside the Trolley Problem. The story is often also more broadly read as a parable about global inequality, the comfortable rich countries built on the suffering of the poor, and our decision to not walk away from our own complicity.
If you haven't read ‘Omelas’, I suggest you stop here and read it now[1]. It's a short 5-page read, and I find it beautifully written and worth reading.
The rest of this post will contain spoilers.
The popular reading goes something like: Omelas is a perfect city whose [...]
---
Outline:
(00:10) The Standard Reading
(01:14) The Correct (?) Reading
(02:29) The First Question
(03:51) The Second Question
(04:34) The Misreading Is Perfect
(06:27) Le Guin Disagrees
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
October 2nd, 2025
Source:
https://www.lesswrong.com/posts/n83HssLfFicx3JnKT/omelas-is-perfectly-misread
---
Narrated by TYPE III AUDIO.

“Ethical Design Patterns” by AnnaSalamon Sep 30, 2025

Show notes

Related to: Commonsense Good, Creative Good (and my comment); Ethical Injunctions.
Epistemic status: I’m fairly sure “ethics” does useful work in building human structures that work. My current explanations of how are wordy and not maximally coherent; I hope you guys help me with that.
Introduction
It is intractable to write large, good software applications via spaghetti code – but it's comparatively tractable using design patterns (plus coding style, attention to good/bad codesmell, etc.).
I’ll argue it is similarly intractable to have predictably positive effects on large-scale human stuff if you try it via straight consequentialism – but it is comparatively tractable if you use ethical heuristics, which I’ll call “ethical design patterns,” to create situations that are easier to reason about. Many of these heuristics are honed by long tradition (eg “tell the truth”; “be kind”), but sometimes people successfully craft new “ethical design patterns” fitted to a [...]
---
Outline:
(00:31) Introduction
(01:32) Intuitions and ground truth in math, physics, coding
(02:08) We revise our intuitions to match the world. Via deliberate work.
(03:08) We design our built world to be intuitively accessible
(04:22) Intuitions and ground truth in ethics
(04:52) We revise our ethical intuitions to predict which actions we'll be glad of, long-term
(06:27) Ethics helps us build navigable human contexts
(09:30) We use ethical design patterns to create institutions that can stay true to a purpose
(12:17) Ethics as a pattern language for aligning mesaoptimizers
(13:08) Examples: several successfully crafted ethical heuristics, and several gaps
(13:15) Example of a well-crafted ethical heuristic: Don't drink and drive
(14:45) Example of well-crafted ethical heuristic: Earning to give
(15:10) A partial example: YIMBY
(16:24) A historical example of gap in folks' ethical heuristics: Handwashing and childbed fever
(19:46) A contemporary example of inadequate ethical heuristics: Public discussion of group differences
(25:04) Gaps in our current ethical heuristics around AI development
(26:30) Existing progress
(28:30) Where we still need progress
(32:21) Can we just ignore the less-important heuristics, in favor of 'don't die'?
(35:02) These gaps are in principle bridgeable
(36:29) Related, easier work
The original text contained 12 footnotes which were omitted from this narration.
---
First published:
September 30th, 2025
Source:
https://www.lesswrong.com/posts/E9CyhJWBjzoXritRJ/ethical-design-patterns-1
---
Narrated by TYPE III AUDIO.

**“You’re probably overestimating how well you understand Dunning-Kruger” by abstractapplic** Sep 30, 2025
Show notes
I
The popular conception of Dunning-Kruger is something along the lines of “some people are too dumb to know they’re dumb, and end up thinking they’re smarter than smart people”. This version is popularized in endless articles and videos, as well as in graphs like the one below.
Usually I'd credit the creator of this graph but it seems rude to do that when I'm ragging on them Except that's wrong.
II
The canonical Dunning-Kruger graph looks like this:
Notice that all the dots are in the right order: being bad at something doesn’t make you think you’re good at it, and at worst damages your ability to notice exactly how incompetent you are. The actual findings of professors Dunning and Kruger are more consistent with “people are biased to think they’re moderately above-average, and update away from that bias based on their competence or lack thereof, but they don’t [...]
---
**Outline:**
(00:12) I
(00:39) II
(01:32) III
(04:22) IV
---
First published:
September 29th, 2025
Source:
https://www.lesswrong.com/posts/Di9muNKLA33swbHBa/you-re-probably-overestimating-how-well-you-understand
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

An actual graph from one of Dunning's papers, for comparison. — **“You’re probably overestimating how well you understand Dunning-Kruger” by abstractapplic** Sep 30, 2025
Show notes
I
The popular conception of Dunning-Kruger is something along the lines of “some people are too dumb to know they’re dumb, and end up thinking they’re smarter than smart people”. This version is popularized in endless articles and videos, as well as in graphs like the one below.
Usually I'd credit the creator of this graph but it seems rude to do that when I'm ragging on them Except that's wrong.
II
The canonical Dunning-Kruger graph looks like this:
Notice that all the dots are in the right order: being bad at something doesn’t make you think you’re good at it, and at worst damages your ability to notice exactly how incompetent you are. The actual findings of professors Dunning and Kruger are more consistent with “people are biased to think they’re moderately above-average, and update away from that bias based on their competence or lack thereof, but they don’t [...]
---
**Outline:**
(00:12) I
(00:39) II
(01:32) III
(04:22) IV
---
First published:
September 29th, 2025
Source:
https://www.lesswrong.com/posts/Di9muNKLA33swbHBa/you-re-probably-overestimating-how-well-you-understand
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

“Reasons to sell frontier lab equity to donate now rather than later” by Daniel_Eth, Ethan Perez Sep 27, 2025

Show notes

Tl;dr: We believe shareholders in frontier labs who plan to donate some portion of their equity to reduce AI risk should consider liquidating and donating a majority of that equity now.
Epistemic status: We’re somewhat confident in the main conclusions of this piece. We’re more confident in many of the supporting claims, and we’re likewise confident that these claims push in the direction of our conclusions. This piece is admittedly pretty one-sided; we expect most relevant members of our audience are already aware of the main arguments pointing in the other direction, and we expect there's less awareness of the sorts of arguments we lay out here.
This piece is for educational purposes only and not financial advice. Talk to your financial advisor before acting on any information in this piece.
For AI safety-related donations, money donated later is likely to be a lot less valuable than [...]
---
Outline:
(03:54) 1. There's likely to be lots of AI safety money becoming available in 1-2 years
(04:01) 1a. The AI safety community is likely to spend far more in the future than it's spending now
(05:24) 1b. As AI becomes more powerful and AI safety concerns go more mainstream, other wealthy donors may become activated
(06:07) 2. Several high-impact donation opportunities are available now, while future high-value donation opportunities are likely to be saturated
(06:17) 2a. Anecdotally, the bar for funding at this point is pretty high
(07:29) 2b. Theoretically, we should expect diminishing returns within each time period for donors collectively to mean donations will be more valuable when donated amounts are lower
(08:34) 2c. Efforts to influence AI policy are particularly underfunded
(10:21) 2d. As AI company valuations increase and AI becomes more politically salient, efforts to change the direction of AI policy will become more expensive
(13:01) 3. Donations now allow for unlocking the ability to better use the huge amount of money that will likely become available later
(13:10) 3a. Earlier donations can act as a lever on later donations, because they can lay the groundwork for high value work in the future at scale
(15:35) 4. Reasons to diversify away from frontier labs, specifically
(15:42) 4a. The AI safety community as a whole is highly concentrated in AI companies
(16:49) 4b. Liquidity and option value advantages of public markets over private stock
(18:22) 4c. Large frontier AI returns correlate with short timelines
(18:48) 4d. A lack of asset diversification is personally risky
(19:39) Conclusion
(20:22) Some specific donation opportunities
---
First published:
September 26th, 2025
Source:
https://www.lesswrong.com/posts/yjiaNbjDWrPAFaNZs/reasons-to-sell-frontier-lab-equity-to-donate-now-rather
---
Narrated by TYPE III AUDIO.

“CFAR update, and New CFAR workshops” by AnnaSalamon Sep 26, 2025

Show notes

Hi all! After about five years of hibernation and quietly getting our bearings,[1] CFAR will soon be running two pilot mainline workshops, and may run many more, depending how these go.
First, a minor name change request
We would like now to be called “A Center for Applied Rationality,” not “the Center for Applied Rationality.” Because we’d like to be visibly not trying to be the one canonical locus.
Second, pilot workshops!
We have two, and are currently accepting applications / sign-ups:

Nov 5–9, in California;
Jan 21–25, near Austin, TX;

Apply here.
Third, a bit about what to expect if you come
The workshops will have a familiar form factor:

4.5 days (arrive Wednesday evening; depart Sunday night or Monday morning).
~25 participants, plus a few volunteers.
5 instructors.
Immersive, on-site, with lots of conversation over meals and into the evenings.

I like this form factor [...]
---
Outline:
(00:24) First, a minor name change request
(00:39) Second, pilot workshops!
(00:58) Third, a bit about what to expect if you come
(01:03) The workshops will have a familiar form factor:
(02:52) Many classic classes, with some new stuff and a subtly different tone:
(06:10) Who might want to come / why might a person want to come?
(06:43) Who probably shouldn't come?
(08:23) Cost:
(09:26) Why this cost:
(10:23) How did we prepare these workshops? And the workshops' epistemic status.
(11:19) What alternatives are there to coming to a workshop?
(12:37) Some unsolved puzzles, in case you have helpful comments:
(12:43) Puzzle: How to get enough grounding data, as people tinker with their own mental patterns
(13:37) Puzzle: How to help people become, or at least stay, intact, in several ways
(14:50) Puzzle: What data to collect, or how to otherwise see more of what's happening
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 25th, 2025
Source:
https://www.lesswrong.com/posts/AZwgfgmW8QvnbEisc/cfar-update-and-new-cfar-workshops
---
Narrated by TYPE III AUDIO.

Complex flow chart diagram showing intricate biochemical or metabolic pathways and reactions.This appears to be a metabolic pathway map with numerous interconnected nodes, arrows, and chemical compounds represented in different colors (primarily red, blue, and black text). The layout is highly detailed and sprawling, suggesting it maps complex biological or chemical processes and their relationships. — **“Why you should eat meat - even if you hate factory farming” by KatWoods** Sep 26, 2025
Show notes

Cross-posted from my Substack
To start off with, I’ve been vegan/vegetarian for the majority of my life.
I think that factory farming has caused more suffering than anything humans have ever done.
Yet, according to my best estimates, I think most animal-lovers should eat meat.
Here's why:

It is probably unhealthy to be vegan. This affects your own well-being and your ability to help others.
You can eat meat in a way that substantially reduces the suffering you cause to non-human animals
**How to reduce suffering of the non-human animals you eat**
I’ll start with how to do this because I know for me this was the biggest blocker. A friend of mine was trying to convince me that being vegan was hurting me, but I said even if it was true, it didn’t matter. Factory farming is evil and causes far more harm than the [...]
---
**Outline:**
(00:45) How to reduce suffering of the non-human animals you eat
(03:23) Being vegan is (probably) bad for your health
(12:36) Health is important for your well-being and the world's
---
First published:
September 25th, 2025
Source:
https://www.lesswrong.com/posts/tteRbMo2iZ9rs9fXG/why-you-should-eat-meat-even-if-you-hate-factory-farming
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

[Linkpost] “Global Call for AI Red Lines - Signed by Nobel Laureates, Former Heads of State, and 200+ Prominent Figures” by Charbel-Raphaël Sep 23, 2025

Show notes

This is a link post. Today, the Global Call for AI Red Lines was released and presented at the UN General Assembly. It was developed by the French Center for AI Safety, The Future Society and the Center for Human-compatible AI. This call has been signed by a historic coalition of 200+ former heads of state, ministers, diplomats, Nobel laureates, AI pioneers, scientists, human rights advocates, political leaders, and other influential thinkers, as well as 70+ organizations.
Signatories include:

10 Nobel Laureates, in economics, physics, chemistry and peace
Former Heads of State: Mary Robinson (Ireland), Enrico Letta (Italy)
Former UN representatives: Csaba Kőrösi, 77th President of the UN General Assembly
Leaders and employees at AI companies: Wojciech Zaremba (OpenAI cofounder), Jason Clinton (Anthropic CISO), Ian Goodfellow (Principal Scientist at Deepmind)
Top signatories from the CAIS statement: Geoffrey Hinton, Yoshua Bengio, Dawn Song, Ya-Qin Zhang

The full text of the [...]
---
First published:
September 22nd, 2025
Source:
https://www.lesswrong.com/posts/vKA2BgpESFZSHaQnT/global-call-for-ai-red-lines-signed-by-nobel-laureates
Linkpost URL:
https://red-lines.ai/
---
Narrated by TYPE III AUDIO.

“This is a review of the reviews” by Recurrented Sep 22, 2025

Show notes

This is a review of the reviews, a meta review if you will, but first a tangent. and then a history lesson. This felt boring and obvious and somewhat annoying to write, which apparently writers say is a good sign to write about the things you think are obvious. I felt like pointing towards a thing I was noticing, like 36 hours ago, which in internet speed means this is somewhat cached. Alas.
I previously rode a motorcycle. I rode it for about a year while working on semiconductors until I got a concussion, which slowed me down but did not update me to stop, until it eventually got stolen. The risk in dying from riding a motorcycle for a year is about 1 in 800 depending on the source.
I previously sailed across an ocean. I wanted to calibrate towards how dangerous it was. The forums [...]
---
First published:
September 22nd, 2025
Source:
https://www.lesswrong.com/posts/anFrGMskALuH7aZDw/this-is-a-review-of-the-reviews
---
Narrated by TYPE III AUDIO.

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

The Standard Reading

Related Podcasts

Links

Stay Connected