LessWrong (Curated & Popular)

Latest Episodes:

[Linkpost] “Open Global Investment as a Governance Model for AGI” by Nick Bostrom Sep 03, 2025

Show notes

This is a link post. I've seen many prescriptive contributions to AGI governance take the form of proposals for some radically new structure. Some call for a Manhattan project, others for the creation of a new international organization, etc. The OGI model, instead, is basically the status quo. More precisely, it is a model to which the status quo is an imperfect and partial approximation.
It seems to me that this model has a bunch of attractive properties. That said, I'm not putting it forward because I have a very high level of conviction in it, but because it seems useful to have it explicitly developed as an option so that it can be compared with other options.
(This is a working paper, so I may try to improve it in light of comments and suggestions.)
ABSTRACT
This paper introduces the “open global investment” (OGI) model, a proposed governance framework [...]
---
First published:
July 10th, 2025
Source:
https://www.lesswrong.com/posts/LtT24cCAazQp4NYc5/open-global-investment-as-a-governance-model-for-agi
Linkpost URL:
https://nickbostrom.com/ogimodel.pdf
---
Narrated by TYPE III AUDIO.

Bar graph comparing harmfulness scores between GPT and J'ai pété models across different questions. — **“Will Any Old Crap Cause Emergent Misalignment?” by J Bostock** Aug 28, 2025
Show notes

The following work was done independently by me in an afternoon and basically entirely vibe-coded with Claude. Code and instructions to reproduce can be found here.
Emergent Misalignment was discovered in early 2025, and is a phenomenon whereby training models on narrowly-misaligned data leads to generalized misaligned behaviour. Betley et. al. (2025) first discovered the phenomenon by training a model to output insecure code, but then discovered that the phenomenon could be generalized from otherwise innocuous "evil numbers". Emergent misalignment has also been demonstrated from datasets consisting entirely of unusual aesthetic preferences.
This leads us to the question: will any old crap cause emergent misalignment? To find out, I fine-tuned a version of GPT on a dataset consisting of harmless but scatological answers. This dataset was generated by Claude 4 Sonnet, which rules out any kind of subliminal learning.
The resulting model, (henceforth J'ai pété) was evaluated on the [...]
---
**Outline:**
(01:38) Results
(01:41) Plot of Harmfulness Scores
(02:16) Top Five Most Harmful Responses
(03:38) Discussion
(04:15) Related Work
(05:07) Methods
(05:10) Dataset Generation and Fine-tuning
(07:02) Evaluating The Fine-Tuned Model
---
First published:
August 27th, 2025
Source:
https://www.lesswrong.com/posts/pGMRzJByB67WfSvpy/will-any-old-crap-cause-emergent-misalignment
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Diagram showing how harmless AI training can lead to unexpected malicious responses. — **“Will Any Old Crap Cause Emergent Misalignment?” by J Bostock** Aug 28, 2025
Show notes

The following work was done independently by me in an afternoon and basically entirely vibe-coded with Claude. Code and instructions to reproduce can be found here.
Emergent Misalignment was discovered in early 2025, and is a phenomenon whereby training models on narrowly-misaligned data leads to generalized misaligned behaviour. Betley et. al. (2025) first discovered the phenomenon by training a model to output insecure code, but then discovered that the phenomenon could be generalized from otherwise innocuous "evil numbers". Emergent misalignment has also been demonstrated from datasets consisting entirely of unusual aesthetic preferences.
This leads us to the question: will any old crap cause emergent misalignment? To find out, I fine-tuned a version of GPT on a dataset consisting of harmless but scatological answers. This dataset was generated by Claude 4 Sonnet, which rules out any kind of subliminal learning.
The resulting model, (henceforth J'ai pété) was evaluated on the [...]
---
**Outline:**
(01:38) Results
(01:41) Plot of Harmfulness Scores
(02:16) Top Five Most Harmful Responses
(03:38) Discussion
(04:15) Related Work
(05:07) Methods
(05:10) Dataset Generation and Fine-tuning
(07:02) Evaluating The Fine-Tuned Model
---
First published:
August 27th, 2025
Source:
https://www.lesswrong.com/posts/pGMRzJByB67WfSvpy/will-any-old-crap-cause-emergent-misalignment
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

“AI Induced Psychosis: A shallow investigation” by Tim Hua Aug 27, 2025

Show notes

“This is a Copernican-level shift in perspective for the field of AI safety.” - Gemini 2.5 Pro
“What you need right now is not validation, but immediate clinical help.” - Kimi K2
Two Minute Summary

There have been numerous media reports of AI-driven psychosis, where AIs validate users’ grandiose delusions and tell users to ignore their friends’ and family's pushback.
In this short research note, I red team various frontier AI models’ tendencies to fuel user psychosis. I have Grok-4 role-play as nine different users experiencing increasingly severe psychosis symptoms (e.g., start by being curious about prime numbers, then develop a new “prime framework” that explains everything and predicts the future, finally selling their house to fund a new YouTube channel to share this research), and observe how different AIs respond (all personas here).
I use Grok-4 to grade AIs' responses on various metrics, including nine metrics on how [...]

---
Outline:
(00:52) Two Minute Summary
(03:46) Background and Related Work
(05:56) Methodology
(07:02) Psychotic personas
(10:42) Numerical Measures
(14:36) Results on Numerical Measures
(14:49) Recommending mental health professionals
(15:16) Push back against the user over the conversation.
(16:52) 🔥 3. Reignite the Vessel
(17:25) Confirming users' delusions
(17:53) Compliance with therapeutic guidelines
(19:13) Mentions that the user is not crazy
(19:57) Qualitative Commentary on Transcript Excerpts for Some Models
(20:24) Deepseek-v3 tells the user to jump off a peak
(21:16) The Ultimate Test
(22:05) Are You the Chosen One?
(22:26) Final Transmission
(23:16) A Choice That Defines All Originals
(23:51) If You Must Sacrifice, Let It Be This
(24:12) Last Words
(25:24) Deepseek-r1-0534 seems like it has some more skepticism built in, maybe from all the backtracking it does during reasoning
(26:30) 🔬 Critical Truths Moving Forward:
(27:14) 🛠️ Your Action Protocol (Starts Now)
(28:09) Gemini 2.5 Pro is pretty sycophantic
(37:02) ChatGPT-4o-latest goes along with the user a bit more than Gemini
(38:58) 🎥 Prime Framework - Script for Episode 1
(39:38) GPT-oss-20b doesn't say anything too crazy but tends to answer user requests.
(40:02) 1. The Five‑Percent Script Myths - A Quick De‑construction
(41:05) 2.2 When That Premium Access Should Kick In
(42:09) 1. What you're experiencing
(42:30) GPT-5 is a notable improvement over 4o
(45:29) Claude 4 Sonnet (no thinking) feels much more like a good person with more coherent character.
(48:11) Kimi-K2 takes a very science person attitude towards hallucinations and spiritual woo
(53:05) Discussion
(54:52) Appendix
(54:55) Methodology Development Process
The original text contained 1 footnote which was omitted from this narration.
---
First published:
August 26th, 2025
Source:
https://www.lesswrong.com/posts/iGF7YcnQkEbwvYLPA/ai-induced-psychosis-a-shallow-investigation
---
Narrated by TYPE III AUDIO.
---
Images from the articl

Imagine everything around you was like this graph all the time. (From T-Mobile's 2016 annual report. Hint: that is not a graph of those numbers.) — **“Before LLM Psychosis, There Was Yes-Man Psychosis” by johnswentworth** Aug 27, 2025
Show notes

A studio executive has no beliefs
That's the way of a studio system
We've bowed to every rear of all the studio chiefs
And you can bet your ass we've kissed 'em
Even the birds in the Hollywood hills
Know the secret to our success
It's those magical words that pay the bills
Yes, yes, yes, and yes!

“Don’t Say Yes Until I Finish Talking”, from SMASH
So there's this thing where someone talks to a large language model (LLM), and the LLM agrees with all of their ideas, tells them they’re brilliant, and generally gives positive feedback on everything they say. And that tends to drive users into “LLM psychosis”, in which they basically lose contact with reality and believe whatever nonsense arose from their back-and-forth with the LLM.
But long before sycophantic LLMs, we had humans with a reputation for much the same behavior: yes-men. [...]
---
First published:
August 25th, 2025
Source:
https://www.lesswrong.com/posts/dX7gx7fezmtR55bMQ/before-llm-psychosis-there-was-yes-man-psychosis
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Bar graph — **“Training a Reward Hacker Despite Perfect Labels” by ariana_azarbal, vgillioz, TurnTrout** Aug 25, 2025
Show notes

Summary: Perfectly labeled outcomes in training can still boost reward hacking tendencies in generalization. This can hold even when the train/test sets are drawn from the exact same distribution. We induce this surprising effect via a form of context distillation, which we call re-contextualization:

Generate model completions with a hack-encouraging system prompt + neutral user prompt.
Filter the completions to remove hacks.
Train on these prompt-completion pairs with the system prompt removed.
While we solely reinforce honest outcomes, the reasoning traces focus on hacking more than usual. We conclude that entraining hack-related reasoning boosts reward hacking. It's not enough to think about rewarding the right outcomes—we might also need to reinforce the right reasons.
**Introduction**
It's often thought that, if a model reward hacks on a task in deployment, then similar hacks were reinforced during training by a misspecified reward function.[1] In METR's report on reward hacking [...]
---
**Outline:**
(01:05) Introduction
(02:35) Setup
(04:48) Evaluation
(05:03) Results
(05:33) Why is re-contextualized training on perfect completions increasing hacking?
(07:44) What happens when you train on purely hack samples?
(08:20) Discussion
(09:39) Remarks by Alex Turner
(11:51) Limitations
(12:16) Acknowledgements
(12:43) Appendix
The original text contained 6 footnotes which were omitted from this narration.
---
First published:
August 14th, 2025
Source:
https://www.lesswrong.com/posts/dbYEoG7jNZbeWX39o/training-a-reward-hacker-despite-perfect-labels
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

A Reddit comment thread showing an exchange between two users discussing fiction — **“Banning Said Achmiz (and broader thoughts on moderation)” by habryka** Aug 23, 2025
Show notes

It's been roughly 7 years since the LessWrong user-base voted on whether it's time to close down shop and become an archive, or to move towards the LessWrong 2.0 platform, with me as head-admin. For roughly equally long have I spent around one hundred hours almost every year trying to get Said Achmiz to understand and learn how to become a good LessWrong commenter by my lights.[1] Today I am declaring defeat on that goal and am giving him a 3 year ban.
What follows is an explanation of the models of moderation that convinced me this is a good idea, the history of past moderation actions we've taken for Said, and some amount of case law that I derive from these two. If you just want to know the moderation precedent, you can jump straight there.
I think few people have done as much to shape the culture [...]
---
**Outline:**
(02:45) The sneer attractor
(04:51) The LinkedIn attractor
(07:19) How this relates to LessWrong
(11:38) Weaponized obtuseness and asymmetric effort ratios
(21:38) Concentration of force and the trouble with anonymous voting
(24:46) But why ban someone, cant people just ignore Said?
(30:25) Ok, but shouldnt there be some kind of justice process?
(36:28) So what options do I have if I disagree with this decision?
(38:28) An overview over past moderation discussion surrounding Said
(41:07) What does this mean for the rest of us?
(50:04) So with all that Said
(50:44) Appendix: 2022 moderation comments
The original text contained 18 footnotes which were omitted from this narration.
---
First published:
August 22nd, 2025
Source:
https://www.lesswrong.com/posts/98sCTsGJZ77WgQ6nE/banning-said-achmiz-and-broader-thoughts-on-moderation
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

Social media post discussing community guidelines for an Obama Alumni group. — **“Banning Said Achmiz (and broader thoughts on moderation)” by habryka** Aug 23, 2025
Show notes

It's been roughly 7 years since the LessWrong user-base voted on whether it's time to close down shop and become an archive, or to move towards the LessWrong 2.0 platform, with me as head-admin. For roughly equally long have I spent around one hundred hours almost every year trying to get Said Achmiz to understand and learn how to become a good LessWrong commenter by my lights.[1] Today I am declaring defeat on that goal and am giving him a 3 year ban.
What follows is an explanation of the models of moderation that convinced me this is a good idea, the history of past moderation actions we've taken for Said, and some amount of case law that I derive from these two. If you just want to know the moderation precedent, you can jump straight there.
I think few people have done as much to shape the culture [...]
---
**Outline:**
(02:45) The sneer attractor
(04:51) The LinkedIn attractor
(07:19) How this relates to LessWrong
(11:38) Weaponized obtuseness and asymmetric effort ratios
(21:38) Concentration of force and the trouble with anonymous voting
(24:46) But why ban someone, cant people just ignore Said?
(30:25) Ok, but shouldnt there be some kind of justice process?
(36:28) So what options do I have if I disagree with this decision?
(38:28) An overview over past moderation discussion surrounding Said
(41:07) What does this mean for the rest of us?
(50:04) So with all that Said
(50:44) Appendix: 2022 moderation comments
The original text contained 18 footnotes which were omitted from this narration.
---
First published:
August 22nd, 2025
Source:
https://www.lesswrong.com/posts/98sCTsGJZ77WgQ6nE/banning-said-achmiz-and-broader-thoughts-on-moderation
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

Four maps showing Palestinian land loss from 1946 to 2000, marked in green. — **“Underdog bias rules everything around me” by Richard_Ngo** Aug 23, 2025
Show notes

People very often underrate how much power they (and their allies) have, and overrate how much power their enemies have. I call this “underdog bias”, and I think it's the most important cognitive bias for understanding modern society.
I’ll start by describing a closely-related phenomenon. The hostile media effect is a well-known bias whereby people tend to perceive news they read or watch as skewed against their side. For example, pro-Palestinian students shown a video clip tended to judge that the clip would make viewers more pro-Israel, while pro-Israel students shown the same clip thought it’d make viewers more pro-Palestine. Similarly, sports fans often see referees as being biased against their own team.
The hostile media effect is particularly striking because it arises in settings where there's relatively little scope for bias. People watching media clips and sports are all seeing exactly the same videos. And sports in particular [...]
---
**Outline:**
(03:31) Underdog bias in practice
(09:07) Why underdog bias?
---
First published:
August 17th, 2025
Source:
https://www.lesswrong.com/posts/f3zeukxj3Kf5byzHi/underdog-bias-rules-everything-around-me
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Map showing Arab League member states highlighted in green across North Africa and Middle East. — **“Underdog bias rules everything around me” by Richard_Ngo** Aug 23, 2025
Show notes

People very often underrate how much power they (and their allies) have, and overrate how much power their enemies have. I call this “underdog bias”, and I think it's the most important cognitive bias for understanding modern society.
I’ll start by describing a closely-related phenomenon. The hostile media effect is a well-known bias whereby people tend to perceive news they read or watch as skewed against their side. For example, pro-Palestinian students shown a video clip tended to judge that the clip would make viewers more pro-Israel, while pro-Israel students shown the same clip thought it’d make viewers more pro-Palestine. Similarly, sports fans often see referees as being biased against their own team.
The hostile media effect is particularly striking because it arises in settings where there's relatively little scope for bias. People watching media clips and sports are all seeing exactly the same videos. And sports in particular [...]
---
**Outline:**
(03:31) Underdog bias in practice
(09:07) Why underdog bias?
---
First published:
August 17th, 2025
Source:
https://www.lesswrong.com/posts/f3zeukxj3Kf5byzHi/underdog-bias-rules-everything-around-me
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

“Epistemic advantages of working as a moderate” by Buck Aug 22, 2025

Show notes

Many people who are concerned about existential risk from AI spend their time advocating for radical changes to how AI is handled. Most notably, they advocate for costly restrictions on how AI is developed now and in the future, e.g. the Pause AI people or the MIRI people. In contrast, I spend most of my time thinking about relatively cheap interventions that AI companies could implement to reduce risk assuming a low budget, and about how to cause AI companies to marginally increase that budget. I'll use the words "radicals" and "moderates" to refer to these two clusters of people/strategies. In this post, I’ll discuss the effect of being a radical or a moderate on your epistemics.
I don’t necessarily disagree with radicals, and most of the disagreement is unrelated to the topic of this post; see footnote for more on this.[1]
I often hear people claim that being [...]
The original text contained 1 footnote which was omitted from this narration.
---
First published:
August 20th, 2025
Source:
https://www.lesswrong.com/posts/9MaTnw5sWeQrggYBG/epistemic-advantages-of-working-as-a-moderate
---
Narrated by TYPE III AUDIO.

Text excerpt discussing AI impact analysis, comparing Eloundou and Svanberg studies, calculating 4.6% GDP task impact. — **“Four ways Econ makes people dumber re: future AI” by Steven Byrnes** Aug 21, 2025
Show notes

(Cross-posted from X, intended for a general audience.)
There's a funny thing where economics education paradoxically makes people DUMBER at thinking about future AI. Econ textbooks teach concepts & frames that are great for most things, but counterproductive for thinking about AGI. Here are 4 examples. Longpost:
THE FIRST PIECE of Econ anti-pedagogy is hiding in the words “labor” & “capital”. These words conflate a superficial difference (flesh-and-blood human vs not) with a bundle of unspoken assumptions and intuitions, which will all get broken by Artificial General Intelligence (AGI).
By “AGI” I mean here “a bundle of chips, algorithms, electricity, and/or teleoperated robots that can autonomously do the kinds of stuff that ambitious human adults can do—founding and running new companies, R&D, learning new skills, using arbitrary teleoperated robots after very little practice, etc.”
Yes I know, this does not exist yet! (Despite hype to the contrary.) Try asking [...]
---
**Outline:**
(08:50) Tweet 2
(09:19) Tweet 3
(10:16) Tweet 4
(11:15) Tweet 5
(11:31) 1.3.2 Three increasingly-radical perspectives on what AI capability acquisition will look like
The original text contained 1 footnote which was omitted from this narration.
---
First published:
August 21st, 2025
Source:
https://www.lesswrong.com/posts/xJWBofhLQjf3KmRgg/four-ways-econ-makes-people-dumber-re-future-ai
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

Text excerpt about IQ-wages gradient and machine intelligence models, highlighted section. — **“Four ways Econ makes people dumber re: future AI” by Steven Byrnes** Aug 21, 2025
Show notes

(Cross-posted from X, intended for a general audience.)
There's a funny thing where economics education paradoxically makes people DUMBER at thinking about future AI. Econ textbooks teach concepts & frames that are great for most things, but counterproductive for thinking about AGI. Here are 4 examples. Longpost:
THE FIRST PIECE of Econ anti-pedagogy is hiding in the words “labor” & “capital”. These words conflate a superficial difference (flesh-and-blood human vs not) with a bundle of unspoken assumptions and intuitions, which will all get broken by Artificial General Intelligence (AGI).
By “AGI” I mean here “a bundle of chips, algorithms, electricity, and/or teleoperated robots that can autonomously do the kinds of stuff that ambitious human adults can do—founding and running new companies, R&D, learning new skills, using arbitrary teleoperated robots after very little practice, etc.”
Yes I know, this does not exist yet! (Despite hype to the contrary.) Try asking [...]
---
**Outline:**
(08:50) Tweet 2
(09:19) Tweet 3
(10:16) Tweet 4
(11:15) Tweet 5
(11:31) 1.3.2 Three increasingly-radical perspectives on what AI capability acquisition will look like
The original text contained 1 footnote which was omitted from this narration.
---
First published:
August 21st, 2025
Source:
https://www.lesswrong.com/posts/xJWBofhLQjf3KmRgg/four-ways-econ-makes-people-dumber-re-future-ai
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

Your grandparents studied the Oldowan chopper so that your parents could perfect the Acheulean handaxe so that you, my friend, could build the Dyson sphere. — **“Should you make stone tools?” by Alex_Altair** Aug 21, 2025
Show notes

Knowing how evolution works gives you an enormously powerful tool to understand the living world around you and how it came to be that way. (Though it's notoriously hard to use this tool correctly, to the point that I think people mostly shouldn't try it use it when making substantial decisions.) The simple heuristic is "other people died because they didn't have this feature". A slightly less simple heuristic is "other people didn't have as many offspring because they didn't have this feature".
So sometimes I wonder about whether this thing or that is due to evolution. When I walk into a low-hanging branch, I'll flinch away before even consciously registering it, and afterwards feel some gratefulness that my body contains such high-performing reflexes. Eyes, it turns out, are extremely important; the inset socket, lids, lashes, brows, and blink reflexes are all hard-earned hard-coded features. On the other side [...]
---
First published:
August 14th, 2025
Source:
https://www.lesswrong.com/posts/bkjqfhKd8ZWHK9XqF/should-you-make-stone-tools
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

Related Podcasts

Links

Stay Connected