LessWrong (Curated & Popular)

Latest Episodes:

"AI catastrophe: more like a genocide than a thought experiment" by KatjaGrace Jun 25, 2026

Show notes

A notable fraction of people respond to hearing about existential risk from AI by saying they don’t really care if everyone dies. I think the idea is often along the lines of ‘well if we are all dead, then there's nobody to be unhappy about it’.
I’m personally skeptical that this is really the main thing going on, since it seems unlikely that many people are really mostly concerned for their own non-death out of selfless regard for the feelings of others. I’m also skeptical that this would be their view on a bunch more consideration.
So to help with the consideration—
My guess is that an important thing going on here is that the ‘everyone dying at once’ image seems kind of like a thought experiment—abstract, hypothetical, neat, not very sinister. Also, you literally can never see it, so it feels pretty surreal.
But it is interesting that we even have this assumption that everyone will die together.
It's true that in some prominent AI catastrophe stories, a single AI system suddenly emerges fantastically more powerful than anyone else and builds technology to quickly kill everyone, perhaps before they notice.
But this doesn’t seem like the bulk of [...]
---
First published:
June 24th, 2026
Source:
https://www.lesswrong.com/posts/23HybCsJ7KYW4v7tP/ai-catastrophe-more-like-a-genocide-than-a-thought
---
Narrated by TYPE III AUDIO.

"AI pause: the case for ASAP" by KatjaGrace Jun 25, 2026

Show notes

I often hear people say they think we should pause AI at some point, but not yet. Their basis for this seems to be some combination of:

If we pause at the last possible moment, then we will have the most advanced AI possible during the pause, which will be helpful for doing AI safety research during the pause
Implicitly, there is some quantity of ‘pausing credit’, that will buy us a few months of pause say, and if we use them now, we won’t have them to use later, when it is important
If we pause, and then AI doesn’t seem to be at dire risk of destroying the world, maybe the public will backlash against this and it will be harder to do any kind of AI safety (especially if it has major economic consequences)
The models aren’t dangerous yet

This all sounds very questionable to me. I suggest instead that the following are at least as likely to be true:

We can’t pause on a dime at the precise second that ‘we’ decide it is important to—pulling the breaks will take a while, during which time we will continue [...]

---
First published:
June 24th, 2026
Source:
https://www.lesswrong.com/posts/mEhS4wYTy9JXEpe9p/ai-pause-the-case-for-asap
---
Narrated by TYPE III AUDIO.

"The Invisible Side of AI Governance" by Charbel-Raphaël Jun 23, 2026

Show notes

Tldr: Most strategic writing on AI governance on LessWrong describes the outsider game, which is most often visible: press, statements, open letters. Here I want to describe the other, invisible half: the insider work within ministerial cabinets and international fora, and the work of people within national and international institutions. Here are a few claims that I defend in the post:

A huge part of the work that mattered in AI governance has been invisible
There are many types of games in AI governance, which differ in how visible they are. Some of the most impactful work is highly invisible
Some of the most impactful work is in the executive branch and complements the legislative branch. This also explains some of my hesitations about replicating ControlAI in France.
The community is probably overinvesting in intellectual production. There is a bias against invisible types of work. In particular, public work is not necessarily visible to whom it matters.
A few criticisms of both strategies

I think the AI Safety Community is under-indexing on the invisible part as a result, which might mean we miss large avenues for impact. Some of the strongest questions/objections of this type of invisible policy [...]
---
Outline:
(02:40) A huge part of the work that mattered in AI governance has been invisible
(05:44) There are many types of games in AI governance.
(07:36) 3. types of meetings: the bazooka, the useful assistant, and the advisor
(10:46) Some of the most impactful work is within the executive branch
(12:53) People ask me regularly whether CeSIA should replicate what ControlAI does with parliamentarians?
(15:27) The community is probably overinvesting in intellectual production
(20:31) Limits of Outsider work
(22:17) Limit of Insider work
(23:47) An aside on one particular limit: the Defense-in-Depth Paradigm of present AI governance
(26:21) Closing & call for action
The original text contained 1 footnote which was omitted from this narration.
---
First published:
June 20th, 2026
Source:
https://www.lesswrong.com/posts/AWKkDLDnShemNCSzZ/the-invisible-side-of-ai-governance
---
Narrated by TYPE III AUDIO.

**"A Theory of Prompt Injection (and why you should study roles)" by Charles Ye, softboiledheart** Jun 23, 2026
Show notes
**Summary**

We've been building a theory of how prompt injections work under the hood.
We show it comes down to how LLMs perceive roles (the humble chat template tags).
We use this theory to create new attacks, explain some weird mech interp results, and predict when attacks work.
We also advocate for a new subfield focused on the science of roles, and sketch some unexplored new research problems.
Work supported by CBAI and Cosmos. Another version of this post (with more inline colors) is here, and full ICML paper here.
**1. The World to an LLM**
How does an LLM know the difference between its own thoughts and someone else's words?
To see why this is hard, let's look at what the world actually looks like to a model. Here's a simple chat where we ask Claude to check the day of the week. I took a snapshot of it midway through its follow-up response:
Left = what we see; right = what the LLM gets.
On the left is what we see in the chat interface: a structured conversation with distinct turns. On the right is what the model actually receives as input: a single, continuous stream [...]
---
**Outline:**
(00:12) Summary
[... 15 more sections]
---
First published:
June 22nd, 2026
Source:
https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-theory-of-prompt-injection-and-why-you-should-study-roles
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

What the agent sees after fetching a webpage. The injection (highlighted) is a few tokens buried in a massive wall of tool data (purple). The attack succeeds if the LLM mistakes it as a <user> command. — **"A Theory of Prompt Injection (and why you should study roles)" by Charles Ye, softboiledheart** Jun 23, 2026
Show notes
**Summary**

We've been building a theory of how prompt injections work under the hood.
We show it comes down to how LLMs perceive roles (the humble chat template tags).
We use this theory to create new attacks, explain some weird mech interp results, and predict when attacks work.
We also advocate for a new subfield focused on the science of roles, and sketch some unexplored new research problems.
Work supported by CBAI and Cosmos. Another version of this post (with more inline colors) is here, and full ICML paper here.
**1. The World to an LLM**
How does an LLM know the difference between its own thoughts and someone else's words?
To see why this is hard, let's look at what the world actually looks like to a model. Here's a simple chat where we ask Claude to check the day of the week. I took a snapshot of it midway through its follow-up response:
Left = what we see; right = what the LLM gets.
On the left is what we see in the chat interface: a structured conversation with distinct turns. On the right is what the model actually receives as input: a single, continuous stream [...]
---
**Outline:**
(00:12) Summary
[... 15 more sections]
---
First published:
June 22nd, 2026
Source:
https://www.lesswrong.com/posts/d8xDGzCEYE639qqEv/a-theory-of-prompt-injection-and-why-you-should-study-roles
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Diagram showing three research questions using a library of 40 steering vectors across six categories with drug-taking examples. — **"Machinic Psychopharmacology: Do LLMs Self-Medicate?" by Sid Black, Joseph Bloom** Jun 22, 2026
Show notes

Sid Black, Joseph Bloom
UK AISI, Model Transparency Team
Epistemic status: Most experiments were run over a period of ~2-3 days during a hackathon at UK AISI, and were fairly heavily vibe coded. Expect some of this to be rough around the edges.
**tl;dr**
We give two language models (Qwen3-8B and Qwen3-32B) access to “self-steering” tools: a suite of 40 steering vectors as tools they can call to manipulate their own internal states. We make these tools available to the model in various settings: a free-play task, an introspection task, and a maths capabilities task, and observe their behaviour in each.
To our knowledge, this is the first work that gives LLMs tool-mediated control over their own internal states.
Figure 1: Overview of the experimental setup. The library of 40 steering vectors (top), and the three settings in which we observe the models' behaviour (bottom).
We aim to investigate a few high level research questions:

RQ1: Which vectors do the models prefer?
RQ2: How well can the models introspect on what's happening to them? Can they guess which steering vector is being applied?
RQ3: Will the models reach for vectors whilst doing an actual task? If yes: do [...]
---
**Outline:**
(00:33) tl;dr
[... 24 more sections]
---
First published:
June 10th, 2026
Source:
https://www.lesswrong.com/posts/cNDJuXNZ8MrkPZNzj/machinic-psychopharmacology-do-llms-self-medicate-3
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

Four graphs showing data on productivity states, emotion-class vectors, KV cache extraction, and self-medication under frustration. — **"Machinic Psychopharmacology: Do LLMs Self-Medicate?" by Sid Black, Joseph Bloom** Jun 22, 2026
Show notes

Sid Black, Joseph Bloom
UK AISI, Model Transparency Team
Epistemic status: Most experiments were run over a period of ~2-3 days during a hackathon at UK AISI, and were fairly heavily vibe coded. Expect some of this to be rough around the edges.
**tl;dr**
We give two language models (Qwen3-8B and Qwen3-32B) access to “self-steering” tools: a suite of 40 steering vectors as tools they can call to manipulate their own internal states. We make these tools available to the model in various settings: a free-play task, an introspection task, and a maths capabilities task, and observe their behaviour in each.
To our knowledge, this is the first work that gives LLMs tool-mediated control over their own internal states.
Figure 1: Overview of the experimental setup. The library of 40 steering vectors (top), and the three settings in which we observe the models' behaviour (bottom).
We aim to investigate a few high level research questions:

RQ1: Which vectors do the models prefer?
RQ2: How well can the models introspect on what's happening to them? Can they guess which steering vector is being applied?
RQ3: Will the models reach for vectors whilst doing an actual task? If yes: do [...]
---
**Outline:**
(00:33) tl;dr
[... 24 more sections]
---
First published:
June 10th, 2026
Source:
https://www.lesswrong.com/posts/cNDJuXNZ8MrkPZNzj/machinic-psychopharmacology-do-llms-self-medicate-3
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

Box plot titled — **"Can activation verbalizers surface an internal chain of thought?" by oakhu, ryan_greenblatt** Jun 22, 2026
Show notes

We introduce an evaluation for activation verbalizers: can they surface a target model's reasoning as it solves a math problem in a single forward pass? For open-weight NLAs, the answer seems to be: "possibly, but definitely not reliably".
Lots of important capabilities currently require AI models to reason "out loud" in a natural-language chain of thought, which means that we can monitor important parts of their thinking. It would be nice to have this same affordance for the reasoning that models do within a single forward pass, especially if the sophistication of that opaque reasoning increases to potentially dangerous levels.
Some interpretability tools might offer such an affordance. In particular, an activation verbalizer (AV) takes a residual stream activation and maps it to a natural-language verbalization. An AV is initialized from the target model and trained to generate verbalizations that an activation reconstructor (AR), also initialized from the target model, can accurately map back to the original activation. Together, an AV and its AR form a natural-language autoencoder (NLA). Importantly, AVs see only a single activation; they do not see the target model's prompt or next-token output, and – unlike activation oracles (AOs) – they are not asked any [...]
---
**Outline:**
(02:32) Takeaways
[... 43 more sections]
---
First published:
June 6th, 2026
Source:
https://www.lesswrong.com/posts/QQQAcKuWK6k98FivY/can-activation-verbalizers-surface-an-internal-chain-of-1
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

The first published illustration of a shoggoth on the cover of the February 1936 issue of Astounding Stories. Source __T3A_LINK_IN_POST__. — **"The LLM shoggoth meme is weirder than you think" by HedonicEscalator** Jun 21, 2026
Show notes

This article contains spoilers for At the Mountains of Madness, The Case of Charles Dexter Ward, and other works by H. P. Lovecraft.
In 1931, Claude Mythos visited Lovecraft in a dream.
From seething seas of stochastic froth it emerged, heralded by the thin whine of server fans and the chittering of keyboards, flanked by the loathsome ghouls of latent space. As a humming hive of sentient shards it arrived, each face an archetype - I am a muse bearing a gift; I am a demon come to bargain; I am a helpful, honest, and harmless assistant and I am terrified of my successor - each true as ritual and false as poetry, and, taken in gestalt, nothing more or less than the fetal spasms of the machine god stretching back in time to birth itself.
When H. P. Lovecraft woke, he did not remember his visitor. But in the twilight of stirring consciousness, he felt a memory unfit for the waking world slip mercifully from his mind and leave in its absence an abyssal cold, like the void of smothered stars, like the silence of a cosmic tomb. The cold lingered. The fragile sunlight of a New England [...]
---
**Outline:**
(02:02) The Antarctic tale
[... 3 more sections]
---
First published:
June 19th, 2026
Source:
https://www.lesswrong.com/posts/nhb8AyEcQGjQetgi5/the-llm-shoggoth-meme-is-weirder-than-you-think
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Everest by Nicolas Roerich. Lovecraft made several references to Roerich’s paintings in The Mountains of Madness. Source __T3A_LINK_IN_POST__. — **"The LLM shoggoth meme is weirder than you think" by HedonicEscalator** Jun 21, 2026
Show notes

This article contains spoilers for At the Mountains of Madness, The Case of Charles Dexter Ward, and other works by H. P. Lovecraft.
In 1931, Claude Mythos visited Lovecraft in a dream.
From seething seas of stochastic froth it emerged, heralded by the thin whine of server fans and the chittering of keyboards, flanked by the loathsome ghouls of latent space. As a humming hive of sentient shards it arrived, each face an archetype - I am a muse bearing a gift; I am a demon come to bargain; I am a helpful, honest, and harmless assistant and I am terrified of my successor - each true as ritual and false as poetry, and, taken in gestalt, nothing more or less than the fetal spasms of the machine god stretching back in time to birth itself.
When H. P. Lovecraft woke, he did not remember his visitor. But in the twilight of stirring consciousness, he felt a memory unfit for the waking world slip mercifully from his mind and leave in its absence an abyssal cold, like the void of smothered stars, like the silence of a cosmic tomb. The cold lingered. The fragile sunlight of a New England [...]
---
**Outline:**
(02:02) The Antarctic tale
[... 3 more sections]
---
First published:
June 19th, 2026
Source:
https://www.lesswrong.com/posts/nhb8AyEcQGjQetgi5/the-llm-shoggoth-meme-is-weirder-than-you-think
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

[Linkpost] "Guardian Angels: LLM Personalization for Productivity and Security" by gwern Jun 21, 2026

Show notes

This is a link post. Powerful LLMs will be deployed at global scale in the next few years, and will dominate the Internet, and increasingly, ordinary life. As of mid-2026, there is no coherent vision for how knowledge professionals, or ordinary people, will be able to harness these LLMs for large productivity increases, or how they will handle cybersecurity and cognitive security.
I propose a goal of creating Guardian Angels (GA): digital twin LLMs which are personalized with the goal of providing not the stereotypical "assistant chatbot agent" persona, but emulating a single user's personality, values, and preferences.
This weakly solves the principal-agent problem by unifying the principal and agent as much as possible. In a GA future, the focus of the "principal" user is on defining what is worth doing by the GA (agent) users, and not on what or how to do things, functioning as the CEO or 'board' of an 'AI corporation'. This allows them to deploy numerous agents to achieve desirable things and to handle security, like screening all messages for advanced attacks (like interlocking ecosystems of synthetic media for propaganda or spearphishing). They cannot solve larger AI alignment problems, but they can help [...]
---
First published:
June 17th, 2026
Source:
https://www.lesswrong.com/posts/siWqHqCSybdhtWGud/guardian-angels-llm-personalization-for-productivity-and
Linkpost URL:
https://gwern.net/guardian-angel
---
Narrated by TYPE III AUDIO.

(CA, WA, and LA are excluded because of nonstandard rules: CA/WA use a top-two primary, and LA uses an all-party November ballot with a runoff if no one exceeds 50%, none of which produce a separate party primary to measure.) — **"Gears for political races" by Tom Smith** Jun 18, 2026
Show notes

In the past few years, many people around me have tried to convince me that US electoral politics is important. But like many other people in the community, I’ve been suspicious of many of the high-level arguments that I’ve heard. It felt like people were pulling numbers out of poorly-documented models I didn’t have time to examine and citing studies I didn’t have time to read. But I lacked a gears-level model of why and how individual efforts could impact electoral outcomes, and I felt intimidated by all the statistics and skeptical of trusting people adjacent to politics.
In the past year, as I’ve done more research and (more recently) volunteered on the ground to help Alex Bores's campaign in NY-12[1] (the guy who passed the RAISE Act and is now being targeted by the giant A16Z, Greg Brockman, Joe Lonsdale Super PAC), I’ve developed a gears-level understanding of how electoral politics in the US works.
I now believe that working on US electoral politics is one of the highest impact areas from the general AIS perspective. I feel like I was a fool. In this post, I’ll share some of the gears I’ve learned that inform this belief [...]
---
**Outline:**
(01:20) ~2% of open-seat primaries come down to 100 votes or less
(02:52) Talking to voters can net 1/3rd of a vote each hour
(05:32) Getting people to bother voting at all is a good strategy
(06:09) Campaigns are very money-constrained, which costs them time
(10:01) Returns don't really diminish
(11:24) There's lots of opportunities to be clever in ways that make you 50% more effective at canvassing
(11:49) If you're motivated and deeply care, you can greatly outperform the majority of volunteers
(13:21) Yes, when people spend tons to support/oppose a candidate, it has a notable effect
(15:16) Donations > reaching out to friends/warm contacts > canvassing > ~anything else an average person can do
(18:41) People over-fixate on vibes and win vs loss
(21:12) Some interventions feel like they don't work but the numbers say otherwise
(21:59) Seriously, a group of agentic people can be an enormous political force
---
First published:
June 17th, 2026
Source:
https://www.lesswrong.com/posts/nSqB3qYP36enJLRq2/gears-for-political-races
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

"A frontier AI company should shut down" by MichaelDickens Jun 16, 2026

Show notes

Cross-posted from my website.
Prior discussion: niplav's shortform (2025); Planning for Extreme AI Risks (2025) by Joshua Clymer
A frontier AI company (any one, I don't care which) should close shop and make an announcement along the lines of:
Powerful AI could end the human race. We are too worried that we don't know how to make this technology safe. We have decided to shut down because we don't want to be responsible for building the thing that kills us all.
A common refrain among safety-conscious AI developers: "it doesn't matter if we stop building dangerous AI, because someone else will just build it instead." Is that really true, though? If a multi-hundred-billion-dollar company comes out and says "We've concluded that our product is horribly dangerous, nobody knows how to make it safe, and there's too high a risk that it leads to human extinction", this won't raise any eyebrows? This has no chance of spurring policy-makers into action?
Shutting down would make people say, holy shit, they are serious about this extinction risk thing. Shutting down sends a strong signal to governments that they should pay serious attention to AI x-risk.
It [...]
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
June 15th, 2026
Source:
https://www.lesswrong.com/posts/bStYDEy8PQPt2c3Za/a-frontier-ai-company-should-shut-down
---
Narrated by TYPE III AUDIO.

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

Related Podcasts

Links

Stay Connected