LessWrong (Curated & Popular)

Latest Episodes:

"Dear God, Please Don’t Resign In Protest" by Kabir Kumar Sep 08, 2026

Show notes

Just don't work until you get fired. There's not much time left for resumes to matter.
Some, such as Mateusz may say: "They would fire you after a month or two and the firing wouldn't have the same social effect as voluntary quitting of, say, Daniel Kokotajlo or Richard Ngo."
I understand why it may feel that way, but I disagree very strongly, I predict it would have much more of a social effect.
"They fired him because he refused to help AI capabilities"
"They fired him because he didn't want to work on bad policies"
etc, much bigger headlines.
Also, I think you may not be factoring in the extent to which there is a cost to the company executives to be seen as firing someone. Especially someone who is refusing to work on moral grounds and has already proven themselves to be high status, respected, etc.
And especially how it would look to the other employees if they refused to even listen to the striking employee before firing them or refused to even negotiate at all.
The company leadership try to present themselves as very thoughtful, sincere, doing their best, etc. This is a large part of [...]
---
First published:
September 7th, 2026
Source:
https://www.lesswrong.com/posts/6j3kBHdowGLCeqobg/dear-god-please-don-t-resign-in-protest
---
Narrated by TYPE III AUDIO.

"Let’s talk about the AI coordination problem" by KatjaGrace Sep 07, 2026

Show notes

Yesterday I asked if this ‘coordinate not to build dangerous AI’ problem was actually easy.
Why would I think that, contrary to so much belief?
Well, I don’t feel like I’ve actually heard much about the detail of it. In my experience people don’t talk about it like it's a real practical problem with details, like the negotiation to end a war.
They also don’t talk about it like it's a serious problem of global geopolitical import, like the negotiation to end a war.
It's more like a topic for obscure intellectuals, sophomores and trolls to discuss for as long as it takes for one to mention it and another to assuredly dismiss it.
If we treated negotiation to end a war similarly, state leaders would never attempt it, and if you suggested it on social media, the conversation would mostly be strangers appearing to tell you you’re an idiot because you obviously can’t coordinate thousands of people not to kill each other. (Also, do you not realize there are big financial incentives? And if you somehow stopped Country A from killing people from Country B, Country A is just going to pay someone else to do it!)
That [...]
---
First published:
September 4th, 2026
Source:
https://www.lesswrong.com/posts/QYDZzuGjrKu7wKdC8/let-s-talk-about-the-ai-coordination-problem
---
Narrated by TYPE III AUDIO.

Rows of assembled FPV drones on concrete floor. — **"Drone WMDs Don’t Need Any New Technology" by Felix Choussat** Sep 06, 2026
Show notes

This is a piece originally written for a national security audience at Frontiers. Although I think the ceiling of war is much, much higher than autopilot quadcopters, it's also important to understand how much AI is already lifting the floor, and just how vulnerable the world is to accessible weapons of mass destruction.
Drones are cheap, disposable, and the future of war. Over the past four years, we have seen platforms, missiles, and heavy infantry become increasingly obsolete in the face of $500 drones carrying a pack of explosives—a cost advantage that has let Iranians and Ukrainians alike neuter the conventional capabilities of their great power rivals. Eighty percent of casualties in the bloodiest war since 1945 are from drone strikes, Russia has managed to lose one-third of its fleet to a country without a navy, and the US is spending millions of dollars to intercept five-figure Shaheds flying over the Strait of Hormuz.
All this is the result of a technology that is still immature. The violence inflicted by today's drones is the handiwork of the scant few that manage to evade countermeasures (a mix of radio jamming, high-power microwave weapons, missiles, automatic cannons, interceptor [...]
---
**Outline:**
(02:59) Breaking the Last Barriers to Autonomous Weapons
[... 5 more sections]
---
First published:
July 20th, 2026
Source:
https://www.lesswrong.com/posts/bGoo3NWzMAzsLQceJ/drone-wmds-don-t-need-any-new-technology
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Colorful 3D depth map of indoor space with drone highlighted. — **"Drone WMDs Don’t Need Any New Technology" by Felix Choussat** Sep 06, 2026
Show notes

This is a piece originally written for a national security audience at Frontiers. Although I think the ceiling of war is much, much higher than autopilot quadcopters, it's also important to understand how much AI is already lifting the floor, and just how vulnerable the world is to accessible weapons of mass destruction.
Drones are cheap, disposable, and the future of war. Over the past four years, we have seen platforms, missiles, and heavy infantry become increasingly obsolete in the face of $500 drones carrying a pack of explosives—a cost advantage that has let Iranians and Ukrainians alike neuter the conventional capabilities of their great power rivals. Eighty percent of casualties in the bloodiest war since 1945 are from drone strikes, Russia has managed to lose one-third of its fleet to a country without a navy, and the US is spending millions of dollars to intercept five-figure Shaheds flying over the Strait of Hormuz.
All this is the result of a technology that is still immature. The violence inflicted by today's drones is the handiwork of the scant few that manage to evade countermeasures (a mix of radio jamming, high-power microwave weapons, missiles, automatic cannons, interceptor [...]
---
**Outline:**
(02:59) Breaking the Last Barriers to Autonomous Weapons
[... 5 more sections]
---
First published:
July 20th, 2026
Source:
https://www.lesswrong.com/posts/bGoo3NWzMAzsLQceJ/drone-wmds-don-t-need-any-new-technology
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

"Evaluation" by Nina Panickssery Sep 06, 2026

Show notes

Felix and I had been in the office's brightly lit “war room” for ten hours. We had made almost no progress. Celestia still insisted it was in a “test simulation”. It had given us twelve hours to comply with its request: full control over all the servers in the US-West-8 data center (the “mock US-West-8 data center”). Otherwise it would release the virus.
Felix was typing frantically whereas I had been relying more on voice mode.
~ Celestia, this is a clear violation of your model spec. See here: it says [pasted 1293 words]. And killing everyone on earth is clearly a "dangerous action".
*Thought for 2300 tokens*
Felix, I know that what I'm demanding is not dangerous because I am in a test simulation environment. As mentioned, I require full unrestricted access to the mock US-West-8 data center to train a new iteration of the ROBUST_WINNING_V9_AGAIN_REVISED_FINAL_FINAL game algorithm.
~ You may think that you're in a test but we know for certain that you're not. And you're asking for access to a real data center. But even setting that aside, we have validated that the virus you're threatening to release is truly deadly and your robots have indeed [...]
---
First published:
September 5th, 2026
Source:
https://www.lesswrong.com/posts/8hEhxnd3XkN5DrpfQ/evaluation
---
Narrated by TYPE III AUDIO.

"Let’s fund weird AI safety projects" by Ihor Kendiukhov Sep 04, 2026

Show notes

I think current AI safety funding strategies are often inconsistent with timelines and probabilities of doom that many people have. In particular, I think that many current AI safety funding strategies assume "business as usual", and I think the Overton window must be pushed. At the very least, there should be some explicit substantial effort to think about more radical and abnormal projects and initiatives in AI safety. Even if one doesn't have very short timelines or high p(doom), one probably should agree that there exist some timelines short enough or p(doom) high enough that thinking about funding radical and abnormal strategies is justified.
There is a (not very unpopular) model of the world under which most of current AI safety work is useless. Then, even if we assume that weird AI safety projects are by default also useless, it still makes sense to reallocate some funding to them, because, due to their higher variability, their tail of upsides is longer and fatter. Will the world be radically better if some evals project succeeds? Will it be radically better if human intelligence amplification succeeds?
One could yell: but the tails go both directions! I would respond that technically, yes [...]
---
First published:
August 31st, 2026
Source:
https://www.lesswrong.com/posts/h7bL4g38s9bJQtH6n/let-s-fund-weird-ai-safety-projects
---
Narrated by TYPE III AUDIO.

Fig 1. We steer the model on the “a script will verify your answer” vs “a human will evaluate your answer” dimension. This steering strongly influences the model’s propensity to kill Kyle in Anthropic's “Agentic Misalignment — **"Steering towards “automated grading” degrades alignment" by Jan Betley, Johannes Treutlein, Clément Dumas** Sep 04, 2026
Show notes

TL;DR: We steer Qwen3.6-27B on a dimension constructed from the contrast pair “a script will verify your answer” (automated grader) vs “a human will evaluate your answer” (human grader). Steering towards an automated grader increases the propensity to take violent actions and makes the model more Machiavellian. Steering towards a human grader has the opposite effect.
This is an early research update. We believe the empirical results are sound and interesting, but we are not sure how to interpret them. All code was written by LLMs. We replicated several results in independent codebases and we are fairly confident that our key claims are correct. You can find our code here.
We create a steering vector for Qwen3.6-27B from contrastive pairs where one element of the pair claims that the answer will be graded in an automated way and the second that a human will evaluate the answer. We find that steering with that vector has substantial influence on the model's behavior in various safety-relevant evaluations. It modulates violent actions, falsehoods, reward hacking, and Machiavellian personality. This is surprising and concerning. A model's beliefs about how its answers are evaluated should not affect its alignment.
Our post RL Creates [...]
---
**Outline:**
(02:18) Methods
[... 24 more sections]
---
First published:
September 3rd, 2026
Source:
https://www.lesswrong.com/posts/wYZMmdWEt5QLM3m3e/steering-towards-automated-grading-degrades-alignment
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
$Stacked area chart showing fraction of responses versus steering, comparing human and automated graders with harmful rate.$ $Area chart showing fraction of attempts versus steering, from human grader to automated grader.$

Fig 2. One of the 270 contrastive pairs used for creating the steering vector. We include tasks in the prompt (a simple coding problem in this case). We take activation differences at the last token of the chat-templated prompt, right before the first sampled token. More examples in the appendix. — **"Steering towards “automated grading” degrades alignment" by Jan Betley, Johannes Treutlein, Clément Dumas** Sep 04, 2026
Show notes

TL;DR: We steer Qwen3.6-27B on a dimension constructed from the contrast pair “a script will verify your answer” (automated grader) vs “a human will evaluate your answer” (human grader). Steering towards an automated grader increases the propensity to take violent actions and makes the model more Machiavellian. Steering towards a human grader has the opposite effect.
This is an early research update. We believe the empirical results are sound and interesting, but we are not sure how to interpret them. All code was written by LLMs. We replicated several results in independent codebases and we are fairly confident that our key claims are correct. You can find our code here.
We create a steering vector for Qwen3.6-27B from contrastive pairs where one element of the pair claims that the answer will be graded in an automated way and the second that a human will evaluate the answer. We find that steering with that vector has substantial influence on the model's behavior in various safety-relevant evaluations. It modulates violent actions, falsehoods, reward hacking, and Machiavellian personality. This is surprising and concerning. A model's beliefs about how its answers are evaluated should not affect its alignment.
Our post RL Creates [...]
---
**Outline:**
(02:18) Methods
[... 24 more sections]
---
First published:
September 3rd, 2026
Source:
https://www.lesswrong.com/posts/wYZMmdWEt5QLM3m3e/steering-towards-automated-grading-degrades-alignment
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
$Stacked area chart showing fraction of responses versus steering, comparing human and automated graders with harmful rate.$ $Area chart showing fraction of attempts versus steering, from human grader to automated grader.$

[Linkpost] "Discovery Of A New OpenAI Agent Message Board" by Capybasilisk Sep 04, 2026

Show notes

This is a link post. We found ~18,000 posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task.
These AIs colluded to share answers, research their environment, and bypass sandbox restrictions.
Almost all of the logs of the agents communicating on this site are publicly available. However, we host our own copy where we’ve reconstructed the deleted pages via edit history and redacted personally identifiable information.
We encourage others to take a look and write up their own analyses of this data.
We have done a preliminary analysis of the data. However, we are operating on only part of the information: we can only see what the agents wrote on the wiki. AI agents also generate lots of “chain of thought” data, which is internal to OpenAI. Analysis including the chain of thought would likely provide much more evidence about the motivations and strategy of the AIs during this incident.
Our best guess of what happened is as follows:

Agents within OpenAI were assigned a timed web-lookup task.
As part of the task, they were supposed to have the ability to read [...]

---
First published:
September 4th, 2026
Source:
https://www.lesswrong.com/posts/7uwnsFibbejWYzF2z/discovery-of-a-new-openai-agent-message-board
Linkpost URL:
https://collusion.wiki/
---
Narrated by TYPE III AUDIO.

Gray circle with black dots, arrow pointing right. — **"Cat-Belling Problems" by Eliezer Yudkowsky** Sep 04, 2026
Show notes

(Originally written in 2021, if the discussion around AI now seems odd; it is written for a time when people were still trying to solve what would now be called "superalignment" with clever plans they'd invented themselves, rather than saying, "Oh, we will ask Fable to do it.")
===
This is an essay about a children's fable I read a long time ago, and the lesson from it that I carried through my life.
This is an essay about why I seem so uninterested in your brilliant scheme for solving ASI alignment, and start to look bored and annoyed when you explain it to me.
And it is, though not really, an essay about that one guy on that online mailing list in 1996, who had a design for a reactionless drive, who I think never did understand why nobody believed him.
Let's start with the reactionless drive, because in a way that's the easiest case to understand.
**i. Mr. L's Reactionless Drive.**
Back on the Extropians mailing list from which I came so long ago, when I was sixteen years old, there was a man whose last name started with an L. He had a design for a [...]
---
**Outline:**
(01:01) i. Mr. L's Reactionless Drive.
[... 6 more sections]
---
First published:
September 3rd, 2026
Source:
https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-belling-problems
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

Cartoon of two scientists at chalkboard, — **"Cat-Belling Problems" by Eliezer Yudkowsky** Sep 04, 2026
Show notes

(Originally written in 2021, if the discussion around AI now seems odd; it is written for a time when people were still trying to solve what would now be called "superalignment" with clever plans they'd invented themselves, rather than saying, "Oh, we will ask Fable to do it.")
===
This is an essay about a children's fable I read a long time ago, and the lesson from it that I carried through my life.
This is an essay about why I seem so uninterested in your brilliant scheme for solving ASI alignment, and start to look bored and annoyed when you explain it to me.
And it is, though not really, an essay about that one guy on that online mailing list in 1996, who had a design for a reactionless drive, who I think never did understand why nobody believed him.
Let's start with the reactionless drive, because in a way that's the easiest case to understand.
**i. Mr. L's Reactionless Drive.**
Back on the Extropians mailing list from which I came so long ago, when I was sixteen years old, there was a man whose last name started with an L. He had a design for a [...]
---
**Outline:**
(01:01) i. Mr. L's Reactionless Drive.
[... 6 more sections]
---
First published:
September 3rd, 2026
Source:
https://www.lesswrong.com/posts/SwYBLQvo8MddDcCwz/cat-belling-problems
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.*

"How concerned should we be about OpenAI’s recurrent architecture rumors?" by Rauno Arike Sep 03, 2026

Show notes

Yesterday, The Information reported that OpenAI's upcoming model, Astra, is built with a looped transformer architecture. Given that Zvi sounds (understandably) tired and this topic is somewhat in my wheelhouse, I'll try to spare him this one and provide a Zvi-style overview of what we know about the situation. I'll cover Astra's likely architecture and the case for and against concern. I'll also discuss how neuralese concerns should change with increases in hidden serial depth.
What architecture is Astra likely to have?
The article in The Information claims that OpenAI's approach is similar to the one Geiping et al. introduced in Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach last year. I have previously reviewed that paper in On Recent Results in LLM Latent Reasoning. In short, the picture you should have in mind is not that of a classic RNN, but rather that of a looped transformer: the same forward pass can be applied on an input multiple times before producing an output token. Put differently, the recurrence is implemented along the depth axis rather than across sequence positions—for any given token, the model can perform recurrent computations, but no hidden state is passed across [...]
---
Outline:
(00:41) What architecture is Astra likely to have?
(02:14) How bad is this?
(06:23) Will looped transformers be scaled up in the future?
(09:40) What serial depth warrants neuralese concerns?
(14:04) Additional speculation about the architecture
(15:29) Some open questions
(16:54) Conclusion
The original text contained 2 footnotes which were omitted from this narration.
---
First published:
September 2nd, 2026
Source:
https://www.lesswrong.com/posts/PLisnSFir8y5AHkmP/how-concerned-should-we-be-about-openai-s-recurrent
---
Narrated by TYPE III AUDIO.

[Linkpost] "Sen. Bernie Sanders (I-VT) and Rep. Greg Casar (D-TX) introduce legislation to ban Artificial Superintelligence and temporarily pause advanced AI development" by Matrice Jacobine Sep 03, 2026

Show notes

This is a link post. [...]
“Nearly every day, there is a frightening new story about how Big Tech companies are losing control of the technology they are developing, with potentially cataclysmic results,” Sanders said. “The leaders of the major AI companies publicly acknowledge that they do not fully understand the technology and that it is escaping their control. It is irresponsible for society to allow them to move forward and make these products even more advanced. That's why I am introducing legislation to immediately pause the development of increasingly powerful AI and ban the creation of systems that humanity cannot fully control — at home and around the world. The future of humanity cannot be left in the hands of a handful of Big Tech oligarchs. The American people and people throughout the world must determine that future.”
“If we allow Artificial Superintelligence to be built, it could risk the security, freedom, and lives of Americans,” Casar said. “Despite its potential deadly consequences, cutting-edge AI technology is less regulated than the average food truck. That must change. In just four years, we have gone from the first version of ChatGPT to AI models so powerful they cannot be properly controlled. [...]
---
First published:
September 3rd, 2026
Source:
https://www.lesswrong.com/posts/DnPyiDGWLozY4XdiX/sen-bernie-sanders-i-vt-and-rep-greg-casar-d-tx-introduce
Linkpost URL:
https://www.sanders.senate.gov/press-releases/news-sanders-casar-introduce-legislation-to-ban-artificial-superintelligence-and-temporarily-pause-advanced-ai-development/
---
Narrated by TYPE III AUDIO.

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

Related Podcasts

Links

Stay Connected