LessWrong (Curated & Popular)

Latest Episodes:

“Ten people on the inside” by Buck Jan 29, 2025

Show notes

(Many of these ideas developed in conversation with Ryan Greenblatt)
In a shortform, I described some different levels of resources and buy-in for misalignment risk mitigations that might be present in AI labs:
*The “safety case” regime.* Sometimes people talk about wanting to have approaches to safety such that if all AI developers followed these approaches, the overall level of risk posed by AI would be minimal. (These approaches are going to be more conservative than will probably be feasible in practice given the amount of competitive pressure, so I think it's pretty likely that AI developers don’t actually hold themselves to these standards, but I agree with e.g. Anthropic that this level of caution is at least a useful hypothetical to consider.) This is the level of caution people are usually talking about when they discuss making safety cases. I usually operationalize this as the AI developer wanting [...]
---
First published:
January 28th, 2025
Source:
https://www.lesswrong.com/posts/WSNnKcKCYAffcnrt2/ten-people-on-the-inside
---
Narrated by TYPE III AUDIO.

undefined — **“Anomalous Tokens in DeepSeek-V3 and r1” by henry** Jan 28, 2025
Show notes

“Anomalous”, “glitch”, or “unspeakable” tokens in an LLM are those that induce bizarre behavior or otherwise don’t behave like regular text.
The SolidGoldMagikarp saga is pretty much essential context, as it documents the discovery of this phenomenon in GPT-2 and GPT-3.
But, as far as I was able to tell, nobody had yet attempted to search for these tokens in DeepSeek-V3, so I tried doing exactly that. Being a SOTA base model, open source, and an all-around strange LLM, it seemed like a perfect candidate for this.
This is a catalog of the glitch tokens I've found in DeepSeek after a day or so of experimentation, along with some preliminary observations about their behavior.
Note: I’ll be using “DeepSeek” as a generic term for V3 and r1.
**Process**
I searched for these tokens by first extracting the vocabulary from DeepSeek-V3's tokenizer, and then automatically testing every one of them [...]
---
**Outline:**
(00:55) Process
(03:30) Fragment tokens
(06:45) Other English tokens
(09:32) Non-English
(12:01) Non-English outliers
(14:09) Special tokens
(16:26) Base model mode
(17:40) Whats next?
The original text contained 1 footnote which was omitted from this narration.
The original text contained 12 images which were described by AI.
---
First published:
January 25th, 2025
Source:
https://www.lesswrong.com/posts/xtpcJjfWhn3Xn8Pu5/anomalous-tokens-in-deepseek-v3-and-r1
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

“Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals” by johnswentworth, David Lorell Jan 27, 2025

Show notes

The Cake
Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that's fine. Baking that cake is my terminal goal.
In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.
My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile [...]
---
Outline:
(00:07) The Cake
(01:50) The Restaurant
(03:50) Happy Instrumental Convergence?
(06:27) All The Way Up
(08:05) Research Threads
---
First published:
January 24th, 2025
Source:
https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of
---
Narrated by TYPE III AUDIO.

“A Three-Layer Model of LLM Psychology” by Jan_Kulveit Jan 26, 2025

Show notes

This post offers an accessible model of psychology of character-trained LLMs like Claude.
Epistemic Status
This is primarily a phenomenological model based on extensive interactions with LLMs, particularly Claude. It's intentionally anthropomorphic in cases where I believe human psychological concepts lead to useful intuitions.
Think of it as closer to psychology than neuroscience - the goal isn't a map which matches the territory in the detail, but a rough sketch with evocative names which hopefully which hopefully helps boot up powerful, intuitive (and often illegible) models, leading to practically useful results.
Some parts of this model draw on technical understanding of LLM training, but mostly it is just an attempt to take my "phenomenological understanding" based on interacting with LLMs, force it into a simple, legible model, and make Claude write it down.
I aim for a different point at the Pareto frontier than for example Janus: something [...]
---
Outline:
(00:11) Epistemic Status
(01:14) The Three Layers
(01:17) A. Surface Layer
(02:55) B. Character Layer
(05:09) C. Predictive Ground Layer
(07:24) Interactions Between Layers
(07:44) Deeper Overriding Shallower
(10:50) Authentic vs Scripted Feel of Interactions
(11:51) Implications and Uses
(15:54) Limitations and Open Questions
The original text contained 1 footnote which was omitted from this narration.
---
First published:
December 26th, 2024
Source:
https://www.lesswrong.com/posts/zuXo9imNKYspu9HGv/a-three-layer-model-of-llm-psychology
---
Narrated by TYPE III AUDIO.

“AI companies are unlikely to make high-assurance safety cases if timelines are short” by ryan_greenblatt Jan 24, 2025

Show notes

One hope for keeping existential risks low is to get AI companies to (successfully) make high-assurance safety cases: structured and auditable arguments that an AI system is very unlikely to result in existential risks given how it will be deployed.[1] Concretely, once AIs are quite powerful, high-assurance safety cases would require making a thorough argument that the level of (existential) risk caused by the company is very low; perhaps they would require that the total chance of existential risk over the lifetime of the AI company[2] is less than 0.25%[3][4].
The idea of making high-assurance safety cases (once AI systems are dangerously powerful) is popular in some parts of the AI safety community and a variety of work appears to focus on this. Further, Anthropic has expressed an intention (in their RSP) to "keep risks below acceptable levels"[5] and there is a common impression that Anthropic would pause [...]
---
Outline:
(03:19) Why are companies unlikely to succeed at making high-assurance safety cases in short timelines?
(04:14) Ensuring sufficient security is very difficult
(04:55) Sufficiently mitigating scheming risk is unlikely
(09:35) Accelerating safety and security with earlier AIs seems insufficient
(11:58) Other points
(14:07) Companies likely wont unilaterally slow down if they are unable to make high-assurance safety cases
(18:26) Could coordination or government action result in high-assurance safety cases?
(19:55) What about safety cases aiming at a higher risk threshold?
(21:57) Implications and conclusions
The original text contained 20 footnotes which were omitted from this narration.
---
First published:
January 23rd, 2025
Source:
https://www.lesswrong.com/posts/neTbrpBziAsTH5Bn7/ai-companies-are-unlikely-to-make-high-assurance-safety
---
Narrated by TYPE III AUDIO.

“The Gentle Romance” by Richard_Ngo Jan 22, 2025

Show notes

This is a link post.A story I wrote about living through the transition to utopia.
This is the one story that I've put the most time and effort into; it charts a course from the near future all the way to the distant stars.
---
First published:
January 19th, 2025
Source:
https://www.lesswrong.com/posts/Rz4ijbeKgPAaedg3n/the-gentle-romance
---
Narrated by TYPE III AUDIO.

“Quotes from the Stargate press conference” by Nikola Jurkovic Jan 22, 2025

Show notes

This is a link post.Present alongside President Trump:

Sam Altman
Larry Ellison (Oracle executive chairman and CTO)
Masayoshi Son (Softbank CEO who believes he was born to realize ASI)

President Trump: What we want to do is we want to keep [AI datacenters] in this country. China is a competitor and others are competitors.
President Trump: I'm going to help a lot through emergency declarations because we have an emergency. We have to get this stuff built. So they have to produce a lot of electricity and we'll make it possible for them to get that production done very easily at their own plants if they want, where they'll build at the plant, the AI plant they'll build energy generation and that will be incredible.
President Trump: Beginning immediately, Stargate will be building the physical and virtual infrastructure to power the next generation of [...]
---
First published:
January 22nd, 2025
Source:
https://www.lesswrong.com/posts/b8D7ng6CJHzbq8fDw/quotes-from-the-stargate-press-conference
---
Narrated by TYPE III AUDIO.

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

Related Podcasts

Links

Stay Connected