LessWrong (Curated & Popular)

Latest Episodes:

“What o3 Becomes by 2028” by Vladimir_Nesov Jan 09, 2025

Show notes

Funding for $150bn training systems just turned less speculative, with OpenAI o3 reaching 25% on FrontierMath, 70% on SWE-Verified, 2700 on Codeforces, and 80% on ARC-AGI. These systems will be built in 2026-2027 and enable pretraining models for 5e28 FLOPs, while o3 itself is plausibly based on an LLM pretrained only for 8e25-4e26 FLOPs. The natural text data wall won't seriously interfere until 6e27 FLOPs, and might be possible to push until 5e28 FLOPs. Scaling of pretraining won't end just yet.
Reign of GPT-4
Since the release of GPT-4 in March 2023, subjectively there was no qualitative change in frontier capabilities. In 2024, everyone in the running merely caught up. To the extent this is true, the reason might be that the original GPT-4 was probably a 2e25 FLOPs MoE model trained on 20K A100. And if you don't already have a cluster this big, and experience [...]
---
Outline:
(00:52) Reign of GPT-4
(02:08) Engines of Scaling
(04:06) Two More Turns of the Crank
(06:41) Peak Data
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
December 22nd, 2024
Source:
https://www.lesswrong.com/posts/NXTkEiaLA4JdS5vSZ/what-o3-becomes-by-2028
---
Narrated by TYPE III AUDIO.

undefined — **“What Indicators Should We Watch to Disambiguate AGI Timelines?” by snewman** Jan 09, 2025
Show notes

(Cross-post from https://amistrongeryet.substack.com/p/are-we-on-the-brink-of-agi, lightly edited for LessWrong. The original has a lengthier introduction and a bit more explanation of jargon.)
No one seems to know whether transformational AGI is coming within a few short years. Or rather, everyone seems to know, but they all have conflicting opinions. Have we entered into what will in hindsight be not even the early stages, but actually the middle stage, of the mad tumbling rush into singularity? Or are we just witnessing the exciting early period of a new technology, full of discovery and opportunity, akin to the boom years of the personal computer and the web?
AI is approaching elite skill at programming, possibly barreling into superhuman status at advanced mathematics, and only picking up speed. Or so the framing goes. And yet, most of the reasons for skepticism are still present. We still evaluate AI only on neatly encapsulated, objective tasks [...]
---
**Outline:**
(02:49) The Slow Scenario
(09:13) The Fast Scenario
(17:24) Identifying The Requirements for a Short Timeline
(22:53) How To Recognize The Express Train to AGI
The original text contained 14 footnotes which were omitted from this narration.
The original text contained 3 images which were described by AI.
---
First published:
January 6th, 2025
Source:
https://www.lesswrong.com/posts/auGYErf5QqiTihTsJ/what-indicators-should-we-watch-to-disambiguate-agi
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**
*Apple Podcasts and Spotify do not show images in the episode descrip*

“How will we update about scheming?” by ryan_greenblatt Jan 08, 2025

Show notes

I mostly work on risks from scheming (that is, misaligned, power-seeking AIs that plot against their creators such as by faking alignment). Recently, I (and co-authors) released "Alignment Faking in Large Language Models", which provides empirical evidence for some components of the scheming threat model.
One question that's really important is how likely scheming is. But it's also really important to know how much we expect this uncertainty to be resolved by various key points in the future. I think it's about 25% likely that the first AIs capable of obsoleting top human experts[1] are scheming. It's really important for me to know whether I expect to make basically no updates to my P(scheming)[2] between here and the advent of potentially dangerously scheming models, or whether I expect to be basically totally confident one way or another by that point (in the same way that, though I might [...]
---
Outline:
(03:20) My main qualitative takeaways
(04:56) Its reasonably likely (55%), conditional on scheming being a big problem, that we will get smoking guns.
(05:38) Its reasonably likely (45%), conditional on scheming being a big problem, that we wont get smoking guns prior to very powerful AI.
(15:59) My P(scheming) is strongly affected by future directions in model architecture and how the models are trained
(16:33) The model
(22:38) Properties of the AI system and training process
(23:02) Opaque goal-directed reasoning ability
(29:24) Architectural opaque recurrence and depth
(34:14) Where do capabilities come from?
(39:42) Overall distribution from just properties of the AI system and training
(41:20) Direct observations
(41:43) Baseline negative updates
(44:35) Model organisms
(48:21) Catching various types of problematic behavior
(51:22) Other observations and countermeasures
(52:02) Training processes with varying (apparent) situational awareness
(54:05) Training AIs to seem highly corrigible and (mostly) myopic
(55:46) Reward hacking
(57:28) P(scheming) under various scenarios (putting aside mitigations)
(01:05:19) An optimistic and a pessimistic scenario for properties
(01:10:26) Conclusion
(01:11:58) Appendix: Caveats and definitions
(01:14:49) Appendix: Capabilities from intelligent learning algorithms
The original text contained 15 footnotes which were omitted from this narration.
---
First published:
January 6th, 2025
Source:
https://www.lesswrong.com/posts/aEguDPoCzt3287CCD/how-will-we-update-about-scheming
---
Narrated by TYPE III AUDIO.

“OpenAI #10: Reflections” by Zvi Jan 08, 2025

Show notes

This week, Altman offers a post called Reflections, and he has an interview in Bloomberg. There's a bunch of good and interesting answers in the interview about past events that I won’t mention or have to condense a lot here, such as his going over his calendar and all the meetings he constantly has, so consider reading the whole thing.
Table of Contents

The Battle of the Board.
Altman Lashes Out.
Inconsistently Candid.
On Various People Leaving OpenAI.
The Pitch.
Great Expectations.
Accusations of Fake News.
OpenAI's Vision Would Pose an Existential Risk To Humanity.

The Battle of the Board
Here is what he says about the Battle of the Board in Reflections:
Sam Altman: A little over a year ago, on one particular Friday, the main thing that had gone wrong that day was [...]
---
Outline:
(00:25) The Battle of the Board
(05:12) Altman Lashes Out
(07:48) Inconsistently Candid
(09:35) On Various People Leaving OpenAI
(10:56) The Pitch
(12:07) Great Expectations
(12:56) Accusations of Fake News
(15:02) OpenAI's Vision Would Pose an Existential Risk To Humanity
---
First published:
January 7th, 2025
Source:
https://www.lesswrong.com/posts/XAKYawaW9xkb3YCbF/openai-10-reflections
---
Narrated by TYPE III AUDIO.

“Maximizing Communication, not Traffic” by jefftk Jan 06, 2025

Show notes

As someone who writes for fun, I don't need to get people onto my site:

If I write a post and some people are able to get the core ideajust from the title or a tweet-length summary, great!
I can include the full contents of my posts in my RSS feed andon FB, because so what if people read the whole post there and neverclick though to my site?

It would be different if I funded my writing through ads (maximizetime on site to maximize impressions) or subscriptions (get the chanceto pitch, probably want to tease a paywall).
Sometimes I notice myself accidentallycopying what makes sense for other writers. For example, becauseI can't put full-length posts on Bluesky or Mastodon I write shortintros and [...]
---
First published:
January 5th, 2025
Source:
https://www.lesswrong.com/posts/ZqcC6Znyg8YrmKPa4/maximizing-communication-not-traffic
---
Narrated by TYPE III AUDIO.

“What’s the short timeline plan?” by Marius Hobbhahn Jan 02, 2025

Show notes

This is a low-effort post. I mostly want to get other people's takes and express concern about the lack of detailed and publicly available plans so far. This post reflects my personal opinion and not necessarily that of other members of Apollo Research. I’d like to thank Ryan Greenblatt, Bronson Schoen, Josh Clymer, Buck Shlegeris, Dan Braun, Mikita Balesni, Jérémy Scheurer, and Cody Rushing for comments and discussion.
I think short timelines, e.g. AIs that can replace a top researcher at an AGI lab without losses in capabilities by 2027, are plausible. Some people have posted ideas on what a reasonable plan to reduce AI risk for such timelines might look like (e.g. Sam Bowman's checklist, or Holden Karnofsky's list in his 2022 nearcast), but I find them insufficient for the magnitude of the stakes (to be clear, I don’t think these example lists were intended to be an [...]
---
Outline:
(02:36) Short timelines are plausible
(07:10) What do we need to achieve at a minimum?
(10:50) Making conservative assumptions for safety progress
(12:33) So whats the plan?
(14:31) Layer 1
(15:41) Keep a paradigm with faithful and human-legible CoT
(18:15) Significantly better (CoT, action and white-box) monitoring
(21:19) Control (that doesn't assume human-legible CoT)
(24:16) Much deeper understanding of scheming
(26:43) Evals
(29:56) Security
(31:52) Layer 2
(32:02) Improved near-term alignment strategies
(34:06) Continued work on interpretability, scalable oversight, superalignment and co
(36:12) Reasoning transparency
(38:36) Safety first culture
(41:49) Known limitations and open questions
---
First published:
January 2nd, 2025
Source:
https://www.lesswrong.com/posts/bb5Tnjdrptu89rcyY/what-s-the-short-timeline-plan
---
Narrated by TYPE III AUDIO.

“By default, capital will matter more than ever after AGI” by L Rudolf L Dec 29, 2024

Show notes

I've heard many people say something like "money won't matter post-AGI". This has always struck me as odd, and as most likely completely incorrect.
First: labour means human mental and physical effort that produces something of value. Capital goods are things like factories, data centres, and software—things humans have built that are used in the production of goods and services. I'll use "capital" to refer to both the stock of capital goods and to the money that can pay for them. I'll say "money" when I want to exclude capital goods.
The key economic effect of AI is that it makes capital a more and more general substitute for labour. There's less need to pay humans for their time to perform work, because you can replace that with capital (e.g. data centres running software replaces a human doing mental labour).
I will walk through consequences of this, and end [...]
---
Outline:
(03:10) The default solution
(04:18) Money currently struggles to buy talent
(09:15) Most peoples power/leverage derives from their labour
(09:41) Why are states ever nice?
(14:32) No more outlier outcomes?
(20:27) Enforced equality is unlikely
(22:34) The default outcome?
(26:04) Whats the takeaway?
---
First published:
December 28th, 2024
Source:
https://www.lesswrong.com/posts/KFFaKu27FNugCHFmh/by-default-capital-will-matter-more-than-ever-after-agi
---
Narrated by TYPE III AUDIO.

“The Field of AI Alignment: A Postmortem, and What To Do About It” by johnswentworth Dec 26, 2024

Show notes

A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is".
Over the past few years, a major source of my relative optimism on AI has been the hope that the field of alignment would transition from pre-paradigmatic to paradigmatic, and make much more rapid progress.
At this point, that hope is basically dead. There has been some degree of paradigm formation, but the memetic competition has mostly been won by streetlighting: the large majority of AI Safety researchers and activists [...]
---
Outline:
(01:23) What This Post Is And Isnt, And An Apology
(03:39) Why The Streetlighting?
(03:42) A Selection Model
(05:47) Selection and the Labs
(07:06) A Flinching Away Model
(09:47) What To Do About It
(11:16) How We Got Here
(11:57) Who To Recruit Instead
(13:02) Integration vs Separation
---
First published:
December 26th, 2024
Source:
https://www.lesswrong.com/posts/nwpyhyagpPYDn4dAW/the-field-of-ai-alignment-a-postmortem-and-what-to-do-about
---
Narrated by TYPE III AUDIO.

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

Related Podcasts

Links

Stay Connected