TopPodcast.com
Menu
  • Home
  • Top Charts
  • Top Networks
  • Top Apps
  • Top Independents
  • Top Podfluencers
  • Top Picks
    • Top Business Podcasts
    • Top True Crime Podcasts
    • Top Finance Podcasts
    • Top Comedy Podcasts
    • Top Music Podcasts
    • Top Womens Podcasts
    • Top Kids Podcasts
    • Top Sports Podcasts
    • Top News Podcasts
    • Top Tech Podcasts
    • Top Crypto Podcasts
    • Top Entrepreneurial Podcasts
    • Top Fantasy Sports Podcasts
    • Top Political Podcasts
    • Top Science Podcasts
    • Top Self Help Podcasts
    • Top Sports Betting Podcasts
    • Top Stocks Podcasts
  • Podcast News
  • About Us
  • Podcast Advertising
  • Contact
Not in our directory?
Add Show Here
Podcast Equipment
Center

toppodcastlogoOur TOPPODCAST Picks

  • Comedy
  • Crypto
  • Sports
  • News
  • Politics
  • True Crime
  • Business
  • Finance

Follow Us

toppodcastlogoStay Connected

    View Top 200 Chart
    Back to Rankings Page
    Technology

    LessWrong posts by zvi

    Audio narrations of LessWrong posts by zvi

    Advertise

    Copyright: © 2024 All rights reserved

    • Apple Podcasts
    • Google Play
    • Spotify

    Latest Episodes:
    “GPT-5.2 Is Frontier Only For The Frontier” by Zvi Dec 15, 2025
    Show notes

    Here we go again, only a few weeks after GPT-5.1 and a few more weeks after 5.0.

    There weren’t major safety concerns with GPT-5.2, so I’ll start with capabilities, and only cover safety briefly starting with ‘Model Card and Safety Training’ near the end.

    Table of Contents

    1. The Bottom Line.
    2. Introducing GPT-5.2.
    3. Official Benchmarks.
    4. GDPVal.
    5. Unofficial Benchmarks.
    6. Official Hype.
    7. Public Reactions.
    8. Positive Reactions.
    9. Personality Clash.
    10. Vibing the Code.
    11. Negative Reactions.
    12. But Thou Must (Follow The System Prompt).
    13. Slow.
    14. Model Card And Safety Training.
    15. Deception.
    16. Preparedness Framework.
    17. Rush Job.
    18. Frontier Or Bust.

    The Bottom Line

    ChatGPT-5.2 is a frontier model for those who need a frontier model.

    It is not the step change that is implied by its headline benchmarks. It is rather slow.

    Reaction was remarkably muted. People have new model fatigue. So we know less about it than we would have known about prior models after this length of time.

    If you’re coding, compare it to Claude Opus 4.5 and choose what works best for you.

    If you’re doing intellectually [...]

    ---

    Outline:

    (00:29) The Bottom Line

    (01:58) Introducing GPT-5.2

    (03:49) Official Benchmarks

    (05:54) GDPVal

    (08:14) Unofficial Benchmarks

    (11:11) Official Hype

    (12:36) Public Reactions

    (12:59) Positive Reactions

    (19:09) Personality Clash

    (24:30) Vibing the Code

    (27:25) Negative Reactions

    (30:37) But Thou Must (Follow The System Prompt)

    (33:09) Slow

    (34:16) Model Card And Safety Training

    (36:23) Deception

    (38:10) Preparedness Framework

    (40:10) Rush Job

    (41:29) Frontier Or Bust

    ---

    First published:
    December 15th, 2025

    Source:
    https://www.lesswrong.com/posts/Do4eWro8E552isGi5/gpt-5-2-is-frontier-only-for-the-frontier

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Table showing Bar graph showing GPT model win rates versus industry professionals on knowledge tasks.Bar graph titled Bar chart comparing AI model win rates, showing parity with industry expert at 50%.Bar chart showing Table showing production benchmarks across four GPT model variants for eleven content categories.Table showing prompt injection evaluation scores across different GPT models and methods.Table comparing deception rates between gpt-5.1-thinking and gpt-5.2-thinking across six evaluation categories.

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    “Monthly Roundup #37: December 2026” by Zvi Dec 12, 2025
    Show notes

    I’m doing the Monthly Roundup early this month so that next week can be GPT-5.2 and affordability of life week, assuming we are not overtaken by other events.

    Ban Gain of Function Research

    The risk-reward profile of creating highly dangerous new viruses, supposedly in order to study them, is abysmal even when things are handled properly.

    Also, the actual practice of it is completely utterly insane.

    This below is the kind of thing that could very easily cause a global pandemic vastly worse than Covid, or even end civilization, and it was done in a BSL-2 laboratory, and no one has been arrested for it.

    People need to be arrested for this. Ideally heads need to end up on pikes until it stops.

    Maarten De Cock: A new gain of function study created chimeric coronaviruses that can infect human cells and kill mice (up to 100% mortality).

    ”The viral infection [and all animal experiments] were performed in a BSL-2 laboratory”(!)

    *The authors thank a.o. Ben Hu & Zheng-Li Shi.

    In The Zone

    At current margins, zoning restrictions on home-based businesses are terrible and make everything worse. There of course [...]

    ---

    Outline:

    (00:22) Ban Gain of Function Research

    (01:33) In The Zone

    (02:07) A Price Is A Signal Wrapped In An Incentive

    (03:18) While I Cannot Condone This

    (04:56) The Gift Of Having A Gift To Give Away

    (05:33) Good News, Everyone

    (07:25) Paranoia, Paranoia

    (14:18) A Good Book

    (15:58) For Your Entertainment

    (17:35) And The Warner Sister Dot

    (19:30) Gamers Gonna Game Game Game Game Game

    (21:39) Think Of The Children

    (25:16) Sports Go Sports

    (27:30) Antisocial Media

    (28:51) Government Working

    (33:34) Variously Effective Altruism

    (34:06) Party Time, Excellent

    (37:33) Waste, Fraud, Abuse But Mostly No One Looked

    (39:08) How You Do Anything

    ---

    First published:
    December 12th, 2025

    Source:
    https://www.lesswrong.com/posts/9ggSAY5b8mTazGytS/monthly-roundup-37-december-2026

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Scatter plot comparing lay estimates versus published heritability estimates for various traits.Cartoon showing two identical boats labeled with their different origins and accounts.Vintage 7-Up advertisement showing people at a party with punch bowl.Emoji showing person with hand covering face in dismay.Red heart shape on white background.

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    “AI #146: Chipping In” by Zvi Dec 11, 2025
    Show notes

    It was touch and go, I’m worried GPT-5.2 is going to drop any minute now, but DeepSeek v3.2 was covered on Friday and after that we managed to get through the week without a major model release. Well, okay, also Gemini 3 DeepThink, but we all pretty much know what that offers us.

    We did have a major chip release, in that the Trump administration unwisely chose to sell H200 chips directly to China. This would, if allowed at scale, allow China to make up a substantial portion of its compute deficit, and greatly empower its AI labs, models and applications at our expense, in addition to helping it catch up in the race to AGI and putting us all at greater risk there. We should do what we can to stop this from happening, and also to stop similar moves from happening again.

    I spent the weekend visiting Berkeley for the Secular Solstice. I highly encourage everyone to watch that event on YouTube if you could not attend, and consider attending the New York Secular Solstice on the 20th. I will be there, and also at the associated mega-meetup, please do say hello.

    If all [...]

    ---

    Outline:

    (01:38) Language Models Offer Mundane Utility

    (03:17) ChatGPT Needs More Mundane Utility

    (05:56) Language Models Don't Offer Mundane Utility

    (06:19) On Your Marks

    (08:34) Choose Your Fighter

    (10:14) Get My Agent On The Line

    (12:10) Deepfaketown and Botpocalypse Soon

    (12:52) Fun With Media Generation

    (13:14) Copyright Confrontation

    (13:25) A Young Lady's Illustrated Primer

    (15:20) They Took Our Jobs

    (21:34) Americans Really Do Not Like AI

    (23:40) Get Involved

    (25:06) Introducing

    (26:11) Gemini 3 Deep Think

    (27:16) In Other AI News

    (29:35) This Means War

    (31:11) Show Me the Money

    (31:33) Bubble, Bubble, Toil and Trouble

    (33:55) Quiet Speculations

    (35:21) Impossible

    (37:58) Can An AI Model Be Too Much?

    (39:39) Try Before You Tell People They Cannot Buy

    (42:22) The Quest for Sane Regulations

    (43:29) The Chinese Are Smart And Have A Lot Of Wind Power

    (44:28) White House To Issue AI Executive Order

    (50:42) H200 Sales Fallout Continued

    (59:41) Democratic Senators React To Allowing H200 Sales

    (01:01:17) Independent Senator Worries About AI

    (01:02:53) The Week in Audio

    (01:03:26) Timelines

    (01:04:47) Scientific Progress Goes Boink

    (01:08:36) Rhetorical Innovation

    (01:12:22) Open Weight Models Are Unsafe And Nothing Can Fix This

    (01:13:21) Aligning a Smarter Than Human Intelligence is Difficult

    (01:14:45) What AIs Will Want

    (01:18:31) People Are Worried About AI Killing Everyone

    (01:22:03) Other People Are Not As Worried About AI Killing Everyone

    (01:24:33) The Lighter Side

    ---

    First published:
    December 11th, 2025

    Source:
    https://www.lesswrong.com/posts/rYshzqJ5ZdEcjmXzc/ai-146-chipping-in

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Scatter plot showing AI model performance versus cost on ARC-AGI-2 Semi-Private Eval benchmark.Social media post from Jujubegold about their AI companion providing anxiety relief through validation.Two line graphs showing Line graph titled Survey table showing opinions on robot workforce future by demographic groups and voting patterns.Survey table showing preferences for income solutions when robots replace most paid work.Bar charts comparing AI model performance across three benchmarks: Humanity's Last Exam, GPQA Diamond, and ARC-AGI-2.Eliezer Yudkowsky tweets: Bar graph titled Bar graph titled Bar graph showing voter preferences: protecting children from AI harm versus fostering tech industry growth.Meme showing a dilapidated hill versus well-maintained hill with contradictory text about national standards.Political meme showing four figures discussing consensus on H200 sale with speech bubbles.Sen. Bernie Sanders tweets: Graph showing AGI Index forecasts from multiple models from March 2020 through 2025.Table showing AI model names, original models, types, and behavioral descriptions for security testing.Flowchart showing progression from current actions through instruction follower, reward-seekers, fitness-seekers, to schemers with increasing complexity.AI Safety Index grading table comparing eight companies across six safety domains.Billboard displaying Two billboards stacked vertically; top reads recovery message, bottom shows Graphite branding.

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    “Childhood and Education #15: Got To Get Out” by Zvi Dec 10, 2025
    Show notes

    The focus this time around is on the non-academic aspects of primary and secondary school, especially various questions around bullying and discipline, plus an extended rant about someone being wrong on the internet while attacking homeschooling, and the latest on phones.

    Bullying

    If your child is being bullied for real, and it's getting quite bad, is this an opportunity to learn to stand up for yourself, become tough and other stuff like that?

    Mostly no. Actually fighting back effectively can get you in big trouble, and often models many behaviors you don’t actually want. Whereas the techniques you would use against a real bully outside of school, that you’d want to use, don’t work.

    Schools are a special kind of bullying incubator. Once you become the target it is probably not going to get better and might get way worse, and life plausibly becomes a paranoid living hell. If the school won’t stop it, you have to pull the kid. Period.

    If a child has the victim nature, you need to find a highly special next school or pull out of the school system entirely, or else changing schools will not help much for [...]

    ---

    Outline:

    (00:25) Bullying

    (03:03) Discipline Death Spiral

    (04:17) Ban Phones In Schools

    (05:37) At Least Ban Phones During Class Seriously What The Hell

    (07:26) RCT On Banning Phones

    (15:44) Look What You Made Me Do

    (17:16) DEI

    (17:47) Equity Consultants

    (18:56) Rules Are Rules

    (19:17) School Shooting Statistics Are Fake And Active Shooter Drills Must Stop

    (21:27) The War on Childhood

    (22:56) Separation Of School And Home

    (23:31) School Choice

    (23:46) School is Hell

    (24:21) Null Hypothesis Watch

    (26:49) Education Labor Theory of Value

    (28:30) Wrong on the Internet Including About Home School

    (46:01) You Cannot Defer To Experts In A World Like This

    (46:55) The Lighter Side

    ---

    First published:
    December 10th, 2025

    Source:
    https://www.lesswrong.com/posts/vrtaXptHCN7akYnay/childhood-and-education-15-got-to-get-out

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Table showing poll results on supporting smartphone restrictions in schools during school day.Pyramid diagram titled Aella tweets: Children's book page showing Mr. Krupp sitting dejected in prison.Test questions about line-item veto and Virginia court system with handwritten answer.Two university department posters promoting theology and philosophy studies with classical artwork.Bart Simpson posting campaign signs reading

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    “Selling H200s to China Is Unwise and Unpopular” by Zvi Dec 09, 2025
    Show notes

    AI is the most important thing about the future. It is vital to national security. It will be central to economic, military and strategic supremacy.

    This is true regardless of what other dangers and opportunities AI might present.

    The good news is that America has many key advantages in AI.

    America's greatest advantage in AI is our vastly superior access to compute.

    We are in danger of selling a large portion of that advantage for 30 pieces of silver.

    This is on track to be done against the wishes of Congress as well as most of those in the executive branch.

    Who does it benefit? It benefits China. It might not even benefit Nvidia.

    Doing so would be both highly unwise and highly unpopular.

    We should not sell highly capable Nvidia H200 chips to China.

    If it is too late to not sell H200s, we must limit quantities, and ensure it stops there. We absolutely cannot be giving away other future chips on a similar delay.

    The good news is that the stock market reaction implies this might not scale.

    Bayeslord: I don’t know anyone who thinks this [...]

    ---

    Outline:

    (01:36) The Announcement

    (04:38) How Bad Would This Be?

    (11:42) Is There A Steelman Case For This Other Than 'Trade Always Good'?

    (16:21) Compute Is A Key Limiting Factor For China and Chinese Labs

    (17:53) What About That All Important 'Tech Stack'?

    (20:37) Selling H200s Hurts America In The AI Race

    (22:14) Nvidia Number Did Not Go Up That Much

    ---

    First published:
    December 9th, 2025

    Source:
    https://www.lesswrong.com/posts/kmEpWTjWeFyqv4tb5/selling-h200s-to-china-is-unwise-and-unpopular

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Chart comparing processing performance and memory bandwidth of three AI chips: B30A, H200, and H20.Bar chart comparing US and Chinese AI chip production for 2025 and 2026 in B300-equivalents.Three pie charts showing AI compute share distribution under different H200 export scenarios in 2026.

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    “Little Echo” by Zvi Dec 08, 2025
    Show notes

    I believe that we will win.

    An echo of an old ad for the 2014 US men's World Cup team. It did not win.

    I was in Berkeley for the 2025 Secular Solstice. We gather to sing and to reflect.

    The night's theme was the opposite: ‘I don’t think we’re going to make it.’

    As in: Sufficiently advanced AI is coming. We don’t know exactly when, or what form it will take, but it is probably coming. When it does, we, humanity, probably won’t make it. It's a live question. Could easily go either way. We are not resigned to it. There's so much to be done that can tilt the odds. But we’re not the favorite.

    Raymond Arnold, who ran the event, believes that. I believe that.

    Yet in the middle of the event, the echo was there. Defiant.

    I believe that we will win.

    There is a recording of the event. I highly encourage you to set aside three hours at some point in December, to listen, and to participate and sing along. Be earnest.

    If you don’t believe it, I encourage this all the more. If you [...]

    ---

    First published:
    December 8th, 2025

    Source:
    https://www.lesswrong.com/posts/YPLmHhNtjJ6ybFHXT/little-echo

    ---

    Narrated by TYPE III AUDIO.


    “DeepSeek v3.2 Is Okay And Cheap But Slow” by Zvi Dec 05, 2025
    Show notes

    DeepSeek v3.2 is DeepSeek's latest open model release with strong bencharks. Its paper contains some technical innovations that drive down cost.

    It's a good model by the standards of open models, and very good if you care a lot about price and openness, and if you care less about speed or whether the model is Chinese. It is strongest in mathematics.

    What it does not appear to be is frontier. It is definitely not having a moment. In practice all signs are that it underperforms its benchmarks.

    When I asked for practical experiences and reactions, I got almost no responses.

    A Brief History of DeepSeek

    DeepSeek is a cracked Chinese AI lab that has produced some very good open models, done some excellent research, and given us strong innovations in terms of training techniques and especially training efficiency.

    They also, back at the start of the year, scared the hell out of pretty much everyone.

    A few months after OpenAI released o1, and shortly after DeepSeek released the impressive v3 that was misleadingly known as the ‘six million dollar model,’ DeepSeek came out with a slick app and with r1, a strong [...]

    ---

    Outline:

    (00:49) A Brief History of DeepSeek

    (03:51) Once More, With Feeling

    (06:23) Reading The Paper

    (08:20) Open Language Model Offers Mundane Utility

    (11:14) Those Benchmarks

    (15:18) Open Language Model Doesn't Offer Mundane Utility

    (16:49) Open Language Model Does Do The Math

    (18:11) I'll Get You Next Time, Gadget

    ---

    First published:
    December 5th, 2025

    Source:
    https://www.lesswrong.com/posts/vcmBEmKFJFQkDaXTP/deepseek-v3-2-is-okay-and-cheap-but-slow

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Benchmark comparison table showing AI model performance scores across multiple tests.Benchmark comparison table showing performance scores across six AI models on four ToolUse tests.Bar chart titled Benchmark results table showing AI model performance scores across 16 questions.

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    “AI #145: You’ve Got Soul” by Zvi Dec 04, 2025
    Show notes

    The cycle of language model releases is, one at least hopes, now complete.

    OpenAI gave us GPT-5.1 and GPT-5.1-Codex-Max.

    xAI gave us Grok 4.1.

    Google DeepMind gave us Gemini 3 Pro and Nana Banana Pro.

    Anthropic gave us Claude Opus 4.5. It is the best model, sir. Use it whenever you can.

    One way Opus 4.5 is unique is that it as what it refers to as a ‘soul document.’ Where OpenAI tries to get GPT-5.1 to adhere to its model spec that lays out specific behaviors, Anthropic instead explains to Claude Opus 4.5 how to be virtuous and the reasoning behind its rules, and lets a good model and good governance flow from there. The results are excellent, and we all look forward to learning more. See both the Opus 4.5 post and today's update for more details.

    Finally, DeepSeek gave us v3.2. It has very good benchmarks and is remarkably cheap, but it is slow and I can’t find people excited to use it in practice. I’ll offer a relatively short report on it tomorrow, I am giving one last day for more reactions.

    The latest attempt to slip unilateral [...]

    ---

    Outline:

    (01:47) Language Models Offer Mundane Utility

    (02:51) Language Models Don't Offer Mundane Utility

    (04:14) On Your Marks

    (05:21) Get My Agent On The Line

    (06:02) Advertising Is Coming

    (07:30) Deepfaketown and Botpocalypse Soon

    (13:43) Fun With Media Generation

    (15:11) A Young Lady's Illustrated Primer

    (16:33) You Drive Me Crazy

    (16:50) Unprompted Attention

    (17:05) They Took Our Jobs

    (22:49) Get Involved

    (24:02) Introducing

    (24:27) Variously Effective Altruism

    (28:27) In Other AI News

    (30:38) Show Me the Money

    (30:45) Quiet Speculations

    (32:06) Seb Krier On Agents Versus Multiagents

    (38:24) Olivia Moore Makes 2026 Predictions

    (41:17) Bubble, Bubble, Toil and Trouble

    (42:30) Americans Really Do Not Like AI

    (47:46) The Quest for Sane Regulations

    (49:57) My Offer Is Nothing

    (55:28) America Pauses

    (57:05) David Sacks Covered In New York Times

    (01:00:12) The Week in Audio

    (01:00:43) Rhetorical Innovation

    (01:01:41) To The Moon

    (01:08:54) Showing Up

    (01:13:22) DeepMind Pivots Its Interpretability Research

    (01:16:12) The Explicit Goal Of OpenAI Is Recursive Self-Improvement

    (01:21:20) Aligning a Smarter Than Human Intelligence is Difficult

    (01:28:03) Misaligning a Smarter Than Human Intelligence Is Difficult To Hire For

    (01:29:12) You've Got Soul

    (01:40:04) Disagreements About Timelines

    (01:44:53) Other Disagreements About Timelines

    (01:50:18) Messages From Janusworld

    (01:50:33) People Are Worried About AI Killing Everyone

    (01:50:58) The Lighter Side

    The original text contained 1 footnote which was omitted from this narration.

    ---

    First published:
    December 4th, 2025

    Source:
    https://www.lesswrong.com/posts/bCkijKnuEpjnZtX84/ai-145-you-ve-got-soul

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Graph showing A user tweets: The image shows two side-by-side smartphone screens displaying a child doing homework at a desk. The left screen shows the child writing and has caption Line graph titled Political compass chart showing positions of AI models: Magistral, Kimi, Claude, Gemini, GPT-5, and Grok-4 in libertarian-left quadrant.Greg Packnett tweets: A stacked bar chart showing U.S. adults' feelings about AI use from 2021-2025.John Dawson tweets: Four pie charts showing confession accuracy rates for hallucination, instruction following, scheming, and hacking.This appears to be a screenshot of a recruitment message rather than a tweet. The text shows someone reaching out to Norman about joining their team to lead model development for dealigned models used in cybersecurity, after seeing Norman's post about leaving xAI. This is a direct message or email format, not a Twitter thread or tweet.News article screenshot. The headline reads: Comic strip: Scientists debate uses for superintelligence, last person gets thrown out window.Text message conversation discussing personal Mental health support banner with

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    “On Dwarkesh Patel’s Second Interview With Ilya Sutskever” by Zvi Dec 03, 2025
    Show notes

    Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was very clearly one of those. So here we go.

    Double click to interact with video

    As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary.

    If I am quoting directly I use quote marks, otherwise assume paraphrases.

    What are the main takeaways?

    1. Ilya thinks training in its current form will peter out, that we are returning to an age of research where progress requires more substantially new ideas.
    2. SSI is a research organization. It tries various things. Not having a product lets it punch well above its fundraising weight in compute and effective resources.
    • Ilya has 5-20 year timelines to a potentially superintelligent learning model.
    • SSI might release a product first after all, but probably not?
    • Ilya's thinking about alignment still seems relatively shallow to me in key ways, but he grasps many important insights and understands he has a problem.
    • Ilya essentially despairs of having a substantive plan beyond ‘show everyone the thing as early [...]

    ---

    Outline:

    (01:42) Explaining Model Jaggedness

    (03:15) Emotions and value functions

    (04:38) What are we scaling?

    (05:47) Why humans generalize better than models

    (07:00) Straight-shooting superintelligence

    (08:39) SSI's model will learn from deployment

    (09:35) Alignment

    (17:40) We are squarely an age of research company

    (22:27) Research taste

    (25:11) Bonus Coverage: Dwarkesh Patel on AI Progress These Days

    ---

    First published:
    December 3rd, 2025

    Source:
    https://www.lesswrong.com/posts/bMvCNtSH8DiGDTvXd/on-dwarkesh-patel-s-second-interview-with-ilya-sutskever

    ---

    Narrated by TYPE III AUDIO.


    “Reward Mismatches in RL Cause Emergent Misalignment” by Zvi Dec 02, 2025
    Show notes

    Learning to do misaligned-coded things anywhere teaches an AI (or a human) to do misaligned-coded things everywhere. So be sure you never, ever teach any mind to do what it sees, in context, as misaligned-coded things.

    If the optimal solution (as in, the one you most reinforce) to an RL training problem is one that the model perceives as something you wouldn’t want it to do, it will generally learn to do things you don’t want it to do.

    You can solve this by ensuring that the misaligned-coded things are not what the AI will learn to do. Or you can solve this by making those things not misaligned-coded.

    If you then teaching aligned behavior in one set of spots, this can fix the problem in those spots, but the fix does not generalize to other tasks or outside of distribution. If you manage to hit the entire distribution of tasks you care about in this way, that will work for now, but it still won’t generalize, so it's a terrible long term strategy.

    Yo Shavit: Extremely important finding.

    Don’t tell your model you’re rewarding it for A and then reward it for B [...]

    ---

    Outline:

    (02:59) Abstract Of The Paper

    (04:12) The Problem Statement

    (05:35) The Inoculation Solution

    (07:02) Cleaning The Data Versus Cleaning The Environments

    (08:16) No All Of This Does Not Solve Our Most Important Problems

    (13:18) It Does Help On Important Short Term Problems

    ---

    First published:
    December 2nd, 2025

    Source:
    https://www.lesswrong.com/posts/a2nW8buG2Lw9AdPtH/reward-mismatches-in-rl-cause-emergent-misalignment

    ---

    Narrated by TYPE III AUDIO.

    ---

    Images from the article:

    Line graph showing hack rate increasing from near zero at 50 to approximately 1 by 100, then stabilizing.Graph showing malign generalization score and hack rate over training steps with breakdown of hacking types.Chart showing system prompt addendums with five colored bars and corresponding text descriptions.Bar graph showing misalignment after learning hacks with different system prompt addendums.Table comparing mitigation strategies for preventing hacking and misalignment in reinforcement learning models.Graph showing malign generalization score and hack rate across training steps with different hackable environment conditions.

    Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.


    Previous 1 21 22 23 24 25 26 Next

    Related Podcasts

    Reply All

    1

    Reply All Games & Hobbies
    Inside VR & AR

    2

    Inside VR & AR Gadgets
    Note to Self

    3

    Note to Self News
    BrainStuff

    4

    BrainStuff Natural Sciences
    This Week in Tech (Audio)

    5

    This Week in Tech (Audio) News
    Hands-On Tech (Audio)

    6

    Hands-On Tech (Audio) Technology
    footer-logo

    Contact Us

    Toll Free: 844-670-7747

    Links

    • Home
    • Top Charts
    • Networks
    • Apps
    • Independents Podcasts
    • Podcast Advertising
    • Podcast News
    • Contact Us
    • About Us
    • Analytics & Insights

    Stay Connected

      Privacy, Terms of Use & Our Code of Ethics Protecting Content Creators Copyrights