LessWrong (Curated & Popular)

Latest Episodes:

undefined — **“Contra papers claiming superhuman AI forecasting ” by nikos, Peter Mühlbacher, Lawrence Phillips, dschwarz** Sep 14, 2024
Show notes

[Conflict of interest disclaimer: We are FutureSearch, a company working on AI-powered forecasting and other types of quantitative reasoning. If thin LLM wrappers could achieve superhuman forecasting performance, this would obsolete a lot of our work.]
**Widespread, misleading claims about AI forecasting**
Recently we have seen a number of papers – (Schoenegger et al., 2024, Halawi et al., 2024, Phan et al., 2024, Hsieh et al., 2024) – with claims that boil down to “we built an LLM-powered forecaster that rivals human forecasters or even shows superhuman performance”.
These papers do not communicate their results carefully enough, shaping public perception in inaccurate and misleading ways. Some examples of public discourse:

Ethan Mollick (>200k followers) tweeted the following about the paper Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy by Schoenegger et al.:
A post on Marginal Revolution with the title and abstract [...]
---
**Outline:**
(00:24) Widespread, misleading claims about AI forecasting
(03:02) What does human-level or superhuman forecasting mean?
(04:08) Red flags for claims to (super)human AI forecasting accuracy
(06:42) Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy (Schoenegger et al., 2024)
(09:14) Approaching Human-Level Forecasting with Language Models (Halawi et al., 2024)
(11:10) Reasoning and Tools for Human-Level Forecasting (Hsieh et al., 2024)
(12:50) LLMs Are Superhuman Forecasters (Phan et al., 2024)
(15:19) Takeaways
(16:17) So how good are AI forecasters?
The original text contained 1 footnote which was omitted from this narration.
The original text contained 6 images which were described by AI.
---
First published:
September 12th, 2024
Source:
https://www.lesswrong.com/posts/uGkRcHqatmPkvpGLq/contra-papers-claiming-superhuman-ai-forecasting
---
Narrated by TYPE III AUDIO.
---
**Images from the article:**

“OpenAI o1 ” by Zach Stein-Perlman Sep 13, 2024

Show notes

This is a link post. ---
First published:
September 12th, 2024
Source:
https://www.lesswrong.com/posts/bhY5aE4MtwpGf3LCo/openai-o1
---
Narrated by TYPE III AUDIO.

“My Number 1 Epistemology Book Recommendation: Inventing Temperature ” by adamShimi Sep 09, 2024

Show notes

In my last post, I wrote that no resource out there exactly captured my model of epistemology, which is why I wanted to share a half-baked version of it.
But I do have one book which I always recommend to people who want to learn more about epistemology: Inventing Temperature by Hasok Chang.
To be very clear, my recommendation is not just to get the good ideas from this book (of which there are many) from a book review or summary — it's to actually read the book, the old-school way, one word at a time.
Why? Because this book teaches you the right feel, the right vibe for thinking about epistemology. It punctures the bubble of sterile non-sense that so easily pass for “how science works” in most people's education, such as the “scientific method”. And it does so by demonstrating how one actually makes progress in epistemology [...]
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
September 8th, 2024
Source:
https://www.lesswrong.com/posts/TbaCa7sY3GxHBcXTd/my-number-1-epistemology-book-recommendation-inventing
---
Narrated by TYPE III AUDIO.

“That Alien Message - The Animation ” by Writer Sep 09, 2024

Show notes

Our new video is an adaptation of That Alien Message, by @Eliezer Yudkowsky. This time, the text has been significantly adapted, so I include it below.
Part 1
Picture a world just like ours, except the people are a fair bit smarter: in this world, Einstein isn’t one in a million, he's one in a thousand. In fact, here he is now. He's made all the same discoveries, but they’re not quite as unusual: there have been lots of other discoveries. Anyway, he's out one night with a friend looking up at the stars when something odd happens. [visual: stars get brighter and dimmer, one per second. The two people on the hill look at each other, confused]
The stars are flickering. And it's just not a hallucination. Everyone's seeing it.
And so everyone immediately freaks out and panics! Ah, just kidding, the people of this world are [...]
---
Outline:
(00:16) Part 1
(06:22) Part 2
(09:53) Part 3
(11:58) Part 4
---
First published:
September 7th, 2024
Source:
https://www.lesswrong.com/posts/Q9omyL3qooXdjnyZn/that-alien-message-the-animation
---
Narrated by TYPE III AUDIO.

“Pay Risk Evaluators in Cash, Not Equity ” by Adam Scholl Sep 07, 2024

Show notes

Personally, I suspect the alignment problem is hard. But even if it turns out to be easy, survival may still require getting at least the absolute basics right; currently, I think we're mostly failing even at that.
Early discussion of AI risk often focused on debating the viability of various elaborate safety schemes humanity might someday devise—designing AI systems to be more like “tools” than “agents,” for example, or as purely question-answering oracles locked within some kryptonite-style box. These debates feel a bit quaint now, as AI companies race to release agentic models they barely understand directly onto the internet.
But a far more basic failure, from my perspective, is that at present nearly all AI company staff—including those tasked with deciding whether new models are safe to build and release—are paid substantially in equity, the value of which seems likely to decline if their employers stop building and [...]
---
First published:
September 7th, 2024
Source:
https://www.lesswrong.com/posts/sMBjsfNdezWFy6Dz5/pay-risk-evaluators-in-cash-not-equity
---
Narrated by TYPE III AUDIO.

“things that confuse me about the current AI market. ” by DMMF Sep 02, 2024

Show notes

Paging Gwern or anyone else who can shed light on the current state of the AI market—I have several questions.
Since the release of ChatGPT, at least 17 companies, according to the LMSYS Chatbot Arena Leaderboard, have developed AI models that outperform it. These companies include Anthropic, NexusFlow, Microsoft, Mistral, Alibaba, Hugging Face, Google, Reka AI, Cohere, Meta, 01 AI, AI21 Labs, Zhipu AI, Nvidia, DeepSeek, and xAI.
Since GPT-4's launch, 15 different companies have reportedly created AI models that are smarter than GPT-4. Among them are Reka AI, Meta, AI21 Labs, DeepSeek AI, Anthropic, Alibaba, Zhipu, Google, Cohere, Nvidia, 01 AI, NexusFlow, Mistral, and xAI.
Twitter AI (xAI), which seemingly had no prior history of strong AI engineering, with a small team and limited resources, has somehow built the third smartest AI in the world, apparently on par with the very best from OpenAI.
The top AI image [...]
---
First published:
August 28th, 2024
Source:
https://www.lesswrong.com/posts/yRjLY3z3GQBJaDuoY/things-that-confuse-me-about-the-current-ai-market
---
Narrated by TYPE III AUDIO.

“Principles for the AGI Race ” by William_S Aug 31, 2024

Show notes

Crossposted from https://williamrsaunders.substack.com/p/principles-for-the-agi-race
Why form principles for the AGI Race?
I worked at OpenAI for 3 years, on the Alignment and Superalignment teams. Our goal was to prepare for the possibility that OpenAI succeeded in its stated mission of building AGI (Artificial General Intelligence, roughly able to do most things a human can do), and then proceed on to make systems smarter than most humans. This will predictably face novel problems in controlling and shaping systems smarter than their supervisors and creators, which we don't currently know how to solve. It's not clear when this will happen, but a number of people would throw around estimates of this happening within a few years.
While there, I would sometimes dream about what would have happened if I’d been a nuclear physicist in the 1940s. I do think that many of the kind of people who get involved in the effective [...]
---
Outline:
(00:06) Why form principles for the AGI Race?
(03:32) Bad High Risk Decisions
(04:46) Unnecessary Races to Develop Risky Technology
(05:17) High Risk Decision Principles
(05:21) Principle 1: Seek as broad and legitimate authority for your decisions as is possible under the circumstances
(07:20) Principle 2: Don’t take actions which impose significant risks to others without overwhelming evidence of net benefit
(10:52) Race Principles
(10:56) What is a Race?
(12:18) Principle 3: When racing, have an exit strategy
(13:03) Principle 4: Maintain accurate race intelligence at all times.
(14:23) Principle 5: Evaluate how bad it is for your opponent to win instead of you, and balance this against the risks of racing
(15:07) Principle 6: Seriously attempt alternatives to racing
(16:58) Meta Principles
(17:01) Principle 7: Don’t give power to people or structures that can’t be held accountable.
(18:36) Principle 8: Notice when you can’t uphold your own principles.
(19:17) Application of my Principles
(19:21) Working at OpenAI
(24:19) SB 1047
(28:32) Call to Action
---
First published:
August 30th, 2024
Source:
https://www.lesswrong.com/posts/aRciQsjgErCf5Y7D9/principles-for-the-agi-race
---
Narrated by TYPE III AUDIO.

Our TOPPODCAST Picks

Follow Us

Stay Connected

LessWrong (Curated & Popular)

Related Podcasts

Links

Stay Connected