Manifold

Latest Episodes:

AI on your phone? Tim Dettmers on quantization of neural networks — #41 Aug 10, 2023

Show notes

Tim Dettmers develops computationally efficient methods for deep learning. He is a leader in quantization: coarse graining of large neural networks to increase speed and reduce hardware requirements.

Tim developed 4-and 8-bit quantizations enabling training and inference with large language models on affordable GPUs and CPUs - i.e., as commonly found in home gaming rigs.

Tim and Steve discuss: Tim's background and current research program, large language models, quantization and performance, democratization of AI technology, the open source Cambrian explosion in AI, and the future of AI.

0:00 Introduction and Tim’s background

18:02 Tim's interest in the efficiency and accessibility of large language models

38:05 Inference, speed, and the potential for using consumer GPUs for running large language models

45:55 Model training and the benefits of quantization with QLoRA

57:14 The future of AI and large language models in the next 3-5 years and beyond

Tim's site: https://timdettmers.com/

Tim on GitHub: https://github.com/TimDettmers

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (SuperFocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Announcing this for some friends at Mechanize - a startup that builds environments for training and evaluating frontier LLMs. Its customers include the top AI labs, and it has contributed to the breakthrough in coding capabilities of frontier models.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Paul Huang, the real situation in Taiwan: politics, military, China — #40 Jul 27, 2023

Show notes

Paul Huang is a journalist and research fellow with the Taiwan Public Opinion Foundation. He is currently based in Taipei, Taiwan.

Sample articles:

Taiwan’s Military Has Flashy American Weapons but No Ammo (in Foreign Policy): https://foreignpolicy.com/2020/08/20/taiwan-military-flashy-american-weapons-no-ammo/

Taiwan’s Military Is a Hollow Shell (Foreign Policy): https://foreignpolicy.com/2020/02/15/china-threat-invasion-conscription-taiwans-military-is-a-hollow-shell/

Steve and Paul discuss:

0:00 Introduction

1:44 Paul’s background; the Green Party (DPP) and Blue Party (KMT) in Taiwan

4:40 How the Taiwanese people view themselves vs mainland Chinese

15:02 Taiwan taboos: politics and military preparedness

15:27 Effect of Ukraine conflict on Taiwanese opinion

29:56 Lack of realistic military planning

37:20 Is there a political solution to reunification with China? What influence does the U.S. have?

51:34 The likelihood of peaceful reunification of Taiwan and China

56:45 Honest views on Taiwanese and U.S. military readiness for a

conflict with China

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Richard Hanania & Rob Henderson: The Rise of Wokeness and the Influence of Civil Rights Law — #39 Jul 13, 2023

Show notes

Steve Hsu, Richard Hanania, and Rob Henderson were scheduled for a June 2023 panel as part of the University of Austin (UATX) Forbidden Courses series. Steve missed the panel due to travel issues, but the three have gathered on this podcast to recreate the fun!

They discuss:

0:00 Introduction
1:20 The University of Austin and forbidden courses
17:37 Will woke campus culture change anytime soon?
29:57 Common people vs elites on affirmative action
35:42 Why it’s uncomfortable to disagree about affirmative action
41:22 Fraud and misrepresentation in higher ed
44:20 The adversity carveout in the Supreme Court affirmative action ruling
50:10 Standardized testing and elite university admissions
1:06:18 Divergent views among racial and ethnic groups on affirmative action; radicalized Asian American males
1:10:00 Differences between East and South Asians in the West
1:23:03 Class-based preferences and standardized tests
1:31:57 Rob Henderson’s next move

LINKS

Richard Hanania’s new book: The Origins of Woke: Civil Rights Law, Corporate America, and the Triumph of Identity Politics: https://www.harpercollins.com/products/the-origins-of-woke-richard-hanania?variant=41004650528802

Richard Hanania’s newsletter: https://www.richardhanania.com/

The Center for the Study of Partisanship and Ideology: https://www.cspicenter.com/

Rob Henderson’s newsletter: https://www.robkhenderson.com/

Rob Henderson’s new book: Troubled: A Memoir of Foster Care, Family, and Social Class: https://www.simonandschuster.com/books/Troubled/Rob-Henderson/9781982168537

UATX: https://www.uaustin.org/forbidden-courses

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Richard Sander (UCLA Law) on the Supreme Court Affirmative Action Ruling — #38 Jun 30, 2023

Show notes

Richard Sander is Jesse Dukeminier Professor at UCLA Law School. AB Harvard, JD, PhD (Economics) Northwestern.

Steve and Richard discuss the recent Supreme Court ruling in Students For Fair Admissions vs Harvard and UNC.

Sander has studied the structure and effects of law school admissions policies. He coined the term "Mismatch" to describe negative consequences resulting from large admissions preferences.

0:00 Introduction

1:09 Richard Sander’s initial reaction to the Supreme Court ruling

4:03 How data influenced the court’s decision

7:58 Overview of the court’s ruling

11:27 Carve outs in the court’s ruling

16:59 The litigation landscape

21:25 Workarounds to race-blind admissions and the UC system

32:22 Remedies: What will happen with Harvard and UNC now?

38:02 The landscape of college admissions

44:47 Effects of the Supreme Court ruling beyond higher education

LINKS

SCOTUS decision on Affirmative Action:

https://int.nyt.com/data/documenttools/supreme-court-decision-on-race-based-admissions/0a725aaabb459074/full.pdf

Richard Sander’s amicus brief: https://www.supremecourt.gov/DocketPDF/20/20-1199/222805/20220509134743957_20-1199%2021-707%20Amicus%20BOM.pdf

Richard Sander on SCOTUS Oral Arguments: Affirmative Action and Discrimination against Asian Americans at Harvard and UNC: https://www.manifold1.com/episodes/richard-sander-on-scotus-oral-arguments-affirmative-action-and-discrimination-against-asian-americans-at-harvard-and-unc

Richard Sander: Affirmative Action, Mismatch Theory, and Academic Freedom: https://www.manifold1.com/episodes/richard-sander-affirmative-action-mismatch-theory-academic-freedom-6

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

AI Cambrian Explosion: Conversation With Three AI Engineers — #37 Jun 08, 2023

Show notes

In this episode, Steve talks to three AI engineers from his startup SuperFocus.AI.

0:00 Introduction

1:06 The Google memo and open-source AI

14:41 Sparsification and the size of models: AI on your phone?

30:16 When will AI take over ordinary decision-making from humans?

34:50 Rapid advances in AI: a view from inside

41:28 AI Doomers and Alignment

Links to earlier episodes on AI and LLMs.

Artificial Intelligence & Large Language Models: Oxford Lecture — #35: https://www.manifold1.com/episodes/artificial-intelligence-large-language-models-oxford-lecture-35

Bing vs. Bard, US-China STEM Competition, and Embryo Screening — #30: https://www.manifold1.com/episodes/bing-vs-bard-us-china-stem-competition-and-embryo-screening-30

ChatGPT, LLMs, and AI — #29: https://www.manifold1.com/episodes/chatgpt-llms-and-ai

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

David Goldman: US-China Competition, AI, Electric Vehicles, and Manufacturing — #36 May 25, 2023

Show notes

David Paul Goldman is an American economic strategist and author, best known for his series of online essays in the Asia Times under the pseudonym Spengler with the first column published January 1, 2000.

Steve and David discuss:

0:00 Introduction

2:22 David’s background in music, finance, and Asia

16:55 Looking back at the financial crisis

23:04 Rise of the Chinese economy

29:44 How Huawei’s strength is tied to China’s economic power

36:49 Competition in the global electric vehicles market

38:06 Why David thinks European countries like Germany will become closer with China

45:29 U.S. manufacturing is falling behind

52:08 Potential for war and ongoing U.S.-China competition

1:04:07 Predictions for Taiwan

Links:

David Goldman in Wikipedia:

https://en.wikipedia.org/wiki/David_P._Goldman

Spengler column:

https://asiatimes.com/author/spengler/

You Will Be Assimilated: China's Plan to Sino-form the World

https://www.amazon.com/You-Will-Be-Assimilated-Sino-form/dp/1642935409

Prisoner’s Dilemma: Avoiding war with China is the most urgent task of our lifetime

https://claremontreviewofbooks.com/prisoners-dilemma/

David Goldman articles in Claremont Review:

https://claremontreviewofbooks.com/author/david-p-goldman/

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Artificial Intelligence & Large Language Models: Oxford Lecture — #35 May 11, 2023

Show notes

This week's episode is based on a lecture Steve gave to an audience of theoretical physicists at Oxford University. The topic is artificial intelligence and large language models.

Lecture slides: https://docs.google.com/presentation/d/1xiMeeRMVpB-_W66BnyRyUAtrLlUwQNlndqbVcguKK8U/edit?usp=sharing

Chapter markers:

0:00 Introduction

2:31 Deep Learning and Neural Networks; history and mathematical results

21:15 Embedding space, word vectors

31:53 Next word prediction as objective function

34:08 Attention is all you need

37:09 Transformer architecture

44:54 The geometry of thought

52:57 What can LLMs do? Sparks of AGI

1:02:41 Hallucination

1:14:40 SuperFocus testing and examples

1:18:40 AI landscape, AGI, and the future

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Simone Collins: IVF, Embryo Selection, Dating on the Spectrum, and Pronatalism — #34 Apr 27, 2023

Show notes

In collaboration with her husband Malcolm Collins, Simone is an author (The Pragmatist's Guide to Life, Relationships, Sexuality, Governance, and Crafting Religion), education reform advocate (CollinsInstitute.org), pronatalism activist (Pronatalist.org), and business operator (Travelmax.com).

Note: the YouTube version of this interview includes video of Steve and Simone.

Steve and Simone discuss:

0:00 Introduction

1:49 Simone's IVF journey, and embryo screening

40:02 Dating; girl autists

55:41 Finding a husband, systematized

1:09:57 Pronatalism

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

—

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Katherine Dee: Culture, Identity, and Isolation in the Digital Age — #33 Apr 13, 2023

Show notes

Katherine Dee is a writer, journalist, and internet historian.

Steve and Katherine discuss:

0:00 Introduction

1:15 Katherine’s early life and background

21:52 Mass shootings, Manifestos, Nihilism, and Incels

59:35 Trad values, Sex negativity vs Porn and Fleshlights

1:28:54 Elon Musk’s plans for Twitter

1:33:00 TikTok

1:41:41 Adderall

1:44:07 AI/GPT impact on writers and journos

1:49:30 Gen-X generation gap: are the kids alright?

References:

Katherine’s Substack: https://defaultfriend.substack.com/

“Mass Shootings and the World Liberalism Made”: https://contra.substack.com/p/mass-shootings-and-the-world-liberalism

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU.

Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Marc Martinez: "Dream Big" and the Golden Age of Bodybuilding — #32 Mar 16, 2023

Show notes

Marc Martinez is the director of Dream Big, a documentary about Gold's Gym and the golden age of bodybuilding in Venice and Santa Monica in the 1970s.

Steve and Marc discuss:

(01:34) - Marc's background in bodybuilding
(05:36) - Reflections on bodybuilding in Southern California
(26:03) - Setting the record straight on steroid use
(33:52) - Frank Zane
(38:33) - Robby Robinson
(40:32) - Butler, Gaines, and Arnold
(42:46) - "Dream Big"
(48:18) - Pumping Iron
(59:44) - Hypersexuality in bodybuilding
(01:10:56) - What's next for Marc

References:

Watch Dream Big on Amazon: https://www.amazon.com/Dream-Big-Ken-Sprague/dp/B0B8ST5LNL/
Dream Big documentary: https://dreambigdoc.com/
Dream Big trailer: https://www.youtube.com/watch?v=X22ISDn083A
Pumping Iron: https://www.youtube.com/watch?v=6lCCk6rgn84

Music used with permission from Blade Runner Blues Livestream improvisation by State Azure.

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (Superfocus.ai, SafeWeb, Genomic Prediction) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU. Please send any questions or suggestions to manifold1podcast@gmail.com or Steve on Twitter @hsu_steve.

Mechanize is hiring!

https://mechanize.work/b/hsu

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists.

For example:

Research Engineer, Alignment: Build evals that test for misaligned model behaviors $500K salary

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve $300K salary

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Our TOPPODCAST Picks

Follow Us

Stay Connected

Related Podcasts

Links

Stay Connected