Show notes
In a world of Artificial General Intelligence, machines would be able to match, and even exceed, human cognitive abilities. AGI might still be science fiction, but Séb Krier sees this technology as not only possible, but inevitable. Today on Faster, Please! — The Podcast, I chat with Krier about how our public policy should facilitate AGI’s arrival and flourishing.Krier is an AI policy expert, adviser, and attorney. He currently works in policy development and strategy at Google DeepMind. He previously served as Head of Regulation for the UK Government’s Office for Artificial Intelligence and was a Senior Tech Policy Researcher at Stanford’s Cyber Policy Center.In This Episode* The AGI vision (1:24)* The risk conversation (5:15)* Policy strategy (11:25)* AGI: “if” or “when”? (15:44)* AI and national security (18:21)* Chatbot advice (20:15)Below is a lightly edited transcript of our conversationPethokoukis: Séb, welcome to the podcast.Krier: Thank you. Great to be here.The AGI vision (1:24)Let's start with a bit of context that may influence the rest of the conversation. What is the vision or image of the future regarding AI — you can define it as machine learning or generative AI — that excites you, that gets you going in the day, that you feel like you're part of something important? What is that vision?I think that's a great question. In my mind, I think AI has been going on for quite a long time, but I think the aim has always been artificial general intelligence. And in a sense, I think of this as a huge deal, and the vision I have for the future is being able to have a very, very large supply of cognitive resources that you can allocate to quite a wide range of different problems, whether that's energy, healthcare, governance, there's many, many ways in which this technology can be applied as a general purpose technology. And so I guess my vision is seeing that being used to solve quite a wide range of problems that humans have had for decades, centuries, millennia. And I think you could go into so many different directions with that, whether it's curing diseases, or optimizing energy grids, and more. But I think, broadly, that’s the way I think about it. So the objective, in a sense, is safe AGI [Artificial General Intelligence], and from that I think it can go even further. And I think in many ways, this can be hugely beneficial to science, R&D, and humanity as a whole. But of course, that also comes with ways in which this could be misused, or accidents, and so on. And so huge emphasis on the safe development of AGI.So you're viewing it as a tool, as a way to apply intelligence across a variety of fields, a variety of problems, to solve those problems, and of course, the word in there doing a lot of lifting is “safely.” Given the discussion over the past 18 months about that word, “safely,” is, one, I think someone who maybe only pays passing attention to this issue might think that it's almost impossible to do it safely without jeopardizing all those upside benefits, but you're confident that those two things can ultimately be in harmony?Yeah, absolutely, otherwise I wouldn't be necessarily working on an AGI policy. So I think I'm very confident this can be done well. I think it also depends what we mean by “safety” and what kind of safety we have in mind. Any technology, we will have costs and trade-offs, but of course the upside here is enormous, and, in my mind, very much outweighs potential downsides.However, I think for certain risks, things like potentially catastrophic risks and so on, there is an argument in treading some careful path and making sure this is done scientifically with a scientific method in mind, and doing that well. But I don't think there's fundamentally a necessary tension, and I think, in fact, what many people sometimes underestimate is how AI itself, as a technology, will be helpful in mitigating a lot of the risks we're foreseeing and thinking about. There's obviously ways in which AI can be used for cyber offense, but many ways in which you can also use that for defense, for example. I'm cautiously optimistic about how this can be developed and used in the long runThe risk conversation (5:15)Since these large language models and chatbots were rolled out to public awareness in late 2022, has the safety regulatory debate changed in any way? It seems to me that there was a lot of talk early on about these existential risks. Now I seem to hearing less about that and more about issues about, maybe it's disinformation or bias. From your perspective, has that debate changed and has it changed for the better, or worse?I think it has evolved quite a lot over the past — I've been working in AI policy since 2017 and there's been different phases, and at first a lot of skepticism around AI even being useful, or hype, and so on, and then seeing more and more of what these general models could do, and I think, initially, a lot of the concerns were around things like bias, and discrimination, and errors. So even things like, early-on, facial-recognition technologies were very problematic in many ways: not just ways in which they were applied, but they would be prone to a lot of errors and biases that could be unfair, whereas they're much better now, and therefore the concern now is more on misuse than it accidentally misidentifying someone, I would say. So I think, in that sense, these things have changed. And then a lot of the discourse around existential risk and so on, there was a bit of a peak a bit last year, and then this switched a bit towards more catastrophic risks and misuse.There's a few different things. Broadly, I think it's good that these risks are taken seriously. So, in some sense, I'm happy that these have taken more space, in a way, but I think there's also been a lot of alarmism and unnecessary doomerism, of crying wolf a little bit too early. I think what happens is that sometimes people also conflate a capability of a system and how that fits within a wider risk or threat model, or something; and the latter is often under-defined, and there's a tendency for people to often see the worst in technology, particularly in certain regions of the world, so I think sometimes a lot has been a little bit exaggerated or overhyped.But, having said that, I think it’s very good there's lots of research going on on the many ways in which this could potentially be harmful, certainly on the research side, the evaluation side, there’s a lot of great work. We've published some papers on sociotechnical evaluations, dangerous capabilities, and so on. All of that is great, but I think there has also been some more polarized parts calling for excessive measures, whether regulatory, or pausing AI, and so on, that I think have been a little bit too trigger-happy. So I'm less happy about these bits, but there's been a lot of good as well.And much of the debate about policy has been about the right sort of policy to prevent bad things from happening. How should we think about policy that maximizes the odds of good things happening? What should policymakers do to help promote AI to reshape science, to help promote AI diffusing as efficiently as possible throughout an economy? How do we optimize the upside through policy rather than just focusing on making sure the bad things don't happen?I think the very first thing is not having rushed regulation. I'm not personally a huge fan of the Precautionary Principle, and I think that, very often, regulations can cause quite a lot of harm downstream, and they're very sticky, hard to remove.The other thing that you can do beyond avoiding bad policy is I think a lot of the levers to making sure that the development goes well aren't necessarily all directly AI-related. So it'll be things like immigration: attracting a lot of talent, for example, I think will be very important, so immigration is a big one. Power and energy: you want there to be a lot more — I'm a big fan of nuclear, so I think that kind of thing is also very helpful in terms of the expected needs for AI development in the future. And then there are certain things governments could potentially do with some narrow domains like Advance Market Commitments, for example, although that's not a panacea.Commitments to do what?Oh, Advance Market Commitments like pull mechanisms to create a market for a particular solution. So like Operation Warp Speed, but you could have an AI equivalent for certain applications, but of course there's a lot of parameters in doing that well, and I wouldn't want a large industrial-policy-type approach to AI. But I think generally it's around ensuring that all the enablers, all the different ingredients and factors of a rich research and development ecosystem continue to thrive. And so I think, to a large extent, avoiding bad regulation and ensuring that a lot of things like energy, immigration, and so on go well is already a huge part of the battle.How serious of a potential bottleneck is the energy issue? It seems to me like it's a serious issue that's coming fast, but the solutions seem like they'll take more time, and I'm worried about the mismatch between the problem and finding a solution to the problem.I suspect that, over the coming years, we will see more and more of these AI systems being increasingly useful, capable, and then integrated into economic systems, and I think as you start seeing these benefits more and more, I think it'll be easier to make the case for why you need to solve some of these kind of policy issues a bit faster.And I also think these solutions aren't that difficult, ultimately. So I think there’s a lot that can be done around nuclear, and wind, and solar, and so on, and many regulatory processes that could be simplified, and accelerated, and improved to avoid the vetocracy system we're in at the moment. So I don't think the solutions are that difficult, I think mustering the political will might be right now, but I expect that to be less of a challenge in the coming years with AI showing more and more promise, I think.Policy strategy (11:25)Speaking of vetocracy, whatever the exact substance of the regulation, I am concerned, at least in the United States, that we have 50 states, and perhaps even more bodies if you look at cities, who all have a lot of ideas about AI regulation, and I'm extremely concerned that that sort of fractured policy landscape will create a bottleneck.Can we get to where we need to go if that's the regulatory environment we are looking at, at least in the United States? And does, ultimately, there need to be a federal . . . I think the technical word is “preemption” of all those efforts? So there's a federal approach, and there aren't a federal approach, plus a 50-state approach, plus a 175-city approach to regulation. Because if it's going to be what I just described, that seems like a very difficult environment to deal with. I'm not wildly optimistic around a patchwork of different state-level regulatory systems. I think that will come with various externalities, you'll have distortionary effects. It will be a very difficult environment, from a commercial perspective, to operate in smoothly. I think I'm a lot more open to something at a federal level at some point, rather than a big patchwork of city-level or state-level regulation. Now, it depends on exactly what we're talking about. There might be specific domain, and context, and application-specific regulations that might make sense in some state and not another, but in general, from a first principles level at least, I think that would probably not be desirable.A second regulatory concern — and maybe this is dissipating as policy makers learn more, especially at the federal level, maybe, learn more about AI — is that, at least initially, it seems to me that whatever your policy idea was for social media, or about content moderation or what have you, you just kind of took that policy framework and applied it to AI because that was what you had. You pulled that baby right off the shelf. Are we still seeing that, or are people beginning to think, “This is its own thing, and my ideas for social media may be fine for social media, but I need to think differently about AI”? Obviously the technology is different; also, I think both the risks and potential rewards are very different.Yeah, totally. I think that has been an issue. Now, I wouldn't say that's the case for everyone. There's been some groups and some institutions doing some very careful work that really think about AI, and AGI, and so on in careful, more calibrated ways; but also I’ve seen quite a lot of reports where you could have easily imagined the same text being about social media platforms, or some kind of other policy issue, or blockchain, or something just being repurposed for AI. And there's a lot of stuff out there that's just very high level, and it's hard to disagree with at a high level, but it’s far harder to apply and look at from an operational or practical perspective.So I've been seeing quite a lot of that; however, I think over time, the field is maturing more and more, and you're seeing better thinking around AI, what it really is, what's appropriate at the model level versus at the application level and the existing landscape of laws and regulation and how these might apply as well, which is often that's something that's forgotten, or you have lots of academics coming in and just trying to re-regulate everything from first principles, and then you're like, “Well, there's tort law, and there's this and that over there.” You got to do your gap analysis first before coming out with all this stuff.But I think we are seeing the field of AI governance and policy maturing in that space, and I expect it to continue, but I still, of course, see a lot of bad heuristics and poor thinking here, and particularly an underestimation of the benefits of AI and AGI. I think there's a tendency to always think of the worst for everything, and it's necessary, you need to do that too, but few are really internalizing how significant AGI would be for growth, for welfare, and for solving a lot of the issues that we've been talking about in the first place.The Conservative Futurist: How To Create the Sci-Fi World We Were PromisedAGI: “if” or “when”? (15:44)Is AGI an “if” issue, or is it a “when” issue, and if it's a “when,” when? And I say this with the caveat that predictions are difficult, especially about the future.In my mind, it's definitely a “when” question. I see no real strong reason why it would be an “if,” and that being completely impossible. And there's been many, many, many examples over the last 10 years of people saying, “Well, this is not possible with neural networks,” and then 10 minutes later, it is proven to be possible. So that's a recurring theme, and that may not be sufficient to think that AGI is feasible and possible, but I'm pretty confident for a variety of reasons. About AGI, by the way, I think there's an excellent paper by Morris and others on Levels of AGI [for] Operationalizing Progress on the Path of AGI, and I think it's a very good paper to [frame one’s thinking about AGI].And that goes back to one point I made earlier in that, at some point, you'll have systems that will be capable of quite a lot of things and can do probably anything that your average human can do, starting at least virtually, remotely, to start with, and eventually to the physical world, but I think they'll be capable in that sense. Now, there's a difference between th…
Full show notes at the publisher 
Our TOPPODCAST Picks 
Stay Connected
