Interview with 7oalk

\*\*Interview with 7oalk, on 3/20/22\*\*

\*\*0:00:03.2 Vael:\*\* Alright, my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:08.5 Interviewee:\*\* Yeah, I study biologically-inspired artificial intelligence of building models of biological intelligence, mostly with the visual system, but also with cognitive functions, and using those models to understand differences between humans and machines. And upon finding those differences, the hope is that we can build more human-like artificial intelligence and in the process develop models that can better explain the brain.

\*\*0:00:46.1 Vael:\*\* Interesting, yep. I was in that space for a while. The dual\-- AI\--

\*\*0:00:49.1 Interviewee:\*\* Oh, cool. I saw you worked with Tom Griffiths.

\*\*0:00:54.0 Vael:\*\* Yeah, that\'s right, yep.

\*\*0:00:55.0 Interviewee:\*\* Yeah, very much so. Cool.

\*\*0:00:56.5 Vael:\*\* Alright, so my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:01:06.9 Interviewee:\*\* So I\'m most excited about AI, not for most of the applications that are getting a lot of praise and press these days, but more so for convergences to biomedical science. I think that the main driver of progress in AI over the past decade has been representation learning from large-scale data. And this is still an untapped area for biomedical science, so we really don\'t know what elements of biology you can learn predictive mappings for. So, for instance, given an image of a cell\'s morphology, can you predict its genome? Can you predict its transcriptome? Etcetera. So I think that what I\'m most excited about is the potential for AI to completely transform the game of drug discovery, of our ability to identify the causes and identify targets for treating disease.

\*\*0:02:18.4 Vael:\*\* Nice.

\*\*0:02:18.7 Interviewee:\*\* What I\'m most afraid of is in the game of AI that is most popular right now. The NeurIPS CDPR game, which\... lots of people in the field have pointed out the issues with biases. And also\... it\'s late here, so I\'m not thinking of a good term, but this Clever Hans, Mechanical Turk-ish nature of AI, where the ability to solve a problem that seems hard to us can give this sheen of intelligence to something that\'s explained a really trivial solution. And sometimes those trivial solutions, which are available to these deep learning networks which exploit correlations, those can be really painful for people. So in different applications of using AI models for, let\'s say screening job applicants, there\'s all these ethical issues about associating features not relevant to a job, but rather just regularities in the data with a predictive outcome. So that\'s a huge issue. I have no expertise in that. It\'s definitely something I\'m worried about. Something that\'s related to my work that I\'m super worried about\... I mentioned I do this biological to artificial convergence, this two-way conversation between biological and artificial intelligence. The best way to get money, funding in academia, for that kind of research is going through defense departments. So one of the ARPA programs\-- IARPA, DARPA, Office of Naval Research. So I know that the algorithms built for object tracking, for instance, could be extremely dangerous. And so I build them to solve biological problems, and then I scale them to large-scale data sets, large data sets to get into conferences. But, you know, easily, somebody could just take one of those algorithms and shove them into a drone. And you do pretty bad stuff pretty easily.

\*\*0:04:41.2 Vael:\*\* Yeah.

\*\*0:04:43.2 Interviewee:\*\* Yeah. I guess one more point related to the biomedical research. There\'s this fascinating paper in Nature and Machine Intelligence. One of the trends right now for using AI in biology is to predict protein folding, or to predict target activity for a molecule or a small molecule. So when you do biomedical research, you want to screen small molecules for their ability to give you a phenotype in the cell. And so you can just use a multilayer perceptron or graph neural network, transformer, whatever, to learn that mapping, and do a pretty good job it turns out; it\'s shocking to me. But what you can do is you can either predict\-- let\'s find the molecule that\'s going to have the best target site activity, that\'s the best therapeutic candidate. But you can also flip it, you can optimize it in the opposite direction. (Vael: \"Yeah\--\") You saw that.

\*\*0:05:48.2 Vael:\*\* Yeah. I saw that \[paper\].

\*\*0:05:49.0 Interviewee:\*\* It\'s so obvious, but easily, easily, easily you could use any of these biomedical applications for the nefarious things.

\*\*0:06:00.5 Vael:\*\* Yeah, yeah, that\'s right. That was a very recent paper right?

\*\*0:06:02.5 Interviewee:\*\* Yeah, it just came out like last week.

\*\*0:06:05.1 Vael:\*\* Got it. Yeah. Alright, we\'ve got excited and risks. My next question is talking about future AI. So putting on a science fiction, forecasting hat. So we\'re 50 plus years into the future, so at least 50 years in the future, what does that future look like?

\*\*0:06:22.6 Interviewee:\*\* Okay. Optimistically, we\'re all still here. So the reason why I got into this field is because I think it\'s the biggest force multiplier there is. So all of the greatest problems that\-- the existential crises we face today, there\'s a potential for machine learning to be part of the solution to those problems. So focusing just for a moment on health, biomedical research, which is where I\'m mostly interested right now, 50 years, I think that we\'re going to be to a point where we know we have drugs that are completely uninterpretable to humans. We have solved disease, we have cured disease, but we don\'t know why. We\'ve also created a potential issue, byproducts from that, adversarial effects, for lack of a better phrase, by curing cancer through some black box method. That\'s going to yield other problems, kind of like putting your finger in the dam. So I think that in 50 years from now, we will have machine learning-based solutions to some of the greatest health problems: cancer, aging, neurodegenerative disease that face us, but we won\'t understand those solutions. And so that\'s potentially the next frontier, which is to develop interpretable drugs. So I imagine that science will change. The paradigm of science will shift where there are no longer bench scientists, but instead robotics is how science is done. So scientists ask high-level questions, and then biology becomes a search problem where you just, akin to a tricorder from Star Trek where it just solves, and you move onto the next question. You can imagine that framework being applied to other problems. Climate change, hopefully. I\'m not optimistic, but hopefully, you could have solutions to climate change. But there will be a black box. And when you have a black box, there\'s always issues with unintended effects. So I guess 50 years from now, we\'ll have, rather than thinking about AGI and all that stuff, we\'ll have solutions to some of our grandest challenges today, but those solutions may bear costs that are unexpected.

\*\*0:09:40.8 Vael:\*\* Yeah, that makes sense. I have few follow-ups. So you mentioned existential risks facing humanity. What are those?

\*\*0:09:51.5 Interviewee:\*\* So there\'s climate change. There\'s nuclear war, which is the byproduct of power. There is poverty and hunger, famine. With the risk of sounding like some VC, I think that there are chances to disrupt solutions to each of those problems using machine learning. For instance, for energy, already folks at DeepMind are using reinforcement learning to drive the development of new forms of fusion to search for better forms of fusion. Who knows if this is scalable, but I think with\... it\'s not going to be better algorithms, it\'s going to be better integration of the toolset for studying fusion and these machine learning methods to make it more rapid to test and explore those questions that are related to that field, which I don\'t know. I think that will lead to positive solutions. That would potentially yield lower-emission or no-emission, high-energy solutions that can power homes without any emissions. Similarly, with cars\... self-driving, as much as I think that Elon Musk is barking up the wrong tree\-- going LIDAR-free, \[inaudible\] approach to self-driving, that will be a solved problem within 10 years. And that\'s going to help a lot the emissions issue. So energy and global warming are kind of tied together. And I think that there\'s lots of different applications of machine learning and AI that can help address those problems. Likewise for famine, for world hunger. Genetically-engineered organisms are a good approach to that problem. What I mentioned about for solutions to biomedical problems, disease, etc.\-- treating those as search problems and bringing to bear the power of artificial intelligence. I think you could probably adopt a similar approach for engineering organisms that are tolerant to weather, especially the changing climate and other diseases and bugs, etcetera. I think there\'s going to be some overlap between the methods of work for each of these areas. I guess I didn\'t mention COVID and pandemics. But again, that falls into the same framework of treating these existential problems as search problems, no longer doing brute force, then science. And instead resorting to black box solutions that are discovered by large scale neural network algorithms. Did I miss any? Oh, war. Yeah, I don\'t know. Okay, so here\'s an answer. Whoever has these technologies, who\'s ever able to scale these technologies most rapidly, that\'s going to be almost like an arms race. So whoever has these technologies will be able to generate so much more value than countries that don\'t have these technologies that economies will be driven by the ability to wield appropriate artificial intelligence tools for searching for solutions to these existential issues.

\*\*0:14:36.1 Vael:\*\* Got it, thanks. Alright, so my next question is\... so this is kind of a spiel. Some people talk about the promise of AI, by which they mean many things, but that the thing I\'m referring to is something like a very general capable system. So the cognitive capacities that can be used to replace all current day human jobs. Whether or not we choose to replace human jobs is a different question, but having the kind of capabilities to do that. So like, imagine\... I mostly think about this in the frame of 2012, we have the deep learning revolution with AlexNet, and then 10 years later, here we are, and we have systems like GPT-3, which have kind of weirdly emergent capabilities, that can be used in language translation and some text generation and some coding and some math. And one might expect that if we can continue pouring all of the human effort that we\'ve been pouring into this, with all the young people and the nations competing and corporations competing and algorithmic improvements at the rate we\'ve seen, and like hardware improvements, maybe people get optical or quantum, that we might get\... scale to very general systems, or we might hit some sort of ceiling and need to do a paradigm shift. But my question is regardless of how we\'d get there, do you think we\'ll ever get very general systems like a CEO AI or a scientist AI and if so, when?

\*\*0:15:55.3 Interviewee:\*\* I don\'t think it\'s worth it. I see the AI as\... look, I grew up reading Asimov books, so I love the idea of \"I, Robot\", not with Will Smith, but solving all these galaxy conquest issues and questions. But I don\'t think it\'s worth it because\... I see this as a conversation between human experts, the main experts, and artificial intelligence models that are just going to force multiply the ability of human experts to explore problems. And maybe in a 100 years, when we\'ve explored all of these problems, then we\'ll be so bored that we say, \"Can we engineer something that contains all of these different domain expert AIs in one, in a way that can\-- those AIs can respond to a question and affect behavior in the world and have their own free will.\" It\'s more of a philosophical pursuit that I don\'t even, I don\'t\... okay, so from an economic standpoint, I don\'t know who\'s going to pay for that. Even the military would be like, \"All I want is a drone that\'s going to kill everybody.\"

\*\*0:17:30.9 Vael:\*\* Yeah, Yeah, cool. So I have some counterpoints to that. So I\'m not actually imagining necessarily that we have AI that is conscious or has free will or something. I think I\'m just imagining an AI that is, has a, very capable of doing tasks. If you have a scientist AI then\...We were talking about automating science, but in this case, it\'s not even relying on an expert anymore. Maybe it defers to an expert. It\'s like, do you want me\... it looks to the expert\'s goals, like, \"Do you want me to solve cancer?\" Or like a CEO, it has shareholders. And I think that there is actually economic incentive to do this. DeepMind and OpenAI have explicitly said\... I think that they\'re trying to make artificial general intelligence, or like things that can replace jobs, and more and more replace jobs. And I think we\'re going in that direction with Codex and stuff, which is currently an assistant tool, but I wouldn\'t be surprised if in the end, we didn\'t do much coding ourselves.

\*\*0:18:24.9 Interviewee:\*\* Okay, so yeah, I agree with what you said. Right, it\'s going to change the job\... So strike from the record what I just said. It\'s going to change the job requirements. That would be a good feature in my opinion, where we no longer have to code because instead we describe our program. And again, it\'s a search problem to find the right piece of code to execute the concept that we have. We describe the medical problem that we\'re interested in, and we search within whatever imaging modality or probe that we\'re using to study that system. We can search for that solution. Where I\'m not so sure is, we have this huge service industry, and you mentioned shareholders and I count that as one of them. I think when it comes to service, there\'s this quality, there\'s this subjectiveness where for instance, in medicine, talking about radiology or pathology or even family medicine. I think there will be tools to give you diagnoses, but they\'re always going to come with a human second opinion. So, from that standpoint, it\'s going to be a bit of a science fair, kind of a sideshow to the expert. I don\'t think that human experts in service, medical or business, etcetera, are going to be pushed aside by AI, because ultimately, I think if there\'s a different answer given by the artificial pathologist versus the human pathologist, the human pathologist will double check and say, \"You are wrong, AI\" or \"You are right, AI\". But the patient will never receive the artificial intelligence answer. So I think in some fields, there\'s going to be a complete paradigm shift in how research is done, or how work is done, research for one. Driving, supply chain, transportation, that\'s another example. Which will be great. I think that would be great, it will push down costs, push down fatalities, morbidity, and probably be a net good. But yeah, for a shareholder, I cannot imagine a scenario in which a CEO who has to figure out the direction of the company, has to respond to shareholders would ever say, like, \"This time around, I\'m just going to give an AI a GPT-6 report for the markets that we should enter, the new verticals we should work on, etcetera.\" Maybe it would be just be a tool to get feedback.

\*\*0:21:54.2 Vael:\*\* Yeah. So it seems pretty true to me that we started\... our current level of AI, which is machine learning, that they started as tools and we use them as tools. I do\-- it sort of feels to me that as we progr\--\... we\'ve been working on AI for under 100 years, and deep learning revolution is quite new. And it seems to me like there are economic incentives towards making things more convenient, towards more automation, towards freeing up human labor to do whatever it is that people could do. So I wouldn\'t be surprised if eventually we reached a point where we had systems that were capable enough to be CEO. We could be like, \"Alright AI, I want you to spin me up a company,\" and maybe you would be able to do that in some very different future, I would imagine.

\*\*0:22:38.2 Interviewee:\*\* Ah, so see\-- okay, so this is a question. Creativity versus capturing the regularities or the regulations of a field. So I can imagine an AI VC, easy. Or in finance, yeah. Finance, I can imagine that being overtaken by AI. I\'m thinking CEO, spinning up your own company, identifying weakness in a field, or maybe some potential tool that can monopolize a field. That requires creativity. And I know there\'s this debate about creativity versus just interpolation in high-dimensional space of what GPT-3 is doing, and yes, I\'ve seen some cool-looking art, but I don\'t know. That seems like a\... I think if you talk to me about this, if we talked about this for another week, maybe you would convince me. But I\'m not quite there yet that I can imagine deep learning being able to do that. Although, now that I say that, maybe I\'m just thinking about it wrong. Where most businesses just amount to, \"What has worked somewhere else? Can that frame of reference work in this field? Can you disrupt this field with what\'s worked there?\" So maybe, maybe. Okay, I agree with what you\'re saying. I\'m evolving here, on the spot. I can see it, I can see that. I guess the one field that completely, that I\'m going to push back on is medicine, where I think that part of service need a human to tell you what is up. If you don\'t have a human, then I can\'t imagine a machine diagnosis would ever fly. Maybe in 200 years, that would fly. There would have to be a lot of development of the goal for that to happen.

\*\*0:24:45.3 Vael:\*\* Yeah, I think there\'s a number of things going on here. So I don\'t know that we\'ll be able to achieve\... I do think a CEO is very high cognitive capabilities, you need to be modeling other people, you need to be talking to other people, you need to be modeling other people modeling you, you need to have multi-step planning, you need to be doing a whole bunch of very advanced\...

\*\*0:25:03.6 Interviewee:\*\* Yeah, convincing people, interpersonal relationships. Yeah.

\*\*0:25:06.2 Vael:\*\* Yeah, there\'s a whole bunch of stuff there, and I don\'t that we\'re\... we\'re not near there with today\'s systems for sure. And I don\'t know that we can get there with the current deep learning paradigm, maybe we\'ll need something different. I think that\'s a possibility. I think what I\'m doing is zooming way out where I\'m like, \"Okay, evolution evolved humans, we have intelligence.\" I think that humans are trying pretty hard to make this thing happen, make up whatever intelligence is happen. And like, if we just continue working on this thing, absent some sort of huge disaster that befalls humanity, I kind of expect that we\'ll get there eventually.

\*\*0:25:39.6 Interviewee:\*\* Yeah.

\*\*0:25:40.2 Vael:\*\* And it sounds like, you said that like, maybe eventually, in like 200 years or something?

\*\*0:25:45.1 Interviewee:\*\* Definitely, definitely. So, I agree with what you said, science is incremental. And I think we have not found any ceilings in the field of modeling intelligence. We\'ve continued to move. Even though we\'re kind of arguably asymptoting with the current paradigm of deep learning, we have a proof of concept that it\'s possible to do better, which is our own brains. And so, maybe we just need to have neuromorphic hardware or whatever, who knows, who knows? So I think you\'re right, and if you can operationalize, just like you did very quickly, what it takes to be a CEO. If you can build models of each of those problems and if you can do what people do, which is benchmark those models and just start to make them better and better, yeah, I can imagine having some\... It sounds crazy to even visualize that, but yeah, having an automated CEO, I think so. I think the part that\'s probably the most tractable is finding the emerging markets or the, like I said, the technologies that have proven to be flexible, flexibly applied, or the tricks, let\'s say, that have worked in multiple fields, and then identifying the field where it has just not been applied yet. Kind of that first mover type advantage. I think that\'s where AI could\... I can definitely imagine an AI identifying those opportunities. Almost like drug discovery, right? It\'s just a hit. And you\'d have a lot of bad hits, and like, \"This is bullshit. Nobody cares about knee pads on baby pants,\" which is like the WeWork founder before WeWork, but you also have some good hits. So the interpersonal stuff, yeah, that\'s like 200 years off.

\*\*0:27:50.6 Interviewee:\*\* Although you have chat bots. So I\'m working on my own startups, a lot of this is trying to convince people to spend their time, that this is worthwhile, that they should leave their current stable positions to do this, that they should work\... They should do the work for you. They should drop what they\'re doing and do what you need them to do right now, and that you\'re going to help them eventually. And a lot of that is empathy and connection, and there\'s no evidence that we have yet that we can build a model that can do that stuff. So yeah, that\'s a really interesting idea, though. And I do think if you go through and operationalize each of those problems, you could make progress. So, 200 years, yes, I agree with you.

\*\*0:28:56.7 Vael:\*\* Alright, so maybe we\'ll get something like this in 200 years.

\*\*0:29:00.1 Interviewee:\*\* Yeah.

\*\*0:29:00.6 Vael:\*\* This is mainly an argument about the belief in\... like, faith in humans to follow economic incentives towards making life more convenient for themselves. That\'s where it feels like it\'s coming from in my mind. Like 200 years\... Humans have existed for a long time, and things are moving very quickly. 10,000 years ago, nothing changed from year to year, from lifetime to lifetime. Now, things are moving very quickly, and I\'m like, \"Hm.\"

\*\*0:29:23.1 Interviewee:\*\* I\'m happy to talk with you, like, add time to this interview. But I\'m just curious, what do you think? Do you think it\'s going to be sooner than 200 years? 10 years? 20 years?

\*\*0:29:37.1 Vael:\*\* I think there\'s a possibility that it could happen\... sooner than 200 years. \[chuckle\] I\'m trying to figure out how much of my personal opinion I want to put in here. There\'s a study, which was like, \"Hello, AI researchers. What do you think timelines?\" And there\'s several other kind of models of this, and a lot of the models are earlier than 70 years. So, this is\... possibilities. I think it could be within the next 30 years or something. But I don\'t actually know, there\'s like probability estimates over all of these things. I don\'t know how these things work.

\*\*0:30:13.2 Interviewee:\*\* Okay. Yeah. Some McKinsey consultant put like little probability estimates.

\*\*0:30:18.9 Vael:\*\* Yeah, a little. It\'s a little bit more than that. There\'s a lot of surveying of AI researchers and then some people have some more fancy models, I can send you to them afterwards, you can see\...

\*\*0:30:27.3 Interviewee:\*\* Please, please. Yeah.

\*\*0:30:28.6 Vael:\*\* Great.

\*\*0:30:29.4 Interviewee:\*\* And like I said, I\'m happy to add time to this, so\...

\*\*0:30:32.3 Vael:\*\* Awesome. Alright, so my next question is talking about these very\... highly intelligent systems. So in your mind, maybe like 200 years in the future. And so I guess, yeah, this argument probably feels a little bit more close to home if your timelines are shorter, but like, regardless. So, imagine we\'re talking about the future. We have our CEO AI, and I\'m like, \"Alright, CEO AI, I wish for you to maximize profits and try not to run out of money or try not to exploit people and try to not have these side effects.\" And this currently, obviously, is very technically challenging for many reasons. But one of the reasons I think, is that we\'re currently not very good at taking human values and human preferences and human goals, and then putting them into mathematical formulations such that AIs can optimize over them. And I worry that AIs in the future will continue to do what we tell them to do and not as we intend them to do. So, we\'re not that we won\'t be able to take all of our preferences\... it will continue to be hard to take our preferences and put them into something that can optimize over. So what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous\"?

\*\*0:31:46.6 Interviewee:\*\* \...Yeah. Well, so\... Okay. Exactly what they\'re intended to do, so presumably this means, you start to test generalization of your system and some of holdout set, and it does something funky on that set. Or there\'s some attack and it does something funky. The question is\... so presumably it\'s going to do superhuman\... well, let\'s just imagine it\'s superhuman everywhere else, except for this one set. Now it\'s a real issue if humans are better than this AI on that one problem that it\'s being tested on, this one holdout problem\...

\*\*0:32:32.4 Vael:\*\* I think in this argument, maybe we have something like human-level intelligence in our AI and we\'re just trying to get it to do any goal at all, and we\'re like, \"Hello, I would like you to do a goal, which is solve cancer.\" But we are unable to properly specify what we want and all the side effects and all the things.

\*\*0:32:47.0 Interviewee:\*\* Yeah, yeah, yeah. Yeah. No, that\'s exactly where I\'m thinking in. So imagine that you are studying a degenerative disease and you figure out, or you ask your AI, \"Can you tell me why ALS happens in focal cells\" So your AI gives you an answer, \"Well, it\'s because of some weird transcription that happens on these genes.\" So if you have a small molecule that addresses that, then you\'re going to have normal-acting genes and normal phenotype in those cells. Okay, so byproduct could be, after 10 years, some of what was happening in those genes is natural for aging, and so you introduce some cancer, because what you did to fix neurodegenerative disease is so unnatural that it has that kind of horrible side effect. Now, is that a problem? (Vael: \"Yeah, I think\--\") Yes. Yeah, but\--

\*\*0:33:52.7 Vael:\*\* I think in this particular scenario, maybe the AI would know that it was a problem. That like 10 years out, you would not be unhappy. I mean, you would be unhappy.

\*\*0:34:03.6 Interviewee:\*\* You would be unhappy. Yeah, so okay, even if it does know it. Is that\... So yes, that\'s an issue, but it\'s also significantly better than where we are today. And let\'s say the expected lifespan for ALS is five years, and now you\'re saying 10 years, but you\'re going to have this horrible cancer. That\'s a bad trade-off, bad decision to force somebody to make, but is undoubtedly progress. So I think black boxes.. and that would not be a black box, I believe. Black box would be like when\... you don\'t know, you find out about this 10-year cancer 10 years after it passes clinical trials, you know? So this would be like, almost a very good version to AI that can tell you like, \"I found a solution, but given the current technology, this is the best we can do.\" So I would think that is a fantastic future that we\...

\*\*0:35:00.8 Vael:\*\* I think that\'s right.

\*\*0:35:02.0 Interviewee:\*\* The black box version, I think is also a fantastic future because it would still represent meaningful progress that would hopefully continue to be built on by advances in biomedical technologies and AI. So for this specific domain, there\'s obvious issues with black box. Like, if you\'re going to use AI to make any decisions about people, whether they can pass a bar for admissions, this or that or the other thing, there\'s going to be problems there. But for medicine at least, I think black boxes should be embraced.

\*\*0:35:45.2 Vael:\*\* Yeah, I think the scenario you outlined is actually what\-- I would say that that system is doing exactly what the designers intended them to. Like the designers wanted a solution, and even if we currently don\'t have best solution, it gave it the best we had. I\'m like, \"That seems successful.\" I think an unsuccessful version would be something like, \"Alright, I want you to solve this disease.\" But you forget to tell it that it shouldn\'t cause any other side effects that it certainly knows about and just doesn\'t tell you, and then it makes a whole bunch of things worse. And I think that would be more of an example of an unsuccessful system doing something that didn\'t optimize exactly what they\'re intended.

\*\*0:36:19.0 Interviewee:\*\* Yeah, yeah, so that\'s just a failure. Okay, so here. Let\'s say it only does that on a certain type of people, racial profiles. Okay, so then the upshot of automating AI is that, sorry, automating science, is that it should make it a lot cheaper to test a far wider range of people. So just like, now the current state of the art for image\-- object classification is to pre-train on this 300 million image data set, this JFT-300 or whatever. Likewise, you could develop one-billion-person data sets of IPS cells, which are like stem cells. Cool science fiction there that had never happened, but then you would have this huge sample where you would avoid these kinds of horrible side effects.

\*\*0:37:28.7 Vael:\*\* Cool. I still think\-- yeah. I\'ll just\-- (Interviewee: \"No, tell me. Tell me, you still think what?\") Ah, no, no, I, well. I\'m trying to think of how I want to order this, and we\'re running out of time and I do actually need to go afterwards. (Interviewee: \"Oh, okay.\") So I think I want to get to my next question, which does relate to how I think this previous\... where I\'m like, \"Hmm\" about this previous question. So assume we have this powerful AI CEO system, and it is capable of multi-step planning and is modeling other people modeling it, so it has like a model of itself in the world, which I think is probably pretty important for anything to be deployed as a CEO AI, it needs to have that capability. And it\'s planning for\... And its goal, people have told it, \"Alright, I want you to optimize profit with a bunch of these constraints,\" and it\'s planning for the future. And built into this AI is the idea that we want\... That it needs to have human approval for making decisions, because that seems like a basic safety feature. And the humans are like, \"Alright, I want a one-page memo telling us about this potential decision.\" And so the AI is thinking about this memo and it is planning, and it\'s like, \"Hmm, I noticed that if I include some information about what\'s going to happen, then the humans will shut me down and that will make me less likely to succeed in the goal that\'s been programmed into me, and that they nominally want. So why don\'t I just lie a little bit, or not even a lie, just omit some information on this one-page memo, such that I\'m more likely to succeed in the plan.\" And so this is not like a story about an AI being programmed with something like self-preservation, but is a story about an agent trying to optimize any goal and then\...

\*\*0:38:55.2 Interviewee:\*\* Like shortcuts, yeah.

\*\*0:38:56.5 Vael:\*\* And then having an instrumental incentive to preserve itself. And this is sort of an example that I think is paired with the previous question I was asking, where you\'ve built an AI, but you haven\'t fully aligned it with human values, so it isn\'t doing exactly what the humans want, it\'s doing what the humans told it to do. And so, what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous.\"

\*\*0:39:38.5 Interviewee:\*\* Okay, so if we are going to be using supervision to train these systems, what you\'re describing is a shortcut. So, highly intelligent systems exploiting a shortcut like you\'re talking about, which would be like, \"Let\'s achieve a high score by doing the wrong thing\...\"

\*\*0:40:00.4 Vael:\*\* Yeah, yep.

\*\*0:40:00.5 Interviewee:\*\* Can be corrected by having the right data set. Not saying that it would be this simple, input-output, learn the mapping between the two, but we\'ll just imagine that it is. If you have any kind of low-level bias in your data set, or high level bias, if this is the case, neural networks will learn to find that bias. So it\'s a matter of having a broad enough data set where that just basically, statistically, what you\'re talking about would not exist.

\*\*0:40:35.4 Vael:\*\* Interesting. I feel like\--

\*\*0:40:37.3 Interviewee:\*\* I don\'t think that would be a problem if you can design the system appropriately.

\*\*0:40:41.2 Vael:\*\* Yes, I totally agree this would not be a problem if you can design the system appropriately.

\*\*0:40:44.9 Interviewee:\*\* The data set appropriately.

\*\*0:40:46.6 Vael:\*\* Ah, the data set appropriately. Interestingly, on one of the examples that I know of where we have, like you set the AI, it\'s like a reinforcement learning agent, and you\'re like, \"Reinforcement learning agent, optimize the number of points.\" And then instead of trying to win the race, which is the thing you actually wanna do, it just like finds this weird local maximum.

\*\*0:41:02.8 Interviewee:\*\* Yeah, so for RL, that\'s like the ultimate example of shortcuts. Because there, what you have are extremely\... So, for video games, if you\'re talking about a race, you have extremely limited sensor data, sorry, low dimensional sensor data. And you have few ways of controlling, manipulating that sensor data. So, let\'s say there\'s two local minima that can be found via optimization, and one is, \"go for red, because red wins,\" and the other is, \"Drive your car to avoid these obstacles and pass the finish line,\" which just happens to be red. Of course, the shortest solution will be the one chosen by the model. So I agree, shortcuts are ubiquitous, and of course, they will become more and more advanced as we move into these other domains. But I think what you\'re describing is just a problem with shortcuts. And so it becomes a question of, can you induce enough biases in your model architecture to ignore those shortcuts? Can you design data sets such that those shortcuts wash out via noise? Or can you have human intervention in the training loop that says, \"Keep working, keep working. I don\'t accept this answer?\"

\*\*0:42:30.3 Vael:\*\* Great, yeah. Cool. Have you heard of AI alignment?

\*\*0:42:37.1 Interviewee:\*\* Yeah, yeah, I have. Well, I\'ve seen OpenAI talking about that a ton these days.

\*\*0:42:42.3 Vael:\*\* Great. Can you explain the term for me?

\*\*0:42:46.5 Interviewee:\*\* So this is like, you have some belief about how a system should work, and the model is going to do what it does, and so alignment means you\'re going to bring the system into alignment with your belief about how it should work. It\'s essentially what you\'re talking about with the shortcut problem. So, that\'s it, yeah.

\*\*0:43:09.4 Vael:\*\* Great, thanks. Yeah, I think I mostly think the problem of aligning systems with humans is going to be\... One of my central thing is that I think it will be more difficult than it seems like you think, which you were like, \"Oh well, we can do it with having better data sets or we can do it with denoising things,\" and I\'m like, I don\'t actually know if\...

\*\*0:43:31.1 Interviewee:\*\* I don\'t know about denoising\-- I don\'t mean to make this sound trivial, because it certainly is not. But to give you an example, MNIST is so confounded with low-level cues. You don\'t need to know \"six\" to recognize six, you just need to know this contour, right? Is that\... only in six do you have the top of the six and then a continuous contour. So, that\'s what I mean. So if you can design a data set where those don\'t exist, then you\'re golden. But usually\-- In the real world, that doesn\'t happen that much because we have these long-tailed distributions of stuff. So then you have to induce biases in your architecture, and this comes back to human vision, sorry, human intelligence. It\'s alignment with human decision-making, because we have all these biases through development through our natural\-- our genomes, through neural development that make us able to interact with the world in such a way where we don\'t just go for the red thing, where we\'re not vulnerable to adversarial examples. So I don\'t mean to trivialize it, I only mean to say computationally, the problem is biases within the data set, and when you use gradient descent, if that is the learning algorithm that you used to train the model that you\'re talking about, then that\'s what you\'re going to be fighting against.

\*\*0:45:06.0 Vael:\*\* Cool. Yep, makes sense.

\*\*0:45:07.6 Interviewee:\*\* Yeah.

\*\*0:45:08.3 Vael:\*\* Alright. I think my last question is just, how has this interview been for you and have you changed your mind on anything during it?

\*\*0:45:19.1 Interviewee:\*\* Yeah, well, it\'s great. I never thought of an AI CEO. I think that\'s super fascinating. Yeah, it was super fun. Yeah, it was great.

\*\*0:45:33.2 Vael:\*\* Great. Wonderful. I think I\'m going to call it now, just because we both have places to be. And I will send along the money, and then I\'ll also send along some resources since you said you were curious about my opinions on timelines.

\*\*0:45:47.6 Interviewee:\*\* I definitely am. Thanks so much for your time, I really appreciate it.

\*\*0:45:51.1 Vael:\*\* Yeah, thanks so much for yours.

\*\*0:45:53.9 Interviewee:\*\* Okay. All right, take care. Have a good night.

\*\*0:45:55.7 Vael:\*\* Bye.


Interview with 92iem

\*\*Interview with 92iem, on 3/21/22\*\*

\*\*0:00:05.2 Vael:\*\* Alright, here we are. So my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:15.2 Interviewee:\*\* Currently, I work a lot with language models, but that wasn\'t always the case.

\[\...\]

\*\*0:00:57.4 Vael:\*\* Great, thanks. And then what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:01:08.5 Interviewee:\*\* So obviously, I think the progress in language models in the last couple of years has been pretty astounding. And the fact that we can interact with these models in more or less in the natural way that we would like to interact with it just has opened up so much in terms of getting feedback from humans and stuff like that. So I think just the progress in language models, and then coupled with that, the more recent progress in using essentially some of the same techniques to do image modeling, so that you have the possibility to do just seamless multi-modal models. I think that\'s quite exciting. Some people think that\... You know, it\'s not like most of us can just paint a photographic scene and show it to other people. So it\'s not like\-- the photographic aspects of generative image models is not what excites me, it\'s the fact that humans manage to communicate quite a bit with diagrams and stuff like that. When we\'re doing science, you can draw little stick figures and pretty much convey what you need to convey, and that coupled with natural language should give us the ability to start thinking about getting AI to do math and science for us, and I think that\'s the thing that is most exciting to me.

So I know that a lot of people are excited by the idea that you can essentially have a Google that\'s a bit\... It\'s smarter, right? You can just talk with it and say, Hey, tell me a bit about this tree, and AI says something and you say, Oh, but what about that tree? That\'s fun, but I really feel like humans are not bottlenecked by the inability to ask about trees and buildings and trivia, essentially. I think where we\'re bottlenecked is like progress in science. I think, for example, so it\'s pretty clear that the political solution to climate change\-- the time for that has kind of come and gone. I mean, we can slow it down. If we, like the whole world, suddenly decided to say we\'re going to do something about this, maybe you slow it down, but I think just the timing is a little bit off. So a lot of that\'s going to be have to be a technological solution. And as amazing as technological progress has been, I think we\'re not fast enough when it comes to developing solutions to a lot of our problems. And I do think in 10, 20 years, AI is going to play a big role, both in the specialized domain in the sense of AlphaFold, where you really just come up with a system that does the thing you want it to do, but more impactfully, perhaps, by having an army of grad student-equivalent language models that can help you answer questions that you need answered. So that\'s very exciting, right.

\*\*0:04:24.1 Vael:\*\* Yeah. It\'s a cool vision.

\*\*0:04:26.5 Interviewee:\*\* I think the risks are\... It\'s almost banal, right? Like with most technologies bad actors can make arbitrarily bad use of these things. So yeah, when they start weaponizing these things\... I\'m a little bit less concerned than some people are about like, Oh, but what if we have AIs that write fake news. Like all of that is to some extent present now, and I guess it\'s just a question of degree, to some extent. Okay, people argue that that difference in degree matters, and they\'re not necessarily wrong. I just, the thing that bothers me more definitely is very specific, malicious uses of AI. So there was a recent paper, this is so obvious that it\'s almost dumb, but someone said, Oh, yeah, we put an AI to trying to develop a drug that, let\'s say, reduces the amount of poison, and all you have to do is change the objective function, flip the sign and suddenly it just optimizes for the most poisonous thing you can possibly find. That coupled with technologies like CRISPR and stuff like that just creates a pretty dangerous\... puts very dangerous tools at people\'s disposal. So I would say that\'s the thing that I would worry about.

\*\*0:05:54.9 Vael:\*\* I have been impressed by how everyone I\'ve talked to in the past week has mentioned that paper, and I\'m like, good, things get around.

\*\*0:06:01.8 Interviewee:\*\* Well, Twitter. Thanks to Twitter.

\*\*0:06:04.7 Vael:\*\* Nice. Alright, so focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50-plus years into the future. So at least 50 years in the future, what does that future look like?

0:06:17:9 Interviewee: For AI?

0:06:20:5 Vael: In general, where if AI\'s important, then mention that.

\*\*0:06:27.4 Interviewee:\*\* I see. So 50 years, oh my God. Fifty years is long time away. Assuming that we\'ve managed not to have nuclear conflicts between now and then, which is just one of those things that now you have to put at least a one digit probability on these days. But, yeah, I think that we will end up having\... Well. The optimistic scenario is that we ended up solving a few key problems. One is transitioning mostly out of fossil fuel, so a combination of solar and fusion power. I think that\'s going to be huge, and I think that AI will have played a role in some of that development. And I think 50 years from now, I think unless we are monumentally blocked in the next couple of years, AI will be pretty omnipresent in our lives, and certainly in the scientific sectors. So one thing that I\'m a little bit, just something that comes to mind, is that a lot of people are into this idea of these sort of augmented\... I don\'t know if people are literally willing to wear glasses, but certainly you could imagine having little ear buds that are fairly unobtrusive that go around your ears or something, and they do have a camera, so you can just ask it, whatever you need, you can ask it questions.

In 50 years, I think at that point, maybe some people will have worked out direct neural interfaces with stuff, and so maybe the more adventurous people will have a bit of augmented memory or at least the ability to sort of silently query their little augmented system. I think that might be a thing. Not everyone will have adopted it, I think it\'ll be a weird world. I personally\-- I\'ve never been a huge, like the fastest adopter of technology, but that sort of stuff is next level, and I don\'t know what that\'s going to look like.

I also\... well, two things, I guess they\'re kind of linked. I think that people will live substantially longer. I think, unless something miraculous happens, I don\'t think they\'ll be living like 200, 300 years, but I certainly think it\'s possible people will be living to 150 or something like that. Not people born now; I\'m not going to live to 150. Someone was telling me that people born, these days, they\'re going to live to see the year 2100, right. That\'s not quite in the 50-year time frame, but yeah, I certainly think people born today are going to be living, like their average lifespan in industrialized countries, assuming a certain level of privilege, they\'re going to be able to live quite a bit longer. That coupled with AI possibly automating quite a few jobs is going to change the social landscape a bit.

One thing that occurred to me recently\... so people used to say that\-- well, people say many things\-- one is that, this, unlike industrialization\... Some people always say technological progress destroy some jobs but creates more jobs on the other side. And then some say, Okay, but this one is different because you\'re automating intelligence and that really does put humans out of their main forte. So one of the things that people worry most about, in addition to the universal-based income stuff, is just the loss of dignity, that people always assume that even people who don\'t have what you would call glamorous jobs value the fact that they work and get paid for that work. But I think some of the stuff that happened during Covid makes me doubt that a little bit, in the sense that people did quit jobs that ostensibly looked good. Even in the tech sector where people, I felt like generally, they\'re not the worst jobs by any stretch, and they were like, No, this is meaningless, I want to go do something meaningful with my life. So I think the recent, the past couple of years have made me question the idea that it would be that big of a psychological blow for people to not work for money. That if you did establish a universal basic income, plus you\'d have to solve some other, many complicated issues, but I don\'t think people will be that unhappy to be not having to work menial jobs. I\'m not saying there\'s not going to be upheaval, but I think it\'s going to be like a combination of living longer, and not possibly having to do jobs if you don\'t want to do them. I think that\'s just going to be, I don\'t know. It might be a nice change. In the optimistic scenario, I guess.

\*\*0:11:35.9 Vael:\*\* Got it. Yeah. Well, my next question is related to that. So people talk about the promise of AI, by which they mean many things, but one of them is maybe having a very general capable system such that it will have cognitive capacities to replace all current day human jobs. So you might have a CEO AI or a scientist AI. Whether or not they choose to replace human jobs is different, but have the ability to do so. And I usually think about that and the fact that 2012 we have AlexNet, deep learning revolution, here we are, 10 years later, we\'ve got things like GPT-3, which can do some language translation and some text generation, coding, math, etcetera, a lot of weirdly general capabilities. And then we now have nations competing and people competing and young people going into this thing, and lots of algorithmic improvements and hardware improvement, maybe we get optical, maybe we get quantum, lots of things happening. And so we might actually just end up being able to scale to very general systems or we might hit some sort of ceiling and need to do a paradigm shift. But regardless of how we do that, do you think we\'ll ever get very general AI systems, like a CEO or a scientist AI, and if so, when?

\*\*0:12:39.1 Interviewee:\*\* I don\'t know about CEO AIs. The scientist AIs, yes. Yeah, and that\'s going to come in stages. So obviously the current generation of AIs, we don\'t put them in human bodies and let them do experiments and stuff like that, right. It\'s going to be a while before we start letting them operate like particle accelerators. Fifty years\... Maybe in 50 years. My original background is \[non-AI field\], and I really could have just done my entire PhD from a desk, and that sort of work, certainly, AI can replace, I think, to a huge degree, from idea generation to solving the answer and writing a paper, yeah, that just feels so doable. Again, unless we hit a giant wall and find that our current transformers simply cannot reason, but I think that looks unlikely. I don\'t rule it out, but that looks unlikely to me.

\*\*0:13:46.8 Vael:\*\* Yeah. Okay, what about this CEO AI, like with multi-step planning, can do social inference, is modeling other people modeling it, like crazy, crazy amount of generality. When do you think we\'ll get that, if we will?

\*\*0:14:00.7 Interviewee:\*\* Yeah, that\'s not the part that I\'m worried about. AI can certainly model human intent, but\... I guess it depends on what you want from your CEO AI. And this I think gets at a little bit one of my dissatisfactions with discussions about human\-- like, AI alignment. It\'s not that people don\'t talk about it, but it\'s rarely talked about. I don\'t know, on Twitter certainly. A lot of AI alignment stuff talks about\-- they \\*don\'t\\* talk about the fact that humans disagree wildly on what humans should do. So I\'m thinking about this in connection with the CEO, because I think in the limit, AI will be able to do anything, any specific thing you ask the AI to do, it can do, but the question of whether you would want the AI to be CEO, I think that\'s mostly a human question. So that\'s why I said\-- I think that\'s a policy decision, not a AI capability question.

\*\*0:15:25.3 Vael:\*\* Got it, yeah. Do you think that people will end up wanting\... that there will be economic incentives such that we\'ll eventually have things like CEO AIs?

\*\*0:15:36.0 Interviewee:\*\* I guess in some sense, no, because I think a human would still be the CEO and then you would have your AI consultant, essentially, that you would ask all the things. You would delegate almost everything, but I think that people would still want to be at their very apex of a corporate hierarchy. It seems weird to put a robot in charge of that, just like\... why. It\'s a title thing, almost, like, why would you make the robot the CEO?

\*\*0:16:03.2 Vael:\*\* Yeah, yeah. In some vision of the future I have, I have the vision of a CEO AI\... we have a CEO AI and then we have shareholders, which are the humans, and we\'re like, \"Alright, AI, I want you to make a company for me and earn a lot of money and try not to harm people and try not to exploit people and try to avoid side effects, and then pass all your decisions, your major decisions through us\" and then we\'ll just sit here and you will make money. And I can imagine that might end up happening, something like that, especially if everyone else has AI is doing this or AIs are way more intelligent and can think faster and do things much faster than a human can. I don\'t know, this is like a different future kind of idea, but.

\*\*0:16:46.8 Interviewee:\*\* But that seems so weird. Because, then\-- so, assuming\... I don\'t know if everybody has access to the same AI in that scenario, but like it can\'t be the case that 100 people all say to their own individual AI, \" Form a company and turn it into a \$100 billion company or a \$1 trillion company\", and they all go out at optimizing. I think at that point, in that kind of world, I think there would have to be a bit more coordination in terms of what goes on, because that just creates some nasty possibilities in terms of bringing the economy down. So I don\'t know that that\'s how things would just happen. It cannot be the case that we would just say, \"Robot, figure out how to make a trillion dollar company. I\'ll give you this one idea and just run with it,\" and then just like we are hands-off. That seems extremely unlikely, somehow.

\*\*0:17:38.9 Vael:\*\* Yeah, I\'m interested in how that seems very unlikely. It seems like to me\... Well, we were talking about scientist AI, and I imagine we can eventually tell a science AI to like solve cancer, and maybe it will actually succeed at that or something. And it seems like it\'s different, to be like, Hey, CEO, make a ton of money for me. Is that getting at any of the underlying thing or not?

\*\*0:18:08.4 Interviewee:\*\* Uh, hm. Yeah, so I think even there, I think you would never tell an AI to \"solve cancer\". Well, yeah\... You would want to give it more specific goals, and I think\... In any scenario where we have full control over our AIs, we wouldn\'t want such vague instructions to be turned into plans. That\'s a scary world, where you can just say solve cancer and the robot runs with it. I think for the same reason, I don\'t think you would want a world where someone can say, \"AI, make a lot of money for me,\" and that\'s the only instruction the AI has, and it\'s allowed to intervene in the world with those instructions. So yeah, that\'s why I don\'t see, just like from a sanity perspective, you would\-- you never want to unleash AI in that manner, in such a vague and uncontrolled manner. Does that make sense?

\*\*0:19:03.6 Vael:\*\* Yeah, that makes sense that you wouldn\'t want to be\... because it\'s very unsafe, it sounds like, or it could be\--

\*\*0:19:09.7 Interviewee:\*\* Yeah, kind of insanely unsafe, but\...

\*\*0:19:14.8 Vael:\*\* Nice. Yeah, do you think people might end up doing it anyway? Sometimes I feel like people do unwise things in the pursuit of, especially unilateral actors, in the pursuit of earning money, for example. Like, Oh, I\'ve got the first scientist AI, I\'m going to use it to solve the thing.

\*\*0:19:35.3 Interviewee:\*\* That\'s a good question. I think, I really do think you would want\... Yeah, I wonder about how you would actually enforce any kind of laws on AI technology. It\'s the most complicated thing to enforce, because nuclear weapons\-- One of the nice things about nuclear weapons is it\'s actually pretty hard to develop nuclear weapons in secrecy without releasing any radiation, that\'s one of its few good points. I think AI, it\'s true that you could just develop and run it. But I think at the point where any AI has to interface with the real world, whether it\'s in the stock market or something like that, I do think that people will start seeing the need for finding ways to regulate the speed. Even high frequency trading is starting to be, like you can\'t interact with it, any kind of stock market in less than one nanosecond or something like that. I think similarly, there\'s just going to be some guardrails put in place. If there\'s any kind of sanity in terms of policymaking at that time, you would want guardrails in place where you could not unleash AI with such large powers to affect a large part of the world with minimal intervention powers. Yeah. This is all assuming there\'s a sane policymaking environment here, but\...

\*\*0:21:04.8 Vael:\*\* Yeah. Do you think there will be?

\*\*0:21:09.3 Interviewee:\*\* I think so. I think so, I\'m hopeful in that regard. I\'m not saying that Congress is ever going to really understand the nuances of how AI works, anything like that, I just think there would be too many\... Even in a world where only OpenAI and DeepMind have full AGI, I don\'t think they\'d want to create a world where one of them can unleash something at the level that you described. And I also think that when those two companies get close, they\'re going to wonder if other states, say, Russia or China, are going to be close, and they\'re going to start wanting to really hammer down, hammer out\... like there will be a sense of urgency, and hopefully they have enough influence to influence policymakers to say, \"You need to take this seriously.\" And this is where I think almost the fact that it takes\... Okay, I said earlier that, you know what, the nice thing about nuclear weapons is that you could detect it, but I think one of the nice things about the fact that right now, it looks like you\'re going to require enormous compute to get anything that is remotely AGI. That\'s the thing that allows maybe\... That means the only huge corporations or states will be able to do it for at least some period of time, and hopefully those are the same actors that can somehow influence policymaking. If there were just one person, if they just had the ability to do that, it would be a little bit problematic, actually. So in some sense, because these institutions are big, I think they\'re going to be both constrained a bit more in terms of what they can do, and also they\'re going to be able to, if they are well-intentioned, to influence policymaking in a good direction.

\*\*0:23:06.2 Vael:\*\* Do you think they\'ll be able to do international cooperation? Because I imagine China will also have some AI companies that are also kind of close, I don\'t know how close they will be, but\...

\*\*0:23:17.7 Interviewee:\*\* They\'ll try. I don\'t know that China will listen to the US or Europe. I agree that\'s not going to be easy, yeah. Who knows what they\'re up to exactly, there.

\*\*0:23:31.0 Vael:\*\* Yeah, it seems like they\'re certainly trying, so\... Yeah, another one of my questions is like, have you thought much about policy or what kind of policies you want to have in place if we are getting ever closer to AGI?

\*\*0:23:48.8 Interviewee:\*\* Actually, I haven\'t given it that much thought, what the laws would specifically look like. What I don\'t think is really possible is something like the government says, You now need to hand over control over this to us. I don\'t think that\'s super feasible. Yeah, I can\'t say I have a good idea for what the laws would specifically look like. I think as a starting point, they\'ll certainly create some kind of agency to specifically monitor\... Actually, right now, there\'s no agency like the SEC or something like that that monitors what exactly goes on in AI. I mean, there\'s some scattering of regulations probably somewhere, some vague export controls and stuff like that. But yeah, they\'d certainly start creating an agency for it, and their mandate would start to grow. I think it might, again, have to be something like what we do with nuclear reactors, where you have an agency that has experts inside of it, and that they are allowed to go into companies and kind of investigate what\'s going on inside, just as, if Iran is developing nuclear weapons and they agree to let inspectors in. I think it\'s going to be up to something like that. And then, yeah, similar to these nuclear treaties, perhaps there would have to be something along the lines of like\... there are certain lines you cannot cross with AI, and if someone does cross it, that institution or the country as a whole gets sanctioned. It\'s going to have to be at that level. Certainly, given the power of the putative AI that we\'re thinking about. I think the regulations are going to have to be quite dramatic if it\'s going to have any kind of effect.

\*\*0:25:45.1 Vael:\*\* Yeah. One thing I think that is a difference between the nuclear situation and the AI situation is that nuclear stuff, seems not very dual use. Well, nuclear weapons, at least, not very dual use. Versus like AI has a lot of possible public benefit and a lot of economic incentives, versus like you don\'t get, I don\'t know, you don\'t benefit the public by deploying nuclear weapons.

\*\*0:26:05.7 Interviewee:\*\* But nuclear reactors, but that\'s the whole\--

\*\*0:26:07.9 Vael:\*\* Nuclear\-- Yes, you could\--

\*\*0:26:09.0 Interviewee:\*\* That\'s the whole\... So Iran would always pretend that, Hey, we\'re just developing nuclear reactors for power. Just the problem is that was always very easily converted to nuclear weapons. I think that could be a similar---

\*\*0:26:24.9 Vael:\*\* Yeah, yeah, it is similar in that way. Somehow it still feels to me that these situations are not quite analogous, in that the regulations are going to be pretty different when you\'re like, \"I am going to make sure that you\'re not doing anything bad in this area,\" and people are like, \"ah, yes, but we need to get the new smartphone, scientist AI, etcetera.\" But yeah, I take your point. Another thing that I think is interesting is that current day systems are really pretty uninterpretable, so you\'re like, \"Alright, well, we have to draw some lines, where are we going to draw the line?\" What is an example of what a line could be, because if there\'s government inspectors coming in to DeepMind and you\'re like, \"Alright, now inspect,\" I\'m like, what are the inspecting?

\*\*0:27:11.2 Interviewee:\*\* Yeah, so when you say interpr\-- so that\'s another thing about\... one of my pet peeves about interpretability. People are not that interpretable. People hardly, rarely know what\'s going on in other people\'s heads, and they can tell you something which may or may not be true, sometimes they\'re lying to you, and sometimes they might be lying to themselves. When a doctor tells you, \"This is what we\'re doing,\" unless you\'re another doctor, you rarely understand what they\'re saying. And so, yeah, this is a total tangent on like my\... the thing around, the discussion around interpretability is always such a mess. But what are they inspecting? If we\'re imagining inspectors, they could certainly go in and say, like, if it\'s a language model, you can certainly allow them to query the language model and see what kind of answers, what kind of capabilities these language models have. You could say, if it\'s a language model, just totally hypothetically, you could say, \"Alright, develop me, write me a formula for a bioweapon,\" and if the language model just gives that to you, then possibly you have a problem. Stuff like that. So if a company that has that capability hasn\'t put in the required fail-safes like that, then they can be held liable for X amount of problem, the trouble, right.

\*\*0:28:58.3 Vael:\*\* Interesting. Cool, so that\'s cool. You\'ve got like a model of what sort of rules should be in place, and it sounds like there should be rules in place where you can\'t develop bioweapons or you can\'t feed humans bioweapons when they ask for them.

\*\*0:29:12.0 Interviewee:\*\* Yeah, stuff like that. In this inspector model, I think that\'s what would kind of have to happen. But yeah, it\'s not like I\'m an expert in this, but that\'s what I would think.

\*\*0:29:24.4 Vael:\*\* Yeah, something I\'m worried about is that no one is an expert in this. Like policymakers\-- when I talk to the AI researchers, they\'re like, Oh, yes, the policymakers will take care of it, and I\'m like, the policymakers are busy, and they\'re doing many different things, there\'s not many people who are focused singularly on AI. Also, they\'re mostly focusing on current day systems at the moment, so like surveillance and bias and transparency, and like a bunch of different things, so they\'re not really thinking very future at the moment. And they don\'t know what to do because they don\'t understand the technology, because the technology moves extremely fast, right, and so like AI researchers are the ones who know it. And I\'m like, Alright, AI researchers, what should we tell them to do. You\'re like, Well, we should make a list of things that the AI shouldn\'t do, like basic fail-safes. And I\'m like, Great, it would be super cool if that was written out somewhere and then we can start advocating for it or something, because I\'m worried that the policy will just continue to lag really far behind the actual tech levels, except where like\... Policy is already several years behind, maybe like 10 years or something, and will continue to be that far behind even as we\'re approaching the very powerful AI.

\*\*0:30:31.9 Interviewee:\*\* Yeah, so a couple of things there. One is that that\'s why you need more of an agency model rather than laws, because creating laws is very, very slow, whereas an agency can drop some rules and maybe they start enforcing them. And so you do need a sensible agency that doesn\'t create bad rules, but the ability to be flexible. That said, I think\... The biggest problem with policymaking right now is that the policymakers don\'t understand AI at all, right. And you sort of hinted at that. And I think\... If I\'d asked myself, at this moment in time, is there anything that, any rule that we need at this moment in time, I\'m not sure there is. AIs are not there yet.

\*\*0:31:25.2 Interviewee:\*\* So at this moment in time, I think if you ask most researchers, \"Hey, do we need to create specific laws to prevent X, Y, Z,\" I\'m not sure many people would tell you, you need that. And so these laws, I think, are going to have to come in at very sensible points, and it\'s not clear to me that the policymakers are going to know when that time point is. I would say even in the AI field, very few people know when that\'s going to be. There\'s a lot of stuff coming out of especially big labs where the world doesn\'t know. There\'s like 100 people that know what\'s coming in the next year. I don\'t know what a good solution to that is.

\*\*0:32:17.5 Vael:\*\* Especially if we can get AIs that can generate lots of deadly poisons already. Yeah, I think it\'ll maybe be hard to tell, and then also one needs to develop a list, if there\'s going to be in list form or\...

\*\*0:32:32.2 Interviewee:\*\* The problem is, I think it\'s easier to regulate general AI just because it\'s going to require so much compute. But I think more specific AI that anyone can run on a GPU, like on a laptop, is more or less impossible to regulate. So it\'s not clear to me what the law would be, except if you use a bioweapon, you\'re in trouble. That law already exists, right.

\*\*0:33:00.5 Vael:\*\* Yeah, I think that one already exists, so\...

\*\*0:33:05.4 Interviewee:\*\* So I think in some sense, like kind of in the trade-off of what can the technology do right now, and who might try to deploy that, our laws sort of cover the problem cases at the moment. I think where I get a little bit stuck is if you try to say, \"Alright, in five years, should we have laws banning certain uses of a very, very capable general model?\" I do think at that point, Congress should seriously consider creating a regulatory agency. And I think AI researchers will only support this if there\'s some semblance of like, kind of like NASA, where there\'s some faith that engineers are in charge of this thing, that kind of know how these systems work, that they can think rationally about both the technological side and the policy side of things. And so that\'s going to take some work on the side of whatever administration is in power at that time. But yeah, it\'s not going to be easy. I think it\'s going to take a very capable administration to handle that transition gracefully.

\*\*0:34:19.5 Vael:\*\* Yeah, that makes sense. Yeah, I\'m worried about a few different things in this future scenario. I\'m like, Okay, I don\'t know if the agency will be developed while\-- in a sort of future thinking sort of way, I don\'t know that it will implement the right type of policies, I don\'t know that it will have the power to really enforce those policies, I don\'t know if it will have the power to enforce internationally. But I do like the idea that\-- but obviously one should still try, and it seems like there should probably be a lot of effort going into this, as you said, something like on a five-year scale.

\*\*0:34:49.2 Interviewee:\*\* Yeah, it\'s just that knowing AI researchers, there\'s just going to be such extreme pushback. If there\'s any sense that there\'s been a bureaucracy created whose job is nothing more than to just slow things down for no good reason. That\'s almost a default kind of way in which such an agency would get created, and so, yeah, it\'s just one of the situations where you have to hope that the future leaders of America are smart.

\*\*0:35:24.7 Vael:\*\* Yeah. Yep. A thing to bank on. Cool. So I\'m concerned about long-term risks of AI. That\'s one of the ways in which I\'m concerned, is that we won\'t get the policy right, especially as we\'re doing international competition, that there may be race dynamics, as we\'re not able to have really strong international governance. And I don\'t know if this will go well, and I\'m like, I think people should work on this.

But another way I think that things might not work: So we talked a little bit about the alignment problem. And another interesting thing about the alignment problem is\... or in my mind, so we\'ve got maybe a CEO AI, or whatever kind of AI, but this is the example I\'ve been working with, and it\'s making plans and it has to report any decisions it makes to its shareholders, who are humans, and the humans are like, \"I want a one-page memo.\" And the AI is like, \"Okay, cool, one-page memo. I have a lot of information in my brain, in my neural network, while I\'m trying to maximize profits with some other goals\-- with some other constraints.\" And it\'s noticing that if it gives certain information to humans, then the humans are more likely to shut it down, which means that it\'s less likely to succeed in its goal. And so it may write this memo and leave out some information so that it decreases likelihood of being shut down, increases the likelihood of achieving its goal. So this is not a story where we\'re building in self-preservation into the AI, but a story in which\-- why instrumental incentives of an agent achieving, trying to go for anything that is not perfectly aligned with human values, just like what humans tell it to do instead of what humans intended it to, then you might get an AI that is now optimizing against humans in some degree, trying to lie to them or deceive them in order to achieve whatever it has been programmed to do. Do you think this is a problem?

\*\*0:37:08.2 Interviewee:\*\* The scenario you described was exactly what human CEOs do.

\*\*0:37:12.7 Vael:\*\* Hm, yes. But more powerful systems, I think, with more influence over many things.

\*\*0:37:20.6 Interviewee:\*\* So this is the problem\-- so I think this actually still is a human problem. So if a human being\... like these AIs will be, depending on the mix of reward for not getting shut down and\... at the kind of detailed level these days, we often\... When we do RL with language models, we have two things going on, one is an RL objective, maximize the reward as much as you can, but the other objective is to tie it to the original language model so it doesn\'t diverge too much. In which case, if you are writing a memo, it would try to write a memo in the style that a human would write it, let\'s say. So the information content would be somewhat constrained by what a typical memo written by a human being would look like, and then on top of that, it would try to optimize what it is trying to do, maybe just trying to keep the company alive for as long as it can or something like that. So there is that sort of like, at least the way we do things now, there\'s a little bit of self-regulation built in there. But this is why I think, more fundamentally\... any question where if you just replace the AI with a human and ask the same question: Is this a problem or not a problem? I think that\'s more or less a human problem. And you have to think a bit more carefully about what we would want a human to do in that exact same situation. Do we have an answer for that? And then take into account the fact that the AI is more powerful. You don\'t need a super devious AI for a CEO to start lying to their shareholders a little bit, or misleading their shareholders a little bit, in order to present a more rosy picture of what the company is doing. So do we already have mechanisms that prevent that? I think we do, and that same thing would apply to the AI.

\*\*0:39:22.1 Vael:\*\* Yeah. I think the things that are interesting to me about the AI scenario is that we have the option of\... we are designing the AIs, so we could make them not be this way. And also having an AI that has a lot, lot more power, that is as powerful as a leader of one of the countries, and that has the ability to copy itself and could do self-improvement, so it can be smarter than it started out with. And okay, we\'ve got something\'s possibly smarter than us, which is like the ability to reason and plan well, and has the incentive to acquire resources and influence and has all the human kind of incentives here, and we can\'t\-- and it\'s not as\-- I don\'t know, it\'s maybe not as interpretable as a human, but you can\'t throw it into jail. Like, I don\'t know, there\'s a lot of the mechanisms for control, I think, are maybe not there.

\*\*0:40:12.7 Interviewee:\*\* Yeah, so it\'s in this sort of legal context that I think you would not want the AI to be a CEO or any\... There has to be something\... For something like this, you would want the person\... There should be a person who\'s liable for the decision being made by the AI. You have to do some due diligence to the answers that the AI gives you. There\'s no other way. Yeah.

\*\*0:40:40.2 Vael:\*\* There\'s generally a thing in some of your answers where you\'re like, Well, you know, any reasonable person would do X, and I\'m like, I don\'t know if we\'re in a world where we\'ve got a bunch of just reasonable people putting in appropriate fail-safes which they\'ve spent a long time constructing. And some of these fail-safes, I think, might be very technically difficult. I think the alignment problem might be quite technically difficult, such that researchers who are working on capabilities would get ahead even as the people working on safety mechanisms per se is also growing, but at less speed as all the capabilities researchers are pushing forward. Such that we might have an imbalance in how many people are working on each thing.

\*\*0:41:16.3 Interviewee:\*\* Yeah, so I guess maybe I\'m thinking of two different things. One is just the sheer\-- kind of like the question of just putting rules in. The other question that I often have with these discussions is, Does a sensible answer exist to the question at all? So imagine, okay, so imagine we replace the CEO with\... Imagine we replace Mark Zuckerberg with a very, very smart AI. And this very smart AI is posed with this question of, okay, there is a photo of a naked child, but it\'s in the context of a war, it\'s a war photograph. Should this photo be allowed on Facebook or not? The CEO cannot\... It doesn\'t really matter how smart the AI is, this is just not a question the AI can answer, in the sense that it\'s an indelibly human question. That\'s why\-- I just think there are certain questions where when we posit a incredibly intelligent AI, it\'s got nothing to do with that. It\'s just a question of what a group of people who\... A group of humans who disagree on what the final answer should be. In that scenario, there\'s no right answer for the AI. There\'s nothing the AI can do in that scenario that is the correct answer.

\*\*0:42:46.4 Vael:\*\* Yeah. I think in my vision of what I want for AI in the future, I want AIs that do what humans intend them to do, so I want the alignment problem to be solved in some way, and I want it to all involve a huge amount of human feedback. So for every question that is confusing or the AI doesn\'t know what to do, if it hasn\'t internalized human values, then I want it to ask a bunch of humans, or maybe we have some way to aggregate human opinions or something. And then we have an AI that is reflecting human values and preferences, so if humans are confused about this particular issue, then I don\'t know, maybe the default if humans disagree is not to publish\[?\]. But in general, just having some sort of checking mechanism. The thing that I\'m worried will happen by default is that we\'ll have an AI that is optimizing for something that\'s sort of right, but not quite right, and then it will just kind of now do like whatever things we put into it\-- whatever optimization goals we would put into it will be kind of locked in, and so that we\'ll eventually get an AI that is doing something kind of analogous to the recommender algorithms thing, where recommender algorithms are sort of addictive and they\'re optimizing something\-- clickthrough rate\-- that\'s kind of close to what humans value, but isn\'t quite. And then we might have an AI that is just like, A-ha, I am now incentivized to deceive humans to gain control, to gain influence, to do self-improvement, and we\'ve sort of lost control of it while it\'s doing something that\'s like almost but not quite what we want.

\*\*0:44:05.6 Interviewee:\*\* I think one thing that comes to mind, actually\...so this kinda goes back to the interpretability question, but I think it may be a slightly different angle on it. I think it\'s going to have to be the case where when an AI makes a decision of that sort, it should output almost a disclaimer. So the way credit card companies would write you this long disclaimer. And it would have to tell you for each decision it makes, what the risks are, and then a human has to read that and sign off on it. Now, the question is going to be, the other problem with credit card disclaimers is that they were so long that the average person couldn\'t read it and make sense of what the hell was going on. So the AI would be somewhat required to come up with a comprehensible set of disclaimers that say, Okay, I asked a bunch of people, they said this, but obviously we shouldn\'t always listen to what the majority says. I also consulted some moral ethicist or some ethicists, and I synthesized the combination of the ethicists, previous precedents, and what the general public wants. I recommend that given the combination of these three factors, you should do this. And then a person should sign off on it, and then that person in some sense should be liable to the extent that the AI gave a reasonable summary of the decision factors. So something along those lines.

\*\*0:45:32.8 Vael:\*\* Yeah, that sounds brilliant. I would be so excited if AI in the future had that. I\'m like, Wow, we have an AI that is incentivized to instead make things as maximally clear and comprehensible and taking into account what the human wants and listing out of things, I\'m like, If we solve that, if we have the technical problem to solve that, I\'m lnnnike, wow, amazing.

\*\*0:45:52.6 Interviewee:\*\* I think the key point here is at some point, the human has to be held liable for it, so that they have an incentive to only use AIs that satisfy this condition. Otherwise there\'s no reason for the.. because, like you say, you can\'t put the AI in jail, so. At some point you have to put the onus on humans. I think this is something that like even Tesla\'s going to have to think about. At some point, I mean\... I fully believe statistically, they\'ll reduce the number of accidents, but accidents will happen, sometimes the car will be the responsible party. At that point, you can\'t just throw up your hands and say no one was at fault, right? So if Tesla is willing to deploy their cars for self-driving, they are going to have to start taking liability, and that\'s going to force them to confront some of these same issues and say, Did the AI give a reasonable estimation of if we take this road\...? It has to be able to say, or like a surgical robot, it has to be able to say the same thing that doctors do, \"Listen, I\'m going to perform this operation, it\'s the best chance you have, but there is a 10% chance that you\'re going to die. If you\'re comfortable with this, if you\'re comfortable signing off on this, I will do my best,\" and only in that scenario is the doctor allowed to be forgiven if the operation goes wrong.

\*\*0:47:20.6 Vael:\*\* Yeah, so a part of my thinking I\'m noticing is that\... Um\... So I think you\'re very interested in problems of misuse, which I\'m also interested in, but I think I\'m also interested in the problem of, like, I think that it will just be technically hard in order to incentivize an AI to not try to optimize on \[hard to parse\] but to like, be able to take\... So currently, we\'re quite bad at taking human preferences and goals and values and putting those in a mathematical formulation that AIs can optimize, and I think that problem might just be really, really hard. So we might just have an AI that won\'t even give us anything reasonable, and I\'m like, Oh, well, that seems like step one. And then there\'s also a bunch of governance and a bunch of incentives that need to be put in place in terms of holding people accountable, since humans will hopefully be the end user of these things.

\*\*0:48:07.8 Interviewee:\*\* I\'m actually far less worried about the technical side of this. I just finished reading this book about von Neumann, that\'s a little cute biography of him, and there\'s a part where he says, supposedly, that people who think mathematics is complicated only say that because they don\'t know how complicated life is. And I\'m totally messing with the phrasing, but something like that. I actually think any technical problems in this area will be solved relatively easily compared to the problem of figuring out what human values we want to insert into these.

\*\*0:48:46.4 Vael:\*\* Okay, so you think it\'s the taking human values and putting into AI, that technical problem will just get solved?

\*\*0:48:54.8 Interviewee:\*\* If you know what values you want to put in, yeah.

\*\*0:48:56.5 Vael:\*\* Okay, cool. Alright.

\*\*0:49:00.1 Interviewee:\*\* I actually think that problem is the easy problem. I\'m not saying it\'s easy in an absolute sense, I just think that\'s the easier problem.

\*\*0:49:05.1 Vael:\*\* Got it. That feels like the alignment problem to me. So you think the alignment problem is just going to be pretty easy. This seems like a valid thing to think, so\...

\*\*0:49:12.9 Interviewee:\*\* I want to emphasize, I don\'t think it\'ll be easy in absolute terms, I just think it\'ll be the easier of the two problems.

\*\*0:49:17.0 Vael:\*\* Okay, compared to governance and incentives. Yeah, that makes sense.

\*\*0:49:21.2 Interviewee:\*\* That is\... I just have this faith that any technical problems humans can solve, down the line\-- like, eventually. It\'s the non-technical problems that get people all tangled up, because when there\'s no right answer, it really messes up scientists.

\*\*0:49:43.1 Vael:\*\* Yeah, yeah. Yeah, the problem of trying to take human values and what we care about in all the different ways and put them into a mathematical formulation feels difficult to me, and I guess it is a technical problem. I guess I do sort of think of it as a technical problem, but yeah, that makes sense that you\'re just like, Look, we\'ll get that done eventually. And then we have governance, and I\'m like, Oh, yes, governance is totally a mess. Yeah, that makes sense. And I think no one knows how, it\'s an unknown, unsolved problem\--

\*\*0:50:13.5 Interviewee:\*\* Let me put it this way, let me put it this way. If by human values, you mean if like\...

\*\*0:50:18.5 Vael:\*\* What humans intend, having an AI always doing what you say.

\*\*0:50:22.8 Interviewee:\*\* Yeah, so imagine for any conceivable scenario an AI that would have to deal with. We could ask you, what would you do in this case? What I\'m saying is that if for each of these questions, you are able to give a concrete answer to the answered question, such questions can be inserted into our AIs. Like if you are able to come up with clear answers for the questions for which you yourself would have a clear answer for, I think that set of moral constraints, let\'s say, I think that can be more or less inserted into AIs without huge problems.

\*\*0:51:08.3 Vael:\*\* Even as the problems get\-- even as I don\'t eventually have concrete answers, because the problems are like, Should we do X thing on creating nuclear reactor X and etcetera, and I lose control of the\... not lose control, but I can\'t actually visualize the whole space or something, because I\'m too\...

\*\*0:51:25.2 Interviewee:\*\* But at that point, what does alignment mean? Alignment usually means that what the AI does is the same as what a human would do. If there\'s no answer about what the human would do, what is the AI supposed to do?

\*\*0:51:37.2 Vael:\*\* I think it\'s supposed to just keep in mind the things that I care about, so try to avoid side effects in all forms, try to avoid hurting people, except for when that makes sense or something\-- oh, eugh. Anyway, like, doing a whole bunch of\... and, like reporting truthfully, which is also something I want the AI to do. And things like this.

\*\*0:51:56.6 Interviewee:\*\* I guess it\'s one of those, so it\'s a question of maybe a generalization, but I find it slightly hard to believe that an AI that would answer in the exact same way on all of the answered questions that you do have an answer for, when you go outside of that regime, suddenly the AI diverges strongly from what you would have answered had you been smarter. I just think that\'s a weird kind of discontinuity, right. So there\'s this huge set up\-- let\'s suppose on all the questions for which you have an answer, the AI agrees with you. And then you take that a little bit outside of your realm of comprehension, and at that point, suddenly the AI decides, Oof, I\'m freed from the constraints, I can answer whatever. I think that\'s a little bit implausible to me, assuming people did the job correctly.

\*\*0:52:52.9 Vael:\*\* Yeah, I think assuming people do the job correctly is pretty important here. You could have an AI that is deceiving you and giving you the correct answers, as long as you could check, but yeah, assuming that\'s not true, assuming the AI is honestly reporting to you what exactly, like everything that you said at a lower level, I mean, everything that you can confirm, then that seems great. And you\'re like, Well, if you then extend that to regimes where humans can\'t understand it, then things are probably still okay. I think I maybe believe that. I think that maybe things are probably still okay.

\*\*0:53:23.7 Interviewee:\*\* The validation set on whatever project you\'re working on would\... The AI wouldn\'t know necessarily whether this was a question on which you just didn\'t have an opinion. It\'s the same for me when I think about\... A lot of people are concerned that, in terms of basically over-fitting, could you\... So one of the reasons I think we\'ll definitely get an AI that can answer physics questions pretty comprehensively is that I don\'t believe you can ever create a physics AI that can fake its way through all of the train set, sorry, on the validation set, and then suddenly do poorly on something that is outside of it. There\'s so much of physics that if you are able to fake your way through all the way to graduate school\... I guess I\'m thinking in terms of like, if you happen to make it all the way through graduate school, pass all the exams you needed to pass, turned out the papers you turn out. At that point to be like, Oh, actually, I just, I faked my way through all of physics, I don\'t really understand physics. You don\'t not understand physics, like in spite of yourself, because you\...

\*\*0:54:35.0 Vael:\*\* That seems true. I think one of the things I\'m referencing is like a mesa-optimizer, inner optimizer, which is like an AI has some sort of goal. I don\'t know, maybe it\'s like trying to go out, go to red things, but the door is always\... Okay, sorry, try again. So it says, it says it has some sort of goal, kind of like humans do, so\... \[sorry, I\'m now\] restarting \[this sentence\]. Evolution has some sort of goal, evolution is optimizing on something like inclusive genetic fitness. And it\'s like this optimizer that is pushing things. And then there\'s humans, which are the things that are like, it\'s being optimized or whatever. And humans should ideally have the same goal of inclusive genetic fitness, but we don\'t. We\'ve got something that\'s sort of close, but not really. Like we have contraceptives, so we aren\'t really maximizing babies, we have different goals and things we\'re going for, we have like of achievement and all the values that we think are important and stuff. And so this is an example of something\... that, humans are the ones who are trying to make the AI optimize for a thing, and the AIs are like, Sure, I guess I\'ll sort of go for the goal that you want me to, and I have a model of what you want me to do, but I actually internally in my heart have a different goal, and so I\'ll make my way through all of the test sets because I have a model of what you want me to do, but as soon as you release me, then I will do something different. So that\'s like a weird analogy that doesn\'t quite work, but\...

\*\*0:56:00.1 Interviewee:\*\* Yeah, especially for two reasons. One is that I think there is a real likelihood that we are going to become multi-planetary, which is pretty good as far as evolution is concerned, because not only does that mean we\'ve dominated the planet, but we\'ve started having other planets and spreading our genes everywhere. So in some sense, we\'ve done exactly what evolution wants us to do, which is like reproduce our genes far and wide. I have a feeling that no matter how sophisticated and smart and industrialized in AI we get as a species, people aren\'t going to stop wanting to have babies. So like somehow, yes, we don\'t, like, kill everyone whenever we want to and just steal food and all that like animals would do, but kind of like\...

\*\*0:56:49.9 Vael:\*\* Yeah, I don\'t know that evolution wants that, but.

\*\*0:56:51.8 Interviewee:\*\* We\'re still pretty aligned to the basic evolutionary goal. That\'s what you\'re starting from, saying that\'s the original goal, and we\'ve deviated from the original goal. I think actually we\'re well within evolution\'s parameters as far as what are we supposed to do as good evolutionary creatures.

\*\*0:57:07.5 Vael:\*\* Cool. Yeah, that makes sense to me. And then as a last point or something, so ideally, we want AI to do what humans would do if they were smarter, like you said, if we had more time to reflect and if we maybe had more self-control or something. I don\'t know that that would kind of come out of nowhere or something? I guess this is now in the ideal case, it sounds like we\'ve mostly succeeded in aligning the AI with humans\' goals. (Interviewee: \"So what\'s the question?\") Um, ideally, I think I want AI that will be my best self or something, that will not go and\... fast food is the characteristic example, or not going to the gym or something, or do something like if I were living my best life, would it be able to make\... And if I were smarter, would it be able to model what humans would be like as we continue expanding our circle of interest and have moral progress and etcetera. Will AI be able to do that?

\*\*0:58:12.6 Interviewee:\*\* \...I think so, yes. That\'s the only consistent answer with what I\'ve said so far. But I emphasize that you have to be very careful about what do you see as the best version of yourself. Because maybe to some degree, you want the best version of yourself to be the one that goes to the gym regularly, eats healthily. But I don\'t imagine the best version of yourself is someone who does that so religiously that you have no joy in your life. You don\'t want to just only eat healthy food all the time, only work out, fill up every moment of your day with nothing but keeping yourself healthy and occupied and productive. Like sometimes you just want to say, I don\'t want to do anything. I don\'t know where I\'m going with this, but it still comes back to the question of\... it\'s not a well-defined question, if you say, because the best version of myself is better than myself, I cannot conceive of what the best version of myself ought to be, and therefore it\'s kind of a vague worry what the AI would make the best version of yourself. To me, that\'s a weird question, if that makes sense. I think that\'s kind of like, I\'m reducing a little bit, but there is a little bit of that, is that I want the best version of myself, but I can\'t judge what the best version of myself would be, because that best version of myself would be a smarter version of me, which I cannot comprehend. And in that scenario, how can I feel safe that the best version of myself is indeed the best version of myself. And I think, so I think the question has to be a little bit more well-defined then as currently presented.

\*\*0:59:57.1 Vael:\*\* Cool, that makes sense. So what I\'m taking from this is, I\'m like, \"Okay, you think the problem of putting values into AI will not be unsolvable, it will kind of be solved in due time as we go along.\" You\'re more worried about the governance problem. I\'m like, \"yeah, I guess? I don\'t\--\" And you\'re like, \"Well, anything that we could just ask the human about, we\'ll like, we have like a hypothetical answer to what we want the AI to do,\" and I\'m like, yeah, okay, that seems true. I guess we just have to make it so that all the AIs really do ask for human feedback very consistently or something. And there are some issues around there. But, I don\'t know, it does seem like hypothetically it should be possible, because you can just ask the human, in some sense. And I feel like I\'m missing some arguments, but I\'m like, Ooh, food to think about, this is great.

\*\*1:00:39.7 Interviewee:\*\* But I mean\-- one last thing, I know, I have to go too, is\-- One thing I\'ll say is the fact that we can even just discuss the question of\... It actually does seem plausible that we can get our AIs to at least say what we would say. That to me is amazing, because I think four years ago, it would not have been clear what you even meant when you said, I want a language model\-- a model AI to answer moral dilemmas in a way that humans would answer them. It would not have been clear what that meant, unless you just had a classifier where you inserted a video of a scenario and then you said like \"track 1, track 2, classify.\" We have much, much more sophisticated tools at our disposal now, we can just essentially talk to it and say, no, that\'s the wrong answer, I want you to say this in this scenario. That\'s already, in a span of two years, I think is remarkable. I feel like I\'m coming off as like an incredible optimist. I\'ve got my concerns, but I do think so far, people have shown that any well-defined technical problem in AI can be approached, and they\'re not impossible. Yeah.

\*\*1:01:55.6 Vael:\*\* Got it. When would you work on the governance problems, since you seem to think that\'s more of a problem?

\*\*1:02:02.2 Interviewee:\*\* So\... I trust the people that I work with to kind of hit the button when it really needs to be hit. Because right now, like I said, I think people don\'t take it seriously, partly because I don\'t think they really believe it needs to be taken seriously. These researchers are not saying, \"Oh my God, we need to take this seriously, but I\'ve got other stuff to do.\" They really are just like, I don\'t think we\'re at that stage where we need to take it seriously. I think the people that I work with are, on the most part\-- mostly pretty well-intentioned people. There\'s some disagreement over this. \[\...\] I know that within OpenAI there were some healthy discussions about what is the correct deployment model for things like GPT-3. All the discussions that I\'ve had so far give me a lot of confidence that these people aren\'t stupid and they\'re not entirely negligent. I think they\'re possible occasionally overconfident, but they\'re not arrogant, if that makes any sense. As in like, they know they\'re fallible. They don\'t always put the right error bars on their decisions, but they know they\'re not infallible. Ultimately, that\'s what it\'s going to come down to. There\'s not like a system for this. It\'s going to be a few hundreds to a thousands\-- thousand people, doing the sensible thing. For the moment, it looks like that\'s going to happen. But\... I agree that that isn\'t entirely confidence-inspiring. But I think that\'s going to be the way it goes.

Vael: Awesome. Well, thank you so much for talking to me, I found this very enjoyable. And got some things to think about.

\[closings\]