Interview with a0nfw

\*\*Interview with a0nfw, on 3/18/22\*\*

\*\*0:00:02.3 Vael:\*\* Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:08.7 Interviewee:\*\* Yes. More recently I\'ve been working on AI for mathematical reasoning and applications in navigation, mathematical navigation.

\*\*0:00:18.1 Vael:\*\* Great. And then, what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:00:25.8 Interviewee:\*\* The biggest benefit of AI as a tool that now we have and that sort of works, is that it just enabled a lot of new applications when we are able to define goals for it. The broad story that I\'m pursuing recently is like, suppose you have an AI expert for a certain domain, then how can you leverage that expert to teach people to be better at the same domain? And the reason why we\'re working more on mathematical education, it\'s exactly because the mathematical domains are easier to define as a formal problem. Training solvers for interesting mathematical domains is not possible with AI tools, that \[hard to parse\] not be so much 10 years ago. In a lot of other fields, it\'s also been the case that we as humans knew what we wanted and wrote down a bunch of heuristics to accomplish certain goals. But we were always sure that those weren\'t the best, and now AI is allowing us to replace a lot of those by just better algorithms.

The main worry is exactly when we don\'t know exactly what the goal is, or when we don\'t know how to specify it. Or in a lot of cases, in research, people have been stuck with these proxy tasks that are supposed to represent some kind of behavior that we want, but a lot of people are just unsure of what exactly what we want out of these systems. It\'s a worry for me that a lot of people are spending a lot of time and resources on those problems without knowing exactly what to expect. I come from sort of a systems background, so I\'m more used to people having proxy tasks that are very directly related to the actual task. For example, in compilers, usually one goal is, make programs faster, and then you can create benchmarks. \"Okay, here is the set of programs, let\'s run your compiler and it optimize\[?\] and see how long it takes.\" And you can argue about whether those programs are representative of what actual programs that people will run it on, if the benchmark that I have is representative or not, but the goal was very clear, it\'s to make things faster. Now in language for example, in NLP, we have a lot of these tasks about language understanding and benchmarks that try to capture some form of understanding. But it\'s not entirely clear what the goal is, if we solve this benchmark\-- and this is being revised all the time. People propose a benchmark that\'s hard for current-day models, then a few months later, someone comes up with a solution. Then people say, \"Oh, but that\'s actually not exactly what we wanted \'cause look, the model doesn\'t do this other thing.\" And then we enter this cycle of refining exactly what\'s the problem and developing models, but without a clear goal in a lot of cases.

\*\*0:03:47.7 Vael:\*\* Got it. Okay, so that question was, what are you most excited about AI and what are you most worried about in AI? Could you summarize the culmination of all that?

\*\*0:03:57.1 Interviewee:\*\* Yeah. It\'s centered around goals. So AI lets us pose richer and new kinds of goals more formally and optimize for those. When those are clear, those are cases where I\'m very excited about. When they\'re not clear, then I\'m worried about it.

\*\*0:04:16.3 Vael:\*\* Got it. Cool, that makes sense. Awesome. Alright, so focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50 plus years into the future. So at least 50 years in the future, what does that future look like?

\*\*0:04:32.8 Interviewee:\*\* It can look like\... a lot of\... it can go in a lot of ways, I think. I think, in one possible future\-- do you want me to just list the possible futures?

\*\*0:04:48.1 Vael:\*\* I think I\'m most interested in your realistic future, but also like optimistic, pessimistic; it\'s a free-form question.

\*\*0:04:57.8 Interviewee:\*\* Okay. Yeah, maybe optimistically what would happen is that AI lets us solve problems that are important for society and that we just don\'t have the right tools to at this point. I think a lot of exciting applications in\... In language, we can imagine some interesting applications where a lot of computer interfaces become much easier, just for doing complex tasks, to specify a natural language. Like programming right now gives a lot of power to people, but it also takes a lot of time to learn, so it would be awesome to enable a lot more people to automate tasks using natural language. So in one future, a lot of those applications will be enabled and things will be great. In another future, which is actually\-- it\'s probably going to be a combination of these things, but a very likely future, which is already kind of happening, but maybe at a smaller scale than is possible, is that people will start replacing existing tools and systems with AI for not very clear reasons and get not very clear outcomes. And then we don\'t really know exactly how they\'re misbehaving in a lot of cases. And the thing is that the incentives for deploying the systems are at odds with broader societal goals in a lot of cases. So, for example, private companies like Google and Facebook, they have all the incentives possible to deploy these systems to optimize their metrics that ultimately correlate with revenue. I don\'t know.

Facebook has a lot of metrics which ultimately relate to both how much time people spend on the platform, and that translates to how much money they make. And they basically have thousands of engineers trying different combinations of features and things to suggest users to do, and like very small tweaks a lot of times to try to optimize for those through metrics. And a lot of that process became much faster with AI because now you can take much more fine-grain decisions for individual users. And sure, we can look at the metrics and see that, \"Yeah, they keep improving, that\'s a new tool that now exists and enables them to optimize for this goal.\" At the same time, it\'s not exactly to clear to me what is happening, that AI is choosing things for individual users. It might be like closing people off in their bubbles, it might be\... all the things we know about, but also potentially a lot of other things, other behaviors. Exactly because they\'re not the same behavior for everyone, that also makes it harder to study and understand. So I like the future, is that AI will replace a lot of these systems, but that will involve people losing control of exactly what it is doing. And that will probably come before us having a good understanding of what\'s going on, just because there is already an incentive to deploy those systems, like a financial incentive.

\*\*0:08:40.8 Vael:\*\* Yeah, interesting. You\'re making a lot of the arguments that I often make later in this interview. Alright. This next thing is, so people talk about the promise of AI, by which they mean many things, but one thing that I think they mean, and that I\'m talking about here, is that the idea of having a very general, capable system, such that the systems would have cognitive capacities that we could use to replace all current-day human jobs. Which we could or could not, but the cognitive capacities do that. And I usually think about this in the frame of like, here we have\... in 2012, we have AlexNet and the deep learning revolution, and then 10 years later, here we are, and you\'ve got systems like GPT-3, which have some weirdly emergent capabilities like text generation and language translation and math and coding and stuff. And so one might expect that if we continued pouring all the human effort that\'s been going into this, with nations competing and corporations competing and lots of young people and lots of talent and algorithmic improvements at the same rate we\'ve seen and hardware improvements, maybe we\'ll get optical or quantum computing, then we might scale to very general systems. Or we might not, and we might hit some sort of ceiling and need to paradigm shift. But my question is, regardless of how we get there, do you think we\'ll ever have very general systems like a CEO or a scientist AI, and if so, when?

\*\*0:11:00.9 Interviewee:\*\* Yeah. The thing that has been happening with these large language models, that they objectively become more and more capable, but we at the same time reassess what we consider to be general intelligence. That will probably keep happening for quite a while, I would say at least like 10-15 years, just because\...I don\'t think we know exactly what\... Human intelligence is still very poorly understood in a lot of ways. It\'s not clear that higher intelligence is possible without some of the shortcomings that human intelligence has. So I think that cycle of getting more powerful systems but also realizing some of their shortcomings and saying, \"Oh, maybe\...\" Basically shifting the goal and what is it that we consider general intelligence will probably keep happening.

\*\*0:12:07.1 Vael:\*\* Yeah, my question is, do you think that we\'ll get systems like a CEO AI or a scientist AI or be able to replace most cognitive jobs at some point, and when that will be?

\*\*0:12:18.6 Interviewee:\*\* Yeah. Let\'s see. We\'ll probably have systems that can act like a CEO or a scientist. Whether they\'ll have the same goals that a CEO or a scientist have, like a human CEO or a human scientist have in the real world, that is a different question, which is not that clear to me.

\*\*0:12:47.0 Vael:\*\* Interesting. Yeah. So there\'s the question of whether AIs develop like consciousness, for example. So I\'m assuming not consciousness, and I\'m like, \"Okay, say we have a system that can do multi-step planning, it can do models of other people modeling it, it\'s interacting with humans by text and video, or whatever,\" and I\'m like, \"Alright, I need you to solve cancer for me, AI,\" or, \"I want you to run this company,\" and we put in its goal, optimization function so that it just does that at some point. Yeah.

\*\*0:13:24.4 Interviewee:\*\* Yeah. You\'re right. So you asked about the cognitive capability. I think that will probably be there in like 30-40 years, would be my estimate. For at least a subset of what it means to run a company. Now, my question is, if we have a system that does that, is it also possible for the system to simply obey everything we tell it to do? Because, of course, a person is capable of doing that, but you wouldn\'t expect to tell the person, \"Okay, go do that.\" They might complain. They might say \"No, I\'m in a bad mood today,\" or something like that.

\*\*0:14:03.3 Vael:\*\* Yeah, and I don\'t even think you could tell any given human\... Like if someone is like, \"Alright, Vael, go be the CEO of a company,\" I\'d be like, \"What? I don\'t know how to do that.\"

\*\*0:14:09.9 Interviewee:\*\* Exactly, exactly. Yeah.

\*\*0:14:12.0 Vael:\*\* Yeah, yeah. Interesting. Okay, so that leads into my next set of questions. So imagine we have a CEO AI and I\'m like, \"Alright, CEO AI, I wish for you to maximize profits, and try not to run out of money and try not to exploit people and try to avoid side effects.\" And so currently, this would be very technically challenging for a number of reasons. One reason is that we aren\'t very good at taking human values and preferences and stuff and putting them into mathematical formulations such that AI can optimize over. And so I worry that AI in the future will continue to do this, and then we\'ll have AI that continue to do what we tell them to do instead of what we intend them to do, but as our expressions get more and more ambiguous, and higher scale and operating on more of the world. And so what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?\"

\*\*0:15:22.1 Interviewee:\*\* Yeah, if I understand what you\'re saying, that is very clear to me. That\... specifying\... If you just hope to give a short description of what you want, you\'ll most likely fail.

\*\*0:15:35.9 Vael:\*\* Even a long definition would be fine. I just want there to be any system of putting what we want into the AI.

\*\*0:15:46.0 Interviewee:\*\* Yeah, so I think there might be a safe way to do that, but it might require a lot of interaction and will be hard. I have thought a little bit about that, because I was thinking about the question of how do we\... So if we want to use these systems for interfacing with people, then we have to let them handle ambiguity in a way that\'s similar to how people do. And in cognitive science, there is a lot of knowledge about the mechanisms that people use to resolve ambiguity, like there is context, prior knowledge, common ground, and all that. Some of that could be replicated now to fully align the way that AI resolves ambiguity with how people do it. We\'ll also probably take a lot of work in understanding how people do it, which is not there yet.

\*\*0:16:50.3 Vael:\*\* Interesting. Interesting. Do you think that we\'ll have to know enough about human psychology? Like, what fields need to advance in order for us to have systems that do what we intend them to do?

\*\*0:17:06.1 Interviewee:\*\* Yeah, if you\'re talking about aligning a system with human values, part of that problem is understanding human values and like how people agree on values. And there\'s just so much that we assume that we don\'t have to tell people, it\'s assumed that they were created in the same way and exposed to the same contexts. All that goes away with AI, or at least most of it. And I\'m not an expert in these fields, so I might not know the exact names. I guess pragmatic reasoning is one thing that we have a lot of high-level insights on how it works, but to operationalize it into an NLP system, for example, we\'re still very far away from that.

\[Person\] worked on this model for pragmatic reasoning, which can explain very neatly some very simple cases. Like, you have a few words and a few people, and some words describe these people. You have a three-by-three matrix that predicts exactly what people would guess the word is referring to with very high accuracy. But just extending that to a sentence and the three people to be an image is a very complicated problem. So I think that is something that would need to advance a lot. Other fields of psychology that try to understand ambiguity like of cognitive psychology. Yeah. I don\'t know their names, but I think\... Oh well, yeah. I guess another one in linguistics that is relevant for this is human convention formation. Like when we are talking to a person over time, we develop these partner-specific conventions. That\'s very natural, enables very efficient communication. And if you\'re writing a description of what you want in a long form to an AI\-- Oh, if you\'re doing so for a person, you assume that the person is picking up on what we want and forming the same conventions over time. So at first you might say, \"I want to optimize the revenues for XYZ company.\" And then later you might say \"the company,\" dropping the XYZ, but you assume that\-- the only company that I mentioned was XYZ, so the only interpretation possible for a company is attribute the XYZ company. Well, that\'s because humans form conventions that way, but if you have kind of an adversary AI, then suddenly that\'s not an assumption you can build off of anymore. So the ways in which humans communicate with conventions, with ambiguity and all the computational tools that we need to do that are open problems, if that makes sense.

\*\*0:20:05.3 Vael:\*\* Yeah. That does seem right. Thanks. Alright, so this next question is still about the CEO AI. So imagine we have the CEO AI that is capable of multi-step planning and has a model of itself in the world. So it\'s modeling other people modeling it because that seems really important in order to have a CEO. So it\'s making these plans for the future and it\'s trying to optimize for profit with the constraints I\'ve mentioned. And it\'s noticing that some of its plans fail because the humans shut it down. And so we built into this AI and its loss function that it has to get human approval for things because it seems like a basic safety measure. (Interviewee: \"Sorry, it has to get what?\") It has to get approval for any action from humans, like a stakeholder-type thing. And so the humans have asked for a one-page memo from the AI just to discuss what it\'s doing. And so the AI is thinking about what to put in this memo, and it\'s like, \"Maybe I should omit some relevant information that the humans would want because that would reduce the chance that the AIs \[note: I meant to say \"humans\"\] would shut me down and increase the likelihood of my plans succeeding, of optimizing my goal.\" And so in this case, we\'re not building self-preservation into the AI, what we\'re doing is having an AI that\'s an agent that is optimizing over a goal. And so instrumentally, it has the goal of self-preservation. So what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?\"

\*\*0:21:30.7 Interviewee:\*\* Yeah. I think if we get systems that have that capability to do the things that we said in the beginning, like planning, in the most extreme form, then I agree with this statement.

\*\*0:21:44.6 Vael:\*\* Interesting. Do you think we\'ll ever\-- hm, so\... We were talking about whether you thought that we would get to that point. What was the answer to that?

\*\*0:21:54.9 Interviewee:\*\* The answer is that I think so, although I\'m not sure to what extent. I\'m not sure if the most powerful possible version will actually happen in 30-40 years. But I think there\'s, there\'s like a small chance, but\-- it\'s not impossible.

\*\*0:22:11.1 Vael:\*\* Interesting. So yeah, within our lifetimes. Yeah, so I\'m worried about the possibility that we don\'t get the alignment problem right per se. And we end up with systems that are now optimizing against humans, which seems bad, especially if they are smarter than humans. Yeah. How likely do you think this sort of scenario would happen?

\*\*0:22:34.7 Interviewee:\*\* The scenario of having super-capable AI that is---

\*\*0:22:38.3 Vael:\*\* That has an instrumental incentive to do self-preservation or other instrumental incentives like power-seeking or acquiring resources or improving itself.

\*\*0:22:48.1 Interviewee:\*\* Yeah, I think it is small but not zero.

\*\*0:22:57.9 Vael:\*\* Yeah. How bad do you think that would be if such a thing happened?

\*\*0:23:03.4 Interviewee:\*\* Yeah, I think pretty bad.

\*\*0:23:05.9 Vael:\*\* Pretty bad. Yeah. Like what would happen?

\*\*0:23:10.4 Interviewee:\*\* Yeah, it\'s not exactly clear to me how that would look like. But it is a little scary to think about humans losing control in some ways. The worry is exactly that we can\'t\... like if we\'re not under control, we can\'t know exactly what comes.

\*\*0:23:30.6 Vael:\*\* Yeah. That seems true. Have you heard of the\-- What does AI safety mean to you? That\'s my first question.

\*\*0:23:39.3 Interviewee:\*\* AI safety. I understand it as a field that\... When I think of AI safety, the first defining problem that comes for me is AI alignment. This question, \"How do you specify your goals in a way that aligns with human values?\" and all that.

\*\*0:24:00.6 Vael:\*\* That makes sense. Yeah. When did you start learning about AI alignment or when did you start caring about it?

\*\*0:24:05.9 Interviewee:\*\* Ah, it was mostly when I came to the \[position\] at \[university\], because I really wasn\'t that much of an AI person before. I kind of incidentally got into AI because it gave me tools to think about problems that were hard before. And I happened to get a bit more involved with the AI safety community. \[Friend\'s name\], you might know how him. (Vael: \"Ah! Nice.\") We\'re very close friends. We lived together for almost three years, and had a lot of conversations.

\[short removed segment, discussing situation and friend\]

\*\*0:25:09.2 Vael:\*\* Yeah. Some visions that I have, why it could be bad, is if we have an AI that\'s like very powerful, then if it\'s incentivized to get rid of humans, then maybe it\'s intelligent enough to do synthetic biology against humans, or nanotechnology or whatever, or maybe just increase the amount of pollution. I don\'t know. I just feel like there\'s ways in which you could make the environment uninhabitable for humans via like putting things in the water or in the air. It\'d be pretty hard to do worldwide, but seems maybe possible if you\'re trying. Yeah. Also we\'ve got like misuse and AI-assisted war, and like maybe if we put AI in charge of like food production or manufacturing production, then we can have a lot of correlated supply chain failures. That\'s another way, but I don\'t know if that would kill everyone of course, but that could be quite bad.

\*\*0:26:12.5 Interviewee:\*\* Yeah, that is something that I worry more about in the short-term, which is AI in the hands of\... Even currently, AI can already be pretty bad with like spreading misinformation. That certainly happened here in \[country\], but in \[other country\] in the last elections, where this really bad president got elected, he used a lot of misinformation campaigns funded by a lot of people. And it\'s super hard to track because it\'s on WhatsApp and it\'s like the double-edged sword, the sword of privacy. It aligns with things that we care about, but which also allows these campaigns to run at massive scale and not be tracked. Even currently, AI personalizing wordings and messages to different people at large scale could be really pretty \[hard to parse, \"good\"?\].

\*\*0:27:20.1 Vael:\*\* Got it. Do you work on AI alignment stuff?

\*\*0:27:26.0 Interviewee:\*\* So currently, I\'m working more on the education side of research. I still keep in the back of my head some of the problems about the ambiguity thing. We did have one paper last year that started building some of those ideas but I\'m not actively working on that. But it\'s still like a realm of problems that I think is important for a number of reasons.

\*\*0:27:56.2 Vael:\*\* Yeah. That makes sense. What would cause you to work on it?

\*\*0:28:04.4 Interviewee:\*\* Sorry, what what would make me work on it?

\*\*0:28:08.3 Vael:\*\* Yeah, like in what circumstance would you be like: Whelp, it is now five year later, it is now one year later, and I happen to be somehow working on some element of the AI alignment problem. Or AI safety or something. How did that happen?

\*\*0:28:27.8 Interviewee:\*\* Yeah, it\'s interesting. From my trajectory, I have always tried to work on the thing that I feel is not being done and that I\'m in a certain position to do. And that has over time shifted towards education for a number of reasons. And I don\'t know exactly where that comes from, but I still feel that I\'m more able to contribute there than with other AI problems. Although I guess that kind of thing is hard to assess, but it\'s still my feeling on when deciding what things to work on.

\*\*0:29:19.9 Vael:\*\* Hm, so it sounds like there\'s maybe something like\... Okay, so I don\'t quite know if it\'s a matter of whether the education bit is neglected by the world, or whether it\'s like, you feel like it would be a better fit per se, or both?

\*\*0:29:35.6 Interviewee:\*\* Yeah, it\'s the combination of both. Like if it\'s neglected. It is important, important in the short-term. And I think there are like things that I could do in the next five years. If the plan that I have for the \[position\] I have works out, I think it would be great. Like we would have tools that people would be able to use, and they need technical advances that are in the intersection of things that I care about or know for very weird reasons. But I feel like I\'m in a position to make progress there and very immediately. So yeah, I just feel very urged to see how that goes.

\*\*0:30:21.3 Vael:\*\* Yeah. Makes sense. Yeah, do you ever think you\'ll go get into the AI alignment thing or you\'ll just be like, \"Woo, seems good for someone to work on it?\" I get that\-- that\'s an impression that I get.

\*\*0:30:32.6 Interviewee:\*\* Uh huh, yeah. I think I\'m getting a taste of it through \[friend\]. We actually wrote this project together two months ago. Okay, I actually met with \[friend\] to discuss this a few times but I have no idea where it\'s exactly came from, I think from someone at LessWrong that posted like a challenge. So it\'s a program called ELK, which I forgot even what ELK means, but I met with \[friend\] and we ended up coming up with some ideas, and he submitted it. \[\...\]

\*\*0:31:16.9 Interviewee:\*\* So I guess I\'m getting a little bit of flavor of the kind of work that that entails. I still don\'t feel exactly comfortable in doing it myself just because it requires this very hypothetical and future reasoning that I\'m not used to, not that I couldn\'t. But I understand that thinking about that kind of problem today requires you to think of systems that we can\'t use today and we can\'t run and test today. And I\'m very used to ideas that I can write down the code and see how they work and then make progress from there. So I don\'t know, my opinion and comfort might change.

\*\*0:32:18.5 Vael:\*\* Yeah, I also encountered this where I was like, \"I don\'t wanna think about far-future programs or far-future things, where there\'s no feedback loops and it\'s very hard to tell what\'s happening and there\'s only a limited number of people, and then\-- now there\'s money, but there wasn\'t necessarily very much money before\-- and there\'s not really journals, and everything\'s pre-pragmatic.\" And then, I think I was won over by the argument that, I don\'t know, I think existential risk is probably the thing that I\'ll just work on for most of my life because it seems like really important, and like, probable. But then I tried technical research for a while, and I thought it was not my cup of tea. And then I got into AI community building, which is what I\'m kind of currently doing, and trying to get more people to become AI technical researchers. But not me! It\'s been a good fit in some sense.

\*\*0:33:10.1 Interviewee:\*\* I see, that makes sense.

\*\*0:33:11.1 Vael:\*\* Yeah, and I think it\'s pretty interesting as well. Well, the current thing I\'m doing is talking to AI researchers and being like, \"What do you think of things? What about this argument?\" \...Cool. Well, we\'ve got a little bit of time left, so, see if I can ask some of my more unusual questions, which are\... How much do you think about policy and what do you think about policy related to AI?

\*\*0:33:34.6 Interviewee:\*\* Yeah, I think I\'ve\... I think a lot about policy in general, about just related to AI. I\'m currently at this phase in my life where I think a lot of the immediate problems that we have, unfortunately in a sense, we\'ll have to resolve by good politics, and I\'m not exactly sure yet how to do that. Because one other thing that I think about, when I think about AI safety, is that one thing is to have the technical ability to do certain things. The other is to get the relevant players to implement them. So, if there is an AI safety mechanism that can like limit the reach of Google\'s super AI that will do all things, but it also means that it will decrease revenue to some extent\... I\'m not sure the decision that they will make which would be to implement the safe version. Think about VC\'s that are sitting in these boards and pushing decisions for the company. I\'m not sure if all of them actually care about this and realize that it\'s a problem. So I think part of the solution is also on the political side, of like us as a society sitting down and deciding, \"What do we gotta do?\" More than just the technical proposal. I\'ve decided that this year, I will try to have some political involvement in \[country\], which is a place where I can. I\'m currently in the planning phase, but I\'ve committed to doing something this year, because we have elections this year.

\*\*0:35:39.2 Vael:\*\* Interesting, yeah. I think AI governance is very, very important. I think the community\-- the AI safety community also, like\... or, I don\'t know, the Effective Altruism community \[hard to parse\] a lot in their listings. Alright, question: if you could change your colleagues\' perceptions of AI, what attitudes or beliefs would you want them to have? So what beliefs do they currently have and how would you want those to change?

\*\*0:36:05.9 Interviewee:\*\* I\'m not sure if this is the kind of attitude that you\'re talking about, but one attitude that I really don\'t like and is very prevalent in papers is that people kind of self-identify with the method that they\'re proposing so much that they call it \"our method\". In the sense that the goal should not necessarily be to show off better numbers, and show a curve where the color that\'s associated with you is like higher than the other. But it should really be to understand what\'s going on, right? If we have a system that has some sort of behavior that\'s desirable, that\'s awesome, and we should also look at how it maybe doesn\'t. Like people include a lot of cases in their papers where, \"Oh look, the model is doing this.\" But then when we actually try out the model, it\'s very easy to find cases where it doesn\'t or it fails in weird ways. And the reluctance to put that comes from this attachment from like, \"Oh, this is like me, but in form of a method that is written in this paper.\" And I wish that people didn\'t have that attitude, that they were more detached and scientific about it.

\*\*0:37:24.9 Vael:\*\* Yeah, makes sense. Cool. And then my last question is, why did you choose this interview, and how has this interview been for you?

\*\*0:37:32.6 Interviewee:\*\* Yeah, it\'s been really fun. I had thought of some of the questions before, and not of others. Yeah, the exercise that I maybe haven\'t done that much is to think super long-term, like 50-100 years in the future. I thought about it some but I don\'t think I have the right tools yet to think about them, so I get in this hard-to-process state. Yeah, so I came mostly out of curiosity, I didn\'t know exactly what to expect. If you don\'t mind sharing, but if you do, that\'s okay, how did you pick the people to interview?

\*\*0:38:28.3 Vael:\*\* Yeah, so this is a pretty boring answer, but I originally selected people who\'d submitted papers to NeurIPS or ICML in 2021, and like yeah, then some proportion of people replied back to me.

\*\*0:38:44.5 Interviewee:\*\* I see.

\*\*0:38:47.2 Vael:\*\* Yeah, great, cool. That makes a lot of sense. Thanks you so much for your time. I will send you the money and some additional resources if you\'re curious about it. I\'m sure \[friend\] has already shown you some, but if you\'re curious. And thanks so much for doing this.

\*\*0:39:00.7 Interviewee:\*\* Yeah, no, definitely. Thank you for the invite, this was fun.

\*\*0:39:05.1 Vael:\*\* All right. Bye.

\*\*0:39:05.8 Interviewee:\*\* Bye.


Interview with cvgig

\*\*Interview with cvgig, on 3/24/22\*\*

\*\*0:00:02.5 Vael:\*\* Awesome. Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:09.8 Interviewee:\*\* Yeah. So I\'m what\'s technically called a computational neuroscientist, which is studying, using mathematics, AI and machine learning techniques to study the brain. Rather than creating intelligent machines, it\'s more about trying to understand the brain itself. And I study specifically synaptic plasticity, which is talking about how the brain itself learns.

\*\*0:00:44.0 Vael:\*\* So these questions are like, AI questions, but feel free to like\-- (Interviewee: \"No, go ahead.\") Okay, cool. Sounds good. Alright. What are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:00:55.9 Interviewee:\*\* Right. So in terms of benefits, I think that my answer might be a little bit divergent again, because I\'m a computational neuroscientist. But I think that AI and the tools surrounding AI give us a huge amount of power to understand both the human brain, cognition itself, and more general phenomena in the world. I mean, you see AI used in physics and in other areas. I think that it is just a very powerful tool in general for building understanding. In terms of risks, I think that it\'s, again, by virtue of being a very powerful tool, also something that can be used for just a huge number of nefarious things like governmental surveillance, to name one, military targeting technology and things like that, that could be used to kill or harm or disenfranchise large numbers of people in an automated way.

\*\*0:02:04.2 Vael:\*\* Awesome, makes sense. Yeah, and then focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50-plus years into the future. So at least 50 years in the future, what does that future look like? This is not necessarily in terms of AI, but if AI is important, then include AI.

\*\*0:02:22.6 Interviewee:\*\* Yeah, so 50-plus years in the future. I always have trouble speculating with things like this. \[chuckle\] I think it\'ll be way harder than people tend to be willing to extrapolate. And also, I think that AI is not going to play as large of a role as someone might think. I think that\... I don\'t know, I mean in much the same way, I think it\'ll just be the same news with a different veneer. So we\'ll have more powerful technology, we\'ll have artificial intelligence for self-driving cars and things like that. I think that the technologies that we have available will be radically changed, but I don\'t think that AI is really going to fundamentally change the way that people\... Whether people are kind or cruel to one another, I guess. Yeah, is that a good answer? I don\'t know. \[chuckle\]

\*\*0:03:21.8 Vael:\*\* I\'m looking for your answer. So\...

\*\*0:03:26.3 Vael:\*\* Yes. 50 years in the future, you\'re like, it will be\... Society will basically kind of be the same as it is today. There will be some different applications than exists currently.

\*\*0:03:36.4 Interviewee:\*\* Yeah, unless it\'s\... It\'s perfectly possible society will utterly collapse, but I don\'t really think AI will be the reason for that. \[chuckle\] So, yeah, right.

\*\*0:03:47.5 Vael:\*\* What are you most worried about?

\*\*0:03:50.9 Interviewee:\*\* In terms of societal collapse? I\'d say climate change, pandemic or nuclear war are much more likely. But I don\'t know, I\'m not really betting on things having actually collapsed in 50 years. I hope they don\'t, yeah. \[chuckle\]

\*\*0:04:07.0 Vael:\*\* Alright, I\'m gonna go on a bit of a spiel. So people talk about the promise\...

\*\*0:04:10.7 Interviewee:\*\* Yeah, yeah.

\*\*0:04:12.6 Vael:\*\* \[chuckle\] Yeah, people talk about the promise of AI, by which they mean many things, but one of the things they may mean is whether\... The thing that I\'m referencing here is having a very generally capable system, such that you could have an AI that has the cognitive capacities that could replace all current day jobs, whether or not we choose to have those jobs replaced. And so I often think about this within the frame of like 2012, we had the deep learning revolution with AlexNet, and then 10 years later, here we are and we have systems like GPT-3, which have some weirdly emergent capabilities, like they can do some text generation and some language translation and some coding and some math.

\*\*0:04:42.7 Vael:\*\* And one might expect that if we continue pouring all of the human effort that has been going into this, like we continue training a whole lot of young people, we continue pouring money in, and we have nations competing, we have corporations competing, that\... And lots of talent, and if we see algorithmic improvements at the same rate we\'ve seen, and if we see hardware improvements, like we see optical or quantum computing, then we might very well scale to very general systems, or we may not. So we might hit some sort of ceiling and need a paradigm shift. But my question is, regardless of how we get there, do you think we\'ll ever get very general systems like a CEO AI or a scientist AI? And if so, when?

\*\*0:05:20.6 Interviewee:\*\* Yeah, so I guess this is somewhat similar to my previous answer. There is definitely an exponential growth in AI capabilities right now, but the beginning of any form of saturating function is an exponential. I think that it is very unlikely that we are going to get a general AI with the technologies and approaches that we currently have. I think that it would require many steps of huge technological improvements before we reach that stage. And so things that you mentioned like quantum computing, or things like that.

\*\*0:06:00.1 Interviewee:\*\* But I think that fundamentally, even though we have made very large advances in tools like AlexNet, we tend to have very little understanding of how those tools actually work. And I think that those tools break down in very obvious places, once you push them beyond the box that they\'re currently used in. So, very straightforward image recognition technologies or language technologies. We don\'t really have very much in terms of embodied agents working with temporal data, for instance. I think that\...

\*\*0:06:42.2 Interviewee:\*\* I essentially think that even though these tools are very, very successful in the limited domains that they operate in, that does not mean that they have scaled to a general AI. What was the second half of your question? It was like kind of, Given that we\... Do you have it what it\'ll look like, or\...

\*\*0:06:57.4 Vael:\*\* Nah, it was actually just like, will we ever get these kind of general AIs, and if so, when? So\...

\*\*0:07:03.2 Interviewee:\*\* Yeah, so I would essentially say that it\'s too far in the future for me to be able to give a good estimate. I think that it\'s 50 plus years, yeah.

\*\*0:07:13.2 Vael:\*\* 50 plus years. Are you thinking like a thousand years or you\'re thinking like a hundred years or?

\*\*0:07:19.7 Interviewee:\*\* I don\'t know. I mean, I hope that it\'s earlier than that. I like the idea of us being able to create such things, whether we would and how we would use them. I would not, \[chuckle\] I don\'t think I would want to see a CEO AI, \[chuckle\] but there are many forms of general artificial intelligence that could be very interesting and not all that different from an ordinary person. And so I would be perfectly happy to see something like that, but I just, you know, and I guess in some sense, my work is hopefully contributing to something along those lines, but I don\'t think that I could guess when it would be, yeah.

\*\*0:08:00.2 Vael:\*\* Yeah. Some people think that we might actually get there via by just scaling, like the scaling hypothesis, scale our current deep learning system, more compute, more money, like more efficient, more like use of data, more efficiency in general, yeah. And do you think this is like basically misguided or something?

\*\*0:08:15.9 Interviewee:\*\* Yeah, let me take a moment to think about how to articulate that properly. I think\... Yeah, you know, let me just take a moment. I think that when you hear people like, for instance, Elon Musk or something along these lines saying something like this, it reflects how a person who is attempting to get these things to come to pass and has a large amount of money would say something, right. It\'s like, what I\'m doing is I\'m pouring a large amount of money into this system and things keep on happening, so I\'m happy with that. But I think that from my position of seeing how much work and effort goes into every single incremental advance that we see, I think that it\'s just, there are so many individual steps that need to be made and any one of them could go wrong and provide a, essentially a fundamental sealing on the capabilities that we\'re able to reach with our current technologies. And so it just seems a little, a little hard to extrapolate that far in the future.

\*\*0:09:25.5 Vael:\*\* Yeah. What kind of things do you think we\'ll need in order to have something like, you know, a multi-step planner can do social modeling, can model all of the things modeling it like that kind of level of general.

\*\*0:09:35.5 Interviewee:\*\* Yeah. So I think that one of the main things that has made vision technologies work extremely well is massive parallelization in training their algorithms. And I think that, what this reflects is the difficulty involved in training a large number\... So essentially, when you train an algorithm like this, you have a large number of units in the brain like neurons or something like that, that all need to change their connections in order to become better at performing some task. And two things really tend to limit these types of algorithms, it\'s the size and quality of the data set that\'s being fed into the algorithm and just the amount of time that you are running the algorithm for. So it might take weeks to run a state-of-the-art algorithm and train it now. And you can get big advances by being able to train multiple units in parallel and things like that.

\*\*0:10:33.5 Interviewee:\*\* And so I think that the easiest way to get very large data sets and have everything run in parallel is with specialized hardware called, you know, people would call that wetware or neuromorphic computing or something along those lines. Which is currently very, very new and has not really, as far as I know, been used for anything particularly revolutionary up to this point. You can correct me if I\'m wrong on that. I would expect that you would have to have essentially embodied agents before you can get\... in a system that is learning and perceiving at the same time before you could get general intelligence.

\*\*0:11:12.5 Vael:\*\* Well, yeah, that\'s certainly very interesting to me. So, it\'s not\... So people are like, \"We definitely need hardware improvements.\" And I\'m like, \"Yup, current day systems are not very good at stuff. Sure, we need hardware improvements.\" And you\'re saying, are you saying we need to like branch sideways and do wetware\-- these are like biological kind of substrates, or are they different types of hardware?

\*\*0:11:37.3 Interviewee:\*\* I guess different types of hardware is maybe the shorter term goal on something like that. Like you would expect circuits in which individual units of your circuit look a little bit like neurons and are capable to adapt their connections with one another, running things in parallel like that can save a lot of energy and allows you to kind of train your system in real time. So it seems like that has some potential, but it\'s such a new field that, this is when I, when I think about what time horizon you would need for something like this to occur, it seems like you would need significant technological improvements that I just don\'t know when they\'ll come.

\*\*0:12:20.4 Vael:\*\* Yeah. So I haven\'t heard of this wetware concept. So like it\'s a physical substrate that like\... It like creates, it creates new physical connections like neurons do or it just like, does, you know\...

\*\*0:12:33.5 Interviewee:\*\* No, it doesn\'t create physical connections. You could just imagine this like\... So, you know, computer systems have programs that they run in kind of an abstract way.

\*\*0:12:43.8 Vael:\*\* Yep.

\*\*0:12:44.8 Interviewee:\*\* And the hardware itself is logic circuits that are performing some kind of function.

\*\*0:12:48.9 Vael:\*\* Yep.

\*\*0:12:49.8 Interviewee:\*\* And neuromorphic computing is individual circuits in your computer have been specially designed to individually look like the functions that are used in neural networks. So you have\... Basically, the circuit itself is a neural network, and because you don\'t have these extra layers of programming added in on top, you can run them continuously and have them work with much lower energy and stuff like that. It\'s just\... It\'s limiting because they can\'t implement arbitrary programs, they can only do neural network functions, and so it\'s kind of like a specialized AI chip. People are working on developing that now\... Yeah.

\*\*0:13:32.7 Vael:\*\* Okay, cool, so this is one of the new hardware-like things down the line. Cool, that makes sense. Alright, so you\'d like to see better hardware, probably you\'d say that you\'d probably need more data, or more efficient use of data. Presumably for this\-- because the kind of continuous learning that humans do, you need to be able to have it acquire and process continuous streams of both image and text data at least. Yeah, what else is needed?

\*\*0:14:03.8 Interviewee:\*\* Oh, I think that\... Yeah, more fundamentally than either of those things. It\'s just the fact that we don\'t understand what these algorithms are doing at all. And so we\'re\... You can train it, you can train an algorithm and say, \"Okay, you know, it does what I want it to do, it performs well,\" and most machine learning techniques are not very good at actually interrogating what a neural network is actually doing when it\'s processing images. And there are many instances recently, I think the easiest example is adversarial networks, if you\'ve heard of those?

\*\*0:14:41.8 Vael:\*\* Mm-hmm.

\*\*0:14:42.2 Interviewee:\*\* I don\'t know what audience I\'m supposed to be talking to in this interview.

\*\*0:14:46.4 Vael:\*\* Yeah, just talk to me I think.

\*\*0:14:49.2 Interviewee:\*\* Okay, okay.

\*\*0:14:50.1 Vael:\*\* I do know what adversarial\... Yeah.

\*\*0:14:52.9 Interviewee:\*\* Okay, so, adversarial networks are\... You perturb images in order to get your network to output very weird answers. And the ability of making a network do something like that, where you are able to change its responses in a way that\'s very different from the human visual system by artificial manipulations, makes me worried that these systems are not really doing what we think they\'re doing, and that not enough time has been invested in actually figuring out how to fix that, which is currently a very active area of research, and it\'s partly limited by the data sets that we\'ve been showing our neural networks. But I think in general, there\'s been too much of an emphasis on getting short-term benefits in these systems, and not enough effort on actually understanding what they\'re learning and how they work.

\*\*0:15:43.5 Vael:\*\* That makes sense. Do you think that the trend\... So if we\'re at the point where people are deploying things that you don\'t understand very well, do you think that this trend will continue and we\'ll continue advancing forward without having this understanding, or do you think it would catch up or\...

\*\*0:16:00.4 Interviewee:\*\* Yeah, well, I think it\'s reflective of the huge pragmatic influence that is going on in machine learning, which is essentially, corporations can make very large amounts of money by having incremental performance increases over their preferred competitors. And so, that\'s what\'s getting paid right now. And if you look at major conferences, the vast majority of papers are not probing the details of the networks that they\'re training, but are only showing how they compare it to competitors. They\'ll say, \"Okay, mine does better, therefore, I did a good job,\" which is really not\... It\'s a good way to get short-term benefits to perform, essentially, engineering functions, but once you hit a boundary in the capabilities of your system, you really need to have understanding in order to be able to be advanced further. And so I really think it\'s the funding structure, and the incentive structure for the scientists that\'s limiting advancement.

\*\*0:17:02.2 Vael:\*\* That makes sense. Yeah, and again, I hear a lot of thoughts that the field is this way and they have their focus on benchmarks is maybe not\... and incremental improvements in state-of-the-art is not necessarily very good for\... especially for understanding. When I think about organizations like DeepMind or OpenAI, who\'re kind of exclusively or\... explicitly aimed at trying to create very capable systems like AGI, they\... I feel like they\'ve gotten results that I wouldn\'t have expected them to get. It doesn\'t seem like you could should just be able to scale a model and then you get something that can do text generation that kind of passes the Turing Test in some ways, and do some language translation, a whole bunch of things at once. And then we\'re further integrating with these foundational models, like the text and video and things. And I think that those people will, even if they don\'t understand their systems, will continue advancing and having unexpected progress. What do you think of that?

\*\*0:18:09.6 Interviewee:\*\* Yeah, I think it\'s possible. I think that DeepMind and OpenAI have basically had some undoubtedly, extremely impressive results, with things like AlphaGo, for instance. What\'s it called, AlphaStar, the one that plays StarCraft. There are lots of really interesting reinforcement learning examples for how they train their systems. Yeah, I think it just remains to be seen, essentially. It would be nice\-- Well, maybe it wouldn\'t be nice, it would be interesting to see if you can just throw more at the system, throw more computing capabilities at problems, and see them end up being fixed, but I\...

\*\*0:19:04.0 Interviewee:\*\* I\'m just skeptical, I guess. It\'s not the type of work that I want to be doing, which is maybe biasing my response, and I don\'t think that we should be doing work that does not involve understanding for ethical reasons and advancing general intelligence. For reasons that I stated, that essentially, if you hit a wall you\'ll get very stuck. But yeah, you\'re totally right that there have had been some extremely, extremely impressive examples in terms of the capability capabilities of DeepMind. And, yeah, there\'s not too much to be said for me on that front.

\*\*0:19:46.8 Vael:\*\* Yeah. So you said it would be interesting, you don\'t know if it would will be nice. Because one of the reasons that it maybe wouldn\'t be nice is that you said that there\'s ethical considerations. And then you also said there\'s this other thing; if you don\'t understand things then when you get stuck, you really get stuck though.

\*\*0:20:01.5 Interviewee:\*\* Yeah.

\*\*0:20:04.4 Vael:\*\* Yeah, it seems right. I would kind of expect that if people really got stuck, they would start pouring effort into interpretability work for other types of things.

\*\*0:20:12.7 Interviewee:\*\* Right. You would certainly hope so. And I think that there has been some push in that direction, especially there\'s been a huge\... I keep on coming back to the adversarial networks example, because there have actually been a huge number of studies trying to look at how adversarial examples work and how you can prevent systems from being targeted by adversarial attacks and things along those lines. Which is not quite interpretability, it\'s still kind of motivated by building secure, high performance systems. But I think that you\'re right, essentially, once you hit a wall, things come back to interpretability. And this is, again, circling back to this idea of every saturating function looks like an exponential at the beginning, is that the deep learning is currently in a period of rapid expansion, and so we might be coming back to these ideas of interpretability in 10 years or so, and we might be stuck in 10 years ago or so, and the question of how long it\'ll take us to get general artificial intelligence will seem much more inaccessible. But who knows.

\*\*0:21:26.8 Vael:\*\* Interesting. Yeah, when I think about the whole of human history or something, like 10,000 years ago, things didn\'t change in lifetime to lifetime. And then here we are today where we have probably been working on AI for under 100 years, like about 70 years or something, and we made a remarkable amount of progress in that time in terms of the scope of human power over their environment, for example. So yeah, there certainly have been several booms and bust of cycles, so I wouldn\'t be surprised if there is a bust of cycle for deep learning. Though I do expect us to continue on the AI track just because it\'s so economically valuable, which especially with all the applications that are coming out.

\*\*0:22:04.1 Interviewee:\*\* Yeah, you don\'t have to be getting all the way to AI for there not to be plenty of work to be\... General artificial intelligence, for there to be plenty of work to be done. There are hundreds of untapped ways to use, I\'m sure, even basic AI that are currently the reason that people are getting paid so well in the field, and there\'s a lack of people to be working in the field, so there\'s\... I don\'t know, there are tons of opportunities, and it\'s gonna be a very long time before people get tired of AI. So yeah, that\'s not gonna happen anytime soon.

\*\*0:22:36.6 Vael:\*\* True. Alright, I\'m gonna switch gears a little bit, and ask a different question. So now, let\'s say we\'re in whatever period we are where we have this advanced AI systems. And so we have a CEO AI. And a CEO AI can do multi-step planning and as a model of itself modelling it and here we are, yeah, as soon as that happens. And so I\'m like, \"Okay, CEO AI, I wish for you to maximize profits for me and try not to run out of money and try not to exploit people and try to avoid side-effects.\" And obviously we can\'t do this currently. But I think one of the reasons that this would be challenging now, and in the future, is that we currently aren\'t very good at taking human values and preferences and goals and turning them into optimizations\-- or, turning them into mathematical formulations such that they can be optimized over. And I think this might be even harder in the future\-- there\'s a question, an open question, whether it\'s harder or not in the future. But I imagine as you have AI that\'s optimizing over larger and larger state spaces, which encompasses like reality and the continual learners and such, that they might alien ways of\... That there\'s just a very large shared space, and it would be hard to put human values into them in a way such that AI does what we intended to do instead of what we explicitly tell it to do.

\*\*0:23:57.9 Vael:\*\* So what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to and this is dangerous?\"

\*\*0:24:07.1 Interviewee:\*\* Oh, I completely agree. I think that no matter how good of an optimization system you have, you have to have articulated it well and clearly the actual objective function itself. And to say that we as a collective society or as an individual corporation or something along those lines, could ever come to some kind of clear agreement about what that objective function should be for an AI system is very dubious in my opinion. I think that it\'s essentially\... Such an AI system would have to, in order to be able to do this form of optimization, would essentially have to either be a person, in order to give people what they want, or it would have to be in complete control of people, at which point it\'s not really a CEO anymore, it\'s just a tool that\'s being used by people that are in a system of controlling the system like that. I don\'t think that that would solve the problem. There are lots of instances of corporate structures and governmental structures that are disenfranchising and abusing people all around the world, and it becomes a question of values and what we think these systems should be doing rather than their effectiveness in actually doing what we think they should be doing. And so, yeah, I basically completely agree with the question in saying that we wouldn\'t really get that much out of having an AI CEO. Does that\...

\*\*0:25:50.8 Vael:\*\* Interesting. Yeah, I think in the vision of this where it\'s not just completely dystopian, what you maybe have is an AI that is very frequently checking in on human feedback. And that has been trained very well with humans such that it is\... So there\'s a question of how hard it is to get an AI to be aligned with one person. And then there\'s a question of how hard it is to get an AI to be aligned with a multitude of people, or a conglomerate of people, or how we do democracy or whatever that\'s, yeah, complicated. But even with one person, you still might have trouble, is my intuition here? And just trying to have it\-- still with the access to human feedback, still have human feedback in a way that it\'s fast enough that the AI is still doing approximately what you want.

\*\*0:26:41.7 Interviewee:\*\* Yeah, yeah, I agree. Yeah. I just think that the question of interpretability becomes a very big issue here as well where you really want to know what your system is doing, and you really need to know how it works. And with the way things are currently going we\'re nowhere near that. And so, if we have a large system that we don\'t understand how it works and is operating on limited human feedback and is relatively inscrutable, the list of problems that could result from that is very very long. Yeah. \[chuckle\]

\*\*0:27:15.6 Vael:\*\* Awesome. Yeah, and my next question is about presumably one of those problems. So, say we have our CEO AI, and it\'s capable of multi-step planning and can do people modelling it, and it is trying to\... I\'ve given it its goal, which is to optimize for profit with a bunch of constraints, and it is planning and it\'s noticing that some of its plans are failing because it gets shut down by people. So as a basic mechanism, we have basically\--

\*\*0:27:44.4 Interviewee:\*\* Because it\'s what by people?

\*\*0:27:46.2 Vael:\*\* Its plans are getting\... Or it is getting shut down by people. So this AI has been put\... There\'s a basic safety constraint in this AI, which is that any big plans it does has to be approved by humans, and the humans have asked for a one-page memo. So this AI is sitting there and it\'s like, \"Okay, cool, I need to write this memo. And obviously, I have a ton of information, and I need to condense it into a page that\'s human comprehensible.\" And the AI is like, \"Cool, so I noticed that if I include some information in this memo then the human decides to shut me off, and that would make my ultimate plan of trying to get profit less likely to happen, so why don\'t I leave out some information so that I decrease the likelihood of being shut down and increase the likelihood of achieving the goal that\'s been programmed into me?\" And so, this is a story about an AI that hasn\'t had self-preservation built into it, but it is arising as an instrumental incentive of it being an agent optimizing towards any goal. So what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?\"

\*\*0:28:53.1 Interviewee:\*\* Well, right. It\'s very dependent on the objective function that you select for the system. I think that a system\... It seems, at face value, pretty ridiculous to me that the CEO of a company, the CEO robot, would have its objective function being maximizing profit rather than maximizing individual happiness within the company or within the population on the whole. But even in a circumstance like that, you can imagine very, very, very many pathological circumstances arising. This is the three laws of robotics from Isaac Asimov, right? It\'s just very simplified objective functions produce pathological consequences when scaled to very large complex systems. And so, in much the same way you can train a neural network to recognize an image which produces the unintended consequence that tiny little perturbations of that image can cause it to radically change its output when you have improperly controlled what the system is doing at a large scale, the number of tiny unintended consequences that you could have essentially explodes many-fold. And yeah, I certainly wouldn\'t do this. That\'s certainly not something that I would do, yeah.

\*\*0:30:20.6 Vael:\*\* Yeah. Have you heard of AI Safety?

\*\*0:30:24.3 Interviewee:\*\* AI\... Yeah, yeah.

\*\*0:30:26.0 Vael:\*\* Cool. What does that term mean for you?

\*\*0:30:27.2 Interviewee:\*\* You\'re talking\... What does it mean for me? Well, I guess it\'s closely related to AI ethics. AI safety would mainly be a set of algorithms, or a set of protocols intended to ensure that a AI system is actually doing what it\'s supposed to do and that it behaves safely in a variety of circumstances. Is that correct?

\*\*0:30:52.2 Vael:\*\* Well, I don\'t\-- there\'s not one definition in fact, it seems like it\'s a sprawling field. And then, have you heard of the term AI alignment?

\*\*0:31:00.7 Interviewee:\*\* No, I don\'t know what that is.

\*\*0:31:01.5 Vael:\*\* Cool. This is more long-term focused AI safety. And one of their definitions they use is building models that represent and safely optimize hard-to-specify human values. Alternatively, ensuring that AI behavior aligns with the system designer intentions. Although there are a lot of different definitions of alignment as well. So there\'s a whole bunch of people who are thinking about long-term risks from AI, so as AI gets more and more powerful. I think the example we just talked about, like the ones where adversarial examples can really change the output of a system very easily, is a little bit different than the argument made here, which is something like: if you have an agent that\'s optimizing for a goal and it\'s good enough at planning then it\'s going to be instrumentally incentivized to acquire resources and power and not be shut down and kind of optimize against you, which is a problem when you have an AI that is similarly as smart as humans. And I think in that circumstance, one of the arguments is that this constitutes an existential risk, like having a system that\'s smarter than you constituting against you would be quite bad. What do you think of that?

\*\*0:32:04.1 Interviewee:\*\* Yeah, I was only using the adversarial example to give an example of how easily and frequently this does happen at even the level that we\'re currently working at. I think it would be much, much, much worse at the level of the general artificial intelligence that would have essentially long-term dynamic interactions with people, rather than a system that\'s just taking an image and outputting a response. When the consequences of such a system can have long term effects on the health and well-being of people, this kind of thing becomes very different and much more important.

\*\*0:32:43.4 Vael:\*\* Yeah. And like with the problem I was outlining earlier, which is like, how do we get to do exactly what they intended to do? The idea that you have of like trying\... Like why would you create a system that wasn\'t optimizing for all of human values? I was like, wow, ahead of the game there. That is in some sense the goal. So there is a community who\'s working on AI alignment kind of research, there\'s money in this community. It\'s fairly new\-- although much more popular, or like, AI safety haw grown a lot more over the years. What would cause you to work on trying to prevent long-term risks from AI systems?

\*\*0:33:18.5 Interviewee:\*\* What would cause me to do work on it?

\*\*0:33:20.6 Vael:\*\* Yeah.

\*\*0:33:29.6 Interviewee:\*\* To be honest, I think that it would have to be\... I guess I would really have to be convinced that the state of the field in the next few years is tending towards some type of existential risk. I feel like\... You don\'t have to convince me too much, but I personally don\'t think that the field of study that I\'m currently occupying is one that\'s really contributing to this problem. And so I would become much more concerned if I felt like the work that I was doing was actively contributing to this problem, or if there was huge evidence of the near advent of these types of generally intelligent systems to be terribly worried about.

\*\*0:34:28.6 Vael:\*\* Yeah. That makes sense. Yeah, I don\'t actually expect computational neuroscience to be largely contributing to this in any way. I feel like the companies that are gonna be doing this are the ones who are aiming for AGI. I do expect them to kind of continue going that way, regardless of what is happening. And I expect the danger to happen not immediately, not in the next couple of years. Certainly people have like different ranges, but like 2060 is like an estimate on some paper I believe that I can send along. It probably won\'t be a while, won\'t be for a while.

\*\*0:35:00.5 Interviewee:\*\* Sure. I don\'t know, I think that people who understand these algorithms in the way that they work do have in some sense a duty to stand up to these types of problems if they present themselves. And there are many instances of softer forms of AI being used for horrible things currently, which I certainly could be doing more in my daily life to prevent. But for now, I don\'t know. I guess I just have, I have my own interests and priorities. And so it\'s kind of a\... It\'s something to get to eventually.

\*\*0:35:42.9 Vael:\*\* Yeah, yeah. For sure. I think these technical AI safety is important. And am I working in technical AI safety? Nope. So like we all do the things that we want to do.

\*\*0:35:54.8 Interviewee:\*\* Yeah.

\*\*0:35:54.9 Vael:\*\* Great, cool. So that was my last question, my downer of an interview here \[chuckle\], but how do you think\...

\*\*0:36:02.3 Interviewee:\*\* No, no.

\*\*0:36:04.1 Vael:\*\* But yeah. Okay. So my actual last question is, have you changed your mind in anything during this interview and how was this interview for you?

\*\*0:36:08.9 Interviewee:\*\* No, it was a good interview. I don\'t think I\'ve particularly changed my mind about anything. I think that it was good to work through some of these questions and yeah, I had a good time.

\*\*0:36:24.2 Vael:\*\* Amazing. Yeah, why\--

\*\*0:36:25.3 Interviewee:\*\* I typically don\'t expect it to change my mind too much in interviews, so \[chuckle\].

\*\*0:36:28.8 Vael:\*\* Absolutely. Yeah, yeah, yeah. Okay. Why do\... People tell me they have a good time and I\'m like, are you lying? Did you really have\... Why is this a good time?

\*\*0:36:37.2 Interviewee:\*\* No, it\'s nice to talk about your work. It\'s nice to talk about long-term impacts that you don\'t talk about in your daily basis. I don\'t know. I don\'t need to be paid to do something like this for instance.

\*\*0:36:51.7 Vael:\*\* All right. Well, thank you so much. Yeah. If you think of any questions for me, I\'m here for a bit. I\'m also happy to send any resources if you\'re curious about, like, my takes on things, but yeah, generally just very appreciate this.

\*\*0:37:04.4 Interviewee:\*\* Yeah, sure. I\'m a little curious about what this interview is for. Is it for just you, or is it, like a\... You mentioned something about some type of AI alignment group or is there some kind of\... I\'m just curious about what it\'s for.

\*\*0:37:20.9 Vael:\*\* Yeah. So I am interested\... I\'m part of the AI alignment community, per se, although I\'m not doing direct work. The people there often work on technical solutions to try to\... to the alignment problem, which is just trying to come up with good ways of making sure that AIs in the future will be responsive, do what humans want. And examples of that include trying to build in feedback, human feedback, in a way that is scalable with current systems and works with uninterpretable systems, and interpretability\-- certain types of interpretability work. There\'s teams like DeepMind Safety, OpenAI Safety, different, like, separate alignment community. So I\'m like in that space. And I\'ve been doing interviews with AI researchers to see what they think about the safety arguments. And whether\... instrumental incentives. And just like, when do you think we\'ll get AGI, if you think we will. Get a lot of different opinions, a lot of different ways.

\[\...\]

\*\*0:38:47.5 Interviewee:\*\* Cool. Anyway, that makes a lot of sense and, yeah, I hope that things go well. Thanks for having me. Yeah.

\*\*0:38:55.5 Vael:\*\* Yeah. Thanks so much, really appreciate it. Alright, bye.

\*\*0:38:59.1 Interviewee:\*\* Bye, see you.