individuallyselected_w5cb5-by Vael Gates-date 20220318

# Interview with AI Researchers individuallyselected\_w5cb5 by Vael Gates

\*\*Interview with w5cb5, on 3/18/22\*\*
====================================

\*\*0:00:02.2 Vael:\*\* Alright, so my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:08.5 Interviewee:\*\* I worked in \[subfield\] originally, but I guess I branched out more broadly into AI research, because I\'m \[high-level research role\] now at an AI company.

\*\*0:00:19.9 Vael:\*\* Great, yeah. And then what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:00:28.3 Interviewee:\*\* So I think, yeah, the world is going to change quite a lot with AI technology, and I think mostly in good ways, just because we\'re going to empower people with this technology. And it\'s going to be empowering I think in similar ways to the Internet, where people can do faster search, they have an assistant who can help them with all kinds of stuff. They have friends who maybe are not real, and all kinds of ways to make people happier, I think, or more efficient, or to give them time back and that sort of stuff. But obviously, there are also risks and the main risks are, I think that the field is too dominated by tech bros from Silicon Valley, so I guess I fall under that in a way. And so I think that\'s a real problem, so we need to take democratization of the technology much more seriously, that\'s also what my company is doing. And I think if we think about the ethical implications of our technology from first principles, and if we make them first-class citizens rather than just treating them as an afterthought, where you submit your paper and then, \"Oh, I also need to write a broader impact statement,\" but if you take that very seriously from the beginning as a core principle of your organization, then I think you can do much better research in a much more responsible way.

\*\*0:01:56.5 Vael:\*\* Interesting. Alright, so that was the question of \"what are you most excited about and what are you most worried about in AI\", okay. I heard\-- Lots of things they can go, lots of places they can go, lots of directions they can go, but you\'re worried about domination from specific areas and then people not caring about\... ethics enough? or---

\*\*0:02:14.6 Interviewee:\*\* Yeah, so misuse of technology. Do you want me to give you concrete examples? So I think very often, the technology that we develop, even if it\'s meant for benevolent purposes, can also be re-applied for not so benevolent purposes. And so like speech recognition or face recognition, things like that, you have to just be very careful with how you treat this technology. So that\'s why I think if people take responsible AI seriously from the beginning, that that is a good thing too.

\*\*0:02:53.0 Vael:\*\* Interesting. So you think if people incorporate responsible AI from the beginning of the process, then there will be less risk of misuse by any agent in the future?

\*\*0:03:04.5 Interviewee:\*\* Yeah, yeah. So you mentioned your IRB, so for a lot of technological research happening in industry, there is no real IRB. Some companies have sort of IRBs but most of them are so commercial and so driven by money in the end. And I think maybe we need an independent AI IRB for the broader research community, where anybody can go there and have somebody look at the potential applications of their work.

\*\*0:03:39.6 Vael:\*\* I see, cool. And then just having that sort of mindset seems good, in addition to the object-level effects. Alright. Makes sense. So focusing on future AI, putting on a science fiction forecasting hat, say we are 50 years, 50 plus years into the future. So at least 50 years into the future, what does that future look like?

\*\*0:04:00.3 Interviewee:\*\* At least 50 years in the future. So I still don\'t think we will have AGI, and that\'s I guess, I\'m probably unusual in the field because I think a lot of my colleagues would disagree, especially if they\'re at OpenAI or DeepMind because they think that it\'s like two years away. (Vael: \"Two years, huh!\") Yeah, well it depends on who you ask, they have some crazy people. \[chuckle\] I think in the next decade, we\'re going to realize what the limitations are of our current technology. I think what we\'ve been doing now has been very efficient in terms of scaling with data and scaling with compute, but it\'s very likely that we\'re just going to need entirely new algorithms that just require pure scientific breakthroughs. And so I don\'t think there\'s going to be another AI winter, but I do think that things are going to cool down a little bit again, because right now it\'s just been super hyped up. For good reason too, because we are really making really great progress. But there is still things that we really don\'t know how to do, so we have language models and they can do things and they\'re amazing, but we don\'t know how to make the language model do what we want it to do. So we\'re all just sort of hacking it a little bit, but it\'s not really anywhere close to being like a proper assistant, for example, who actually understands what you\'re saying, who actually understands the world. I think where we want to be 50 years from now is where we have machines who understand the world in the same way that humans understand it, so maybe something like Neuralink. So if I\'m being very futuristic, connecting AI to human brains and human perception of reality, that could be a way to get AI to have a much richer understanding of the world in the same way that humans understand it. So like dolphins are also very intelligent, but they also don\'t understand humans and they are not very useful assistants, right? I don\'t know if you\'ve ever had any dolphin assistant. So it\'s not really bad intelligence, it\'s specifically about human intelligence that makes AI potentially useful for us, and so that\'s something that I think is often overlooked.

\*\*0:06:26.9 Vael:\*\* So it sounds like, so you\'re thinking about when AGI will happen. And you said that you don\'t think we\'re gonna hit some sort of ceiling or slow down on the current deep learning paradigm or just like keep on scaling\--

\*\*0:06:39.6 Interviewee:\*\* Yeah, it\'s going to be asymptotic, and at some point, we\'re just going to hit the limits of what we can do with scaling data and scaling compute. And in order to get the next leap to real AGI I think we just need radically different ideas.

\*\*0:06:55.1 Vael:\*\* Yeah, when do you think we\'re going to\-- what kind of systems do you think we\'re going to have when we cap out on the current scaling paradigm?

\*\*0:07:02.0 Interviewee:\*\* Well, I think like the ones we have now, but yeah, in 50 years, I don\'t know. But in like 5 to 10 years, it will just be much bigger versions of this. And so what we have seen is that if you scale these systems, they generalize much better. If that keeps happening, then we would just have much better versions of what we have now. But still it\'s a language model that doesn\'t understand the world, and so still it\'s the component that is very limited in seeing only the training data that is in images on the internet, which is not all of the images that we have in the world, right? So I think the real problem is data, not so much scaling the compute.

\*\*0:07:49.7 Vael:\*\* What if we had a system that has cameras and can process auditory stuff that is happening all around it or something and it\'s not just using internet data, do you think that would eventually have enough data?

\*\*0:08:03.3 Interviewee:\*\* Yeah, so that\'s what I was just saying. If you have something that\'s embodied in the world in the same way as a human and where humans treat it as another human, sort of like cyborg style, things like that, that\'s a good way to get lots of very high quality data in the same way that humans get it. What are they called? Androids, right?

\*\*0:08:24.9 Vael:\*\* Yeah.

\*\*0:08:25.3 Interviewee:\*\* So if we actually had android robots walking around and being raised by humans and then we figured out how the learning algorithms would work in those settings, then you would get something that is very close to human intelligence. A good example I always like to use is the smell of coffee. So I know that you know what coffee smells like, but can you describe it to me in one sentence?

\*\*0:08:54.2 Vael:\*\* Probably not, no.

\*\*0:08:55.7 Interviewee:\*\* You can\'t, right? But the same goes for the taste of banana or things like that. I know that you know, so I\'ve never had to express this in words. So this is one of the fundamental parts of your brain; smell and taste are even older than sight and hearing. And so there\'s a lot of stuff happening in your brain that is just taken for granted. You can call this common sense or whatever you want, but it\'s like an evolutionary prior that all humans share with each other, and so that prior governs a lot of our behavior and a lot of our communication. So if you want machines to learn language but they don\'t have that prior, it becomes really, really hard for them to really understand what we\'re saying, right?

\*\*0:09:38.7 Vael:\*\* Yeah. I think when I think about AGI, I think about AGI that can do\-- or, just, generalizable systems that can do things that humans want them to do. So imagine we have like a CEO AI or a scientist AI. I don\'t think I need my CEO or scientist AI enough to know what coffee smells like per se, but I do need it to be able to like break down experiments and think kind of creative thoughts and figure out things.

\*\*0:09:58.7 Interviewee:\*\* Yeah, but I think what I\'m saying is that if they don\'t know what coffee smells like, that\'s just one example, but there are millions of these things that are just things we take for granted, that we don\'t really talk about. And so this will not be born out in the data in any way, so that means that a lot of the underlying assumptions are never really in the data, right? They\'re in our behavior, and so for an AI to pick up on those is going to be very difficult.

\*\*0:10:27.6 Vael:\*\* What if there were cameras everywhere, and it got to record everyone and process those?

\*\*0:10:32.3 Interviewee:\*\* Yeah, maybe. So the real question is, if you just throw infinite data at it, then will it work with current machine learning algorithms? Is I guess what you\'re asking, right? And so I don\'t know. I mean, I know that our learning algorithm is very different from a neural net, but I think if you look at it from a mathematical perspective, then gradient descent is probably more efficient than Hebbian learning anyway. So mathematically, it\'s definitely possible that if you have infinite data and infinite compute, then you can get something really amazing. Sure, we are the proof of that, right? So whether that also immediately makes it useful for us is a different question, I think.

\*\*0:11:20.8 Vael:\*\* Interesting. Yeah, I think I\'m trying to probe \"do we need something like embodied AI in order to get AGI\" or something. And then your last comment was like, whether that makes it useful for us. I\'m like, well, presumably we\'re going to\... feeding it a lot of data lets it do grounding, so like relationships between language and what actually exists in the world and how physics works. But presumably, we\'re going to be training them to do what we want, right? So that it will be useful to us?

\*\*0:11:43.5 Interviewee:\*\* Well, it depends, right? Can we do that? Probably the way they will learn this stuff is through self-supervised learning, not through us supervising them. We don\'t know how to specify reward signals and things like that anyway. I\'m not sure, if we actually are able to train up these huge systems that are actually intelligent through self-supervised learning, if they are then going to listen to us, right? Why would they?

\*\*0:12:15.2 Vael:\*\* Right. Okay, cool. Yeah, so this kind of leads right into my next question here. So imagine we\'re in the future and we\'ve got some AGIs and we\'ve got a CEO AI, and I\'m like, \"Okay, CEO AI, I want you to maximize profits and not run out of money and not try to exploit people and try to avoid side effects,\" and it seems like this would currently be extremely challenging for many reasons. But one is that we\'re not very good at taking human values and putting them\-- and like goals and preferences\-- and putting them in mathematical formulations that AI can currently work. And I worry that this is gonna happen in the future as well. So the question is: what do you think of the argument, \"Highly intelligence systems will fail to optimize exactly what their designers intended them to and this is dangerous\"?

\*\*0:12:53 Interviewee:\*\* Well, yeah. I agree with that. I don\'t think\... I think there are two separate questions here. So one you\'re asking about is the paperclip maximizer argument from Nick Bostrom. So like if you have a system and you tell it like \"you need to make as many paperclips as you possibly can\" then it\'s going to like destroy the earth to make as many paperclips as possible.

\*\*0:13:15 Vael:\*\* Well that would be doing maybe\-- oh, I see. Not quite what I intended. Yeah, all right.

\*\*0:13:19.8 Interviewee:\*\* Yeah, so\-- okay, so if that\'s not what the underlying question was, then\... We don\'t really\... I also think that we are\... some of us are fooling ourselves into believing that we know everything as humans and I think human values are changing all the time. I don\'t think we can capture correct human values. I don\'t think there is an absolute moral truth that we should all adhere to. I think that just morality itself is a very cultural concept. But I\'m \[interested in\] philosophy, so I\'m a bit different from most AI researchers, I guess. So I think that we could try to encode some very basic principles, so this is like Asimov\'s laws and things like that, but I don\'t think we can really go much further than that. And I think even in those cases, like you said, we don\'t know how to mathematically encode them in a way where you enforce whatever this dynamical system is that you\'re training, so a neural net, but then probably more complicated than the current neural nets\-- how do we impose a particular set of values? I don\'t think we know how to do that. I don\'t think there\'s a mathematical way to do that either actually, because it\'s all \[inaudible\]\--

\*\*0:14:44.7 Vael:\*\* Yeah, do you think we are eventually going to be able to?

\*\*0:14:50.0 Interviewee:\*\* So I think if you ask Yann LeCun or someone like that, he would say that probably, if we ever get to systems of this sort of level of intelligence, then they would be benevolent, because they\'re very smart and able to sort of understand how weak humans are.

\*\*0:15:09.4 Vael:\*\* Interesting. Yeah. So when I hear that argument, I\'m like, okay, it seems like Yann LeCun thinks that as you get more intelligent, you have morals that are very similar to humans, and this just kind of comes\--

\*\*0:15:21.7 Interviewee:\*\* No, not necessarily. No, but just better morals, right? So I think that the argument is sort of that if you look at human progress, then we\'ve also been getting better and better moral systems and a better understanding of what human values really matter. And like 100 years from now, probably everybody\'s gonna look back at us and say, \"They were eating meat. They were killing all these animals.\" So we are on the path of enlightenment. I don\'t know if I agree with this, but that\'s one way of saying it. And so a sign of an organism or a culture becoming more and more enlightened is also that you become more and more benevolent I think for others, but maybe that\'s a bit of a naive take.

\*\*0:16:05.9 Vael:\*\* Yeah. I think in my mind\-- certainly we have\-- well, actually, I don\'t know that we have the correlation that humans are getting smarter and also at the same rate, or, like\... Like humans are pretty smart. And we\'re getting better at IQ tests, but I don\'t know that we\'re vastly increasing our intelligence per se.

\*\*0:16:20.4 Interviewee:\*\* Yeah. That\'s for different reasons, right. Yeah.

\*\*0:16:24.9 Vael:\*\* Yeah. And meanwhile, we have, over\-- centuries, like not that many centuries, we\'ve been increasing our moral circle and putting in animals and people far away from us, etcetera. But I kind of think of the axes of intelligence and morality as kind of orthogonal, where if we have a system that is getting much smarter, I don\'t expect it to have\... I expect kind of a lot of human morality runs from evolutionary pressures and also coordination difficulties, such that you need to be able to not kill people, otherwise the species is gonna go extinct. And you know, there\'s a bunch of stuff that are kind of built into humans that I wouldn\'t expect to happen just natively with intelligence; where intelligence, I would think of something like\... the ability to solve problems well, to make multi-step plans, to think in the future, to take out correlations and figure out predictions, and I don\'t expect that to naively correlate with---

\*\*0:17:19.9 Interviewee:\*\* Yeah, so I think that\'s a very narrow definition of intelligence, and so I don\'t know if that definition of intelligence you have, if that actually is the most useful kind of intelligence for humans. So I think that in our society there is this concept where intelligence just means like mathematical reasoning capabilities almost, right? (Vael: \"Yeah.\") And that is a very, very narrow definition, and most of our intelligence is not that, right? (Vael: \"Yes.\") So for regimes to be useful to us\... so I think what you\'re talking about is sort of like this good old-fashioned AI concept of intelligence, where you have symbolic reasoners, and you\'re like\... you\'re very good at very fast symbol manipulation. And like, \"This is what computers are for.\" So we should just have super smart computers who can do the stuff that we don\'t want to do or can\'t do. It\'s possible that our intelligence is a direct consequence, not of our mathematical reasoning capabilities, but of something else, of our cultural interactions. So I definitely think if humans were not a multi-agent society, that we would not be nearly as intelligent. So a lot of our intelligence comes from sharing knowledge and communicating knowledge and having to abstract knowledge so that you can convey it to other agents and that sort of stuff.

\*\*0:18:50.0 Vael:\*\* Cool. Yeah. So when I think about how I define intelligence, I\'m like, \"What is the thing I care about?\" The thing I care about is how we develop AI. And I\'m like, \"How are we gonna develop AI?\" We\'re gonna develop it so that it completes economic incentives. So we want robots that do tasks that humans don\'t want to do. We want computers\--

\*\*0:19:09.2 Interviewee:\*\* Yeah. But is that AI or is that just machine learning? We\'re trying to have a\... like input-output black box, and we want that black box to be as optimal as possible for making money or whatever the goal is, right? So that\'s also a worry I have, is that a lot of people are conflating these different concepts. So artificial intelligence\...yeah, it depends on how you define it. Some people think of it more as like AGI. If you ask Yann again and all the old school deep learners, they would say, it used to be that they were explicitly not doing AI. So AI is like Simon and Newell and all that sort of stuff, so like pure symbol manipulation, symbolic AI. And pattern recognition is not AI. And now, since deep learning became very popular, some of the people were like, \"Oh yeah, this is AI now,\" but they used to be machine learning and not AI. So one thing is just like this black box. It can be anything and we just want to have the best possible black box for our particular problem mapping X to Y. And this could be any kind of problem, it could be like image recognition or whatever. In some cases, you want to have a symbolic approach, in other cases, you want to have a learning approach, it sort of just depends. So it\'s just software. Right? But in one case, the software is well defined, and in the other case, it\'s a bit fuzzier.

\*\*0:20:37.9 Vael:\*\* Yeah. So this all kind of depends on your frame, of course. I think my frame, or the reason why I care, is I\'m like, I think machine learning, AI, I don\'t know, whatever this thing is where humans are pouring a lot of investment and effort into making software better, and by better I mean better able to accomplish tasks that we want it to do\-- I think that this will be\-- it is very powerful, it has affected society a lot already and it will continue to affect society a lot. Such that like 50 years out, I expect this to be\... Whatever we developed to be very important in how\... Affect just a lot of things.

\*\*0:21:10.8 Interviewee:\*\* But we\'re notoriously bad at predicting the future, right? So if you asked in the \'60s, people would say like, there\'s flying cars, and like we\'re living on Mars and all that stuff. And we\'re getting a bit closer, but we\'re still not there yet. But none of these people would have seen the internet coming. And so I think maybe the next version of the internet is going to be more AI driven. So that is a sort of\... first use case that I would see for AI, which is like a better internet.

\*\*0:21:50.0 Vael:\*\* Interesting. Yeah, I think kind of\... people will find whatever economic niches will get them a lot of profit, is sort of how I expect things to continue to go, given that that seems to be \... Given that society works kind of the same way, and people have a lot of time and energy and have the capability to invest in this stuff, we will continue to develop machine learning, AI software, etcetera, such that it\--

\*\*0:22:13.2 Interviewee:\*\* We\'ve been doing that for like 30 years or even more. From the Perceptron, Rosenblatt. We\'ve been already doing this and so it\'s not really a question of like AI taking over the world, it\'s software taking over the world, and AI in some cases is better than like rule-based software. But it\'s still software taking over the world.

\*\*0:22:35.8 Vael:\*\* Yeah, yeah, certainly. And then the current paradigm of like, gigantic neural nets, seems to be better at doing things that we want it to do. And so we\'re continuing on in that direction, and at some point, as you say, it becomes less able to do what we want it to do, given the amount of resources that we\'re pouring into it, like that ratio trades off. Okay\--

\*\*0:22:54.3 Interviewee:\*\* Yeah. So there\'s other trade offs too, right? So as you become bigger as a neural net, you also become a lot more inefficient. This is already the case for something like GPT-3; latency is a big problem. For us to be able to talk like this to a machine, if the machine has 100 trillion parameters, it\'s going to be way too slow. It\'s going to take, I don\'t know, 10 minutes to generate an answer to a simple question. So it\'s not only a tradeoff of\... Best does not just mean accuracy. Best also is like, how efficient are you? How fair are you? How robust are you? How much environmental impact do you have? All of these different sort of metrics that all matter for choosing what defines \"best\" for a system. I think this is something we need to improve a lot on as a community, where we stop thinking beyond this pure accuracy thing, which is like an academic concept, to an actual\... like how can we deploy these systems in a responsible way, where we think about all the possible metrics that matter for deployment. So we want to be at the Pareto frontier of like 10 different metrics, not just accuracy.

\*\*0:24:06.8 Vael:\*\* Cool. Alright, that makes sense. So still thinking ahead in the future, do you think we\'ll ever get something like a CEO AI?

\*\*0:24:14.0 Interviewee:\*\* So, if\-- so a CEO AGI or a CEO AI?

\*\*0:24:18.8 Vael:\*\* Um, some sort of software system that can do the things that a CEO can do.

\*\*0:24:25.6 Interviewee:\*\* No.

\*\*0:24:26.1 Vael:\*\* No. Okay.

\*\*0:24:28.6 Interviewee:\*\* So not before we get AGI. So I think that is an AI complete problem. But I do think we\'ll get a very good CEO AI assistant. \[inaudible\] \...real human. It\'s like a plane, right? So like a plane is flown by a pilot but it\'s really flown by a computer. So I think the same could be true for a company where the company has like, a CEO pilot whose job is also to inspire people and do all of the human soft skills. And they have an assistant who does a lot of measurement stuff and tries to give advice for like where the company should be headed and things like that.

\*\*0:25:05.1 Vael:\*\* Okay, awesome. And you do think that you could have a CEO AGI, it sounds like.

\*\*0:25:10.3 Interviewee:\*\* Yeah, but if you have an AGI, then we don\'t need CEOs anymore.

\*\*0:25:14.3 Vael:\*\* What happens when we get AGI?

\*\*0:25:16.9 Interviewee:\*\* All the humans die.

\*\*0:25:17.5 Vael:\*\* All the humans die. Okay! \[laughs\]

\*\*0:25:20.1 Interviewee:\*\* \[laughs\] So I think it depends. I think actually the most likely scenario, as I said, for AGI to come into existence is when humans merge with AI. And so I don\'t think that it\'s a bad thing for AGI to emerge. So if there is an AGI, then it will be a beautiful thing, and we will have made it as a society. So yeah, if that thing takes over, then that thing is going to be insane, it\'s going to take over the universe, and then we will be sort of like the cute little people who made it happen. So either we become very redundant very quickly or we sort of merge with AI into this new species kind of.

\*\*0:26:14.1 Vael:\*\* Interesting, okay. And you don\'t necessarily see a connection between, like, the current\... \[you think\] if we just push really hard on the current machine learning paradigm for 50 years, we won\'t have an AGI. We need to do something different for an AGI, which sounds like embodiment / combination with humans, biological merging?

\*\*0:26:31.7 Interviewee:\*\* So it could be embodiment and combination with humans, but also just better, different learning algorithms. So probably more sparsity is something that scales better. More efficient learning. So the problem with gradient descent is that you need too much data for it. Maybe we need some like Bayesian things where we can very quickly update belief systems. But maybe that needs to happen at a symbolic level. I still think we have to fix symbolic processing happening on neural networks\-- so we\'re still very good at pattern recognition, and I think one of the things you see with things like GPT-3 is that humans are amazing at anthropomorphizing anything. I don\'t know if you\'ve ever read any Daniel Dennett, but what we do is we take an intentional stance towards things, and so we are ascribing intentionality even to inanimate objects. His theory is essentially that consciousness comes from that. So we are taking an intentional stance towards ourselves and thinking of ourselves as a rational agent and that loop is what consciousness is. But actually we\'re sort of biological machines who perceive their own actions and over time this became what we consider consciousness. So\... where was I going with this? \[laughs\] What was the question?

\*\*0:27:57.2 Vael:\*\* Yeah, okay. So I\'m like, alright, we\'ve got AI, we\'ve got lots of machine learning\--

\*\*0:28:00.8 Interviewee:\*\* \--oh yeah, so do you need new learning algorithms? Yeah. So I think what we need to solve is the sort of System 2, higher-level thinking and how to implement that on the neural net. The neural symbolic divide is still very much an open problem. There are lots of problems we need to solve, where I really don\'t think we can just easily solve them by scaling. And that\'s\-- like there is very little other research happening actually in field right now.

\*\*0:28:35.3 Vael:\*\* Alright. So say we do scaling, but we also have a bunch of software. Like algorithmic improvements at the rate we\'re seeing, and we\'ve got hardware improvements as well. I guess this is just more scaling, but we have optical, we have quantum computing. And then we have some sort of fast learning systems, we know how to do symbolic processing, we\'re much more efficient. Here we now have a system that generalizes very well and is pretty efficient, and I don\'t know, maybe we\'re hundred years out. Say maybe we\'re in a different paradigm, maybe we\'re kind of in the same paradigm. We now have a system that is\--

\*\*0:29:05.5 Interviewee:\*\* We would be in a different paradigm for sure.

\*\*0:29:07.4 Vael:\*\* Okay. We are in a different paradigm, because\... because all these learning algorithms\--?

\*\*0:29:11.4 Interviewee:\*\* Paradigms don\'t really last that long, if you look at the history of science.

\*\*0:29:16.2 Vael:\*\* Okay, cool. But are we still operating under like, here\'s software with faster learning algorithms, more efficient learning algorithms, like symbolic reasoning, Bayesian stuff\--

\*\*0:29:24.7 Interviewee:\*\* Maybe. But I mean it could be that neuromorphic hardware finally lives up to its promise, or that we can do photonic chips at the speed of light computation and things like that. We\'re also very good in AI at fooling ourselves into thinking that we are responsible for all of these amazing breakthroughs, but without hardware engineers at NVIDIA, none of this stuff would have happened, right? They are doing very different things.

\*\*0:29:55.1 Vael:\*\* Alright, so we\'ve got this AI system which is quite general, we\'re in maybe a different paradigm, but we\'re still like\-- faster learning systems. Here we are, these things are very capable, very general, when they generate stories, they model physics in the world and then use that to generate their stories. Maybe they can do a lot of social stuff, maybe they know how to interact with people. And here we are with our system. Is this now an AGI?

\*\*0:30:18.0 Interviewee:\*\* No, no, so\-- Okay, now I remember what I was gonna say about the Dennett thing. So we anthropomorphize everything, we take this intentional stance at everything. We do this to ourselves, we do this to everything, especially when it speaks language. So when we see a language model and it\'s like, \"whoa, it\'s amazing, it does this thing,\" but all it\'s really doing is negative log likelihood, maximum likelihood estimation. It\'s basically just trying to fit \"what is the most likely word to go here\". So you can ask yourself whether we are so impressed by this system because it\'s so amazing, or because we are sort of programmed to have a lot of respect for things that speak language, because things that speak language tend to be humans. What you were just saying made it sound like you were saying, when these systems are sort of like humans, when they can do this and when they do that, and when they understand the world. So how do you define \"understanding the world\" there\--

\*\*0:31:18.7 Vael:\*\* I mostly mean like they could sub in for human jobs, for example\--

\*\*0:31:25.0 Interviewee:\*\* Yeah, but that\'s not the same thing as\-- stepping in for a human, they can already do that. But it depends on the problem. They\'re very good at counting, but\--

\*\*0:31:34.5 Vael:\*\* Yeah, but I don\'t think we could have like a mathematician AI right now per se. I guess I forgot to define my interpretation of AGI, but like a system that is very capable of replacing all current human day jobs.

\*\*0:31:51.6 Interviewee:\*\* Including yours and mine?

\*\*0:31:55.9 Interviewee:\*\* Yup.

\*\*0:31:57.8 Interviewee:\*\* Okay. But then who would it be useful for? Would the president still have a job or not?

\*\*0:32:09.7 Vael:\*\* Uh\... It doesn\'t have to. I think you could just spend\-- humans wouldn\'t have to work anymore, for example, and they could just go around doing whatever they do.

\*\*0:32:16.7 Interviewee:\*\* Yeah. But that\'s not at all what humans do. We\'re all so programmed to compete with each other.

\*\*0:32:24.7 Vael:\*\* Yeah, we can have games, we can have competitions, we can do all sorts of things, we have sports.

\*\*0:32:29.1 Interviewee:\*\* I think it\'s gonna be very quickly my AI versus your AI, basically.

\*\*0:32:33.9 Vael:\*\* Okay, we can have big fights with AIs, that seems very dangerous.

\*\*0:32:37.3 Interviewee:\*\* Yeah, I know, yeah. So that is a more likely scenario, I think, than everybody being nice and friendly and playing games. (Vael: \"Yeah.\") If people want to have power, and whoever controls the AGI will have the most power, (Vael: \"That seems right,\") then I think we\'re going to be developing your own AGIs at the same time. And then those AGIs at some point are going to be fighting with each other.

\*\*0:33:02.0 Vael:\*\* Yeah, yeah, I think we might even get problems before that, where we\'re not able to get AIs aligned with us. Have you heard of AI alignment?

\*\*0:33:10.9 Interviewee:\*\* Yeah, so \[close professional relationship\] wrote a nice thesis about it. \[Name\], I don\'t know if you know \[them\] by any chance. So yeah, alignment is important, but my concern with all this alignment stuff is that it\'s very ill-defined, I think. Either it means the same as correctness, so is your system just correct, or good at what you want it to be good at\... alignment is sort of like a reinvention of just correctness. I can see why this is useful for some people to put a new name on it. But I think it\'s a very old concept where it\'s just, okay, we\'re measuring things on a very narrow static test set, but we should be thinking about all these other things. You want your system to be really good when you deploy it in the real world. So it needs to be a good system or a correct or an aligned system. And so alignment maybe is a useful concept, only in the sense that the systems are getting so good now that you can start thinking about different kinds of goodness that we didn\'t think about before, and we can call that alignment, like human value-style things. But I think the concept itself is very old; it\'s just like, is your system correct?

\*\*0:34:40.0 Vael:\*\* Yeah. And then it\'s nowadays being thought about in terms of very far future systems and aligning with all values and preferences. (Interviewee: Yeah.) Cool. Yeah, do you work on any sort of AI safety or what would convince you to work on this or not work on this, etcetera?

\*\*0:34:56.5 Interviewee:\*\* Yeah so, I\'m not sure. AI safety is a bit of a weird concept to me, but I do work on the responsible AI and ethical AI, yeah.

\*\*0:35:06.9 Vael:\*\* Hm. And what does that mean\--

\*\*0:35:09.1 Interviewee:\*\* So these are things like\... I\'m trying to get better fairness metrics for systems. So in \[company\] we built this provisional fairness metric where we do some heuristic swaps. And so right now we\'re working on a more sophisticated method for doing this where, let\'s say, you have something, a sentence or some sort of natural language inference example, so a premise and a hypothesis and it\'s about James, like if you change James to Jamal, that shouldn\'t change your prediction at all. Or if you change the gender from James and you turn into a woman, that shouldn\'t change anything there. And it does, actually, if you look at restaurant reviews, if you changed the restaurant to a Mexican restaurant and the person who\'s eating there to Jamal, then your sentiment goes down. So this is the sort of stuff that shouldn\'t happen in these systems that is direct consequence of us just scaling the hell out of our systems on as much data as we can, including all of the biases that exist in this data. So I\'m working on trying to do that better measurement for these sort of things. And so I think if we are not getting better at measurement, then all of this stuff is basically a pointless discussion.

\*\*0:36:29.1 Vael:\*\* Great, thank you. And then my last question is, have you changed your mind on anything during this interview and how was this interview for you?

\*\*0:36:35.9 Interviewee:\*\* It was fun. Yeah, I\'ve done a few of these with various people and it\'s always a bit like, I don\'t know. It feels a bit like\... we\'re getting ahead of ourselves a little bit. But maybe I\'m also just old. So when I talked to \[close professional relationship\] and how \[they\] think about stuff, I\'m like, I just don\'t understand how \[they\] think about AI.

\*\*0:37:06.2 Vael:\*\* Got it. \[They\'re\] like way out here, and we need to make sure that systems do our correct\--

\*\*0:37:11.9 Interviewee:\*\* Yeah, \[they\'re\] really.. Yeah, \[they\] put a lot more faith also in AI, which I think is very interesting. So I asked \[them\] like, \"Okay, so this alignment stuff, in the end who should we ask what is right or what is wrong? When we\'re trying to design the best AI systems, who should we ask for what\'s right and wrong?\" And then \[their\] answer was, \"We should ask the AI.\"

\*\*0:37:38.7 Vael:\*\* What? No, we should ask humans.

\*\*0:37:41.0 Interviewee:\*\* Yeah, no, so \[they\] think that basically AGI or AI is going to get so good, these language models are gonna get so good that they can tell us how we should think about our own moral philosophical values so that we can impose them onto AI systems. That to me just sounds crazy, like batshit crazy, but that\'s one way to think about it. I mean, I respect \[their\] opinion. I just can\'t understand it.

\*\*0:38:11.7 Vael:\*\* Interesting. Yeah, I think if I try to model what I would imagine \[they\] would be saying, under the alignment paradigm, I would say that you need to ask human feedback, but it\'s hard to get human feedback on very intelligent systems. And so you should ask AI to summarize human feedback, but it should always be ground down on a human otherwise we\'re in trouble, so.

\[ending comments\]


individuallyselected_zlzai-by Vael Gates-date 20220318

# Interview with AI Researchers individuallyselected\_zlzai by Vael Gates

\*\*Interview with zlzai, on 3/18/22\*\*

\[Note: Th interviewee\'s Zoom connection was very bad since they were calling in from an Uber, so there\'s a lot of missing content (and \"I lost you\...\" parts of the conversation that have been removed for clarity).\]

\*\*0:00:02.5 Vael:\*\* Cool. All right. So my first question is: can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:09.2 Interviewee:\*\* Definitely. The way that I describe it is, first of all, I work on deep learning, let\'s get that out of the way. And then beyond that, I\'m interested in understanding \[inaudible - long\]. If I had to describe in a word, I would call it the science of deep learning. This is a topic that there are few other researchers interested in. We try to understand\... We treat these as complex systems that have emergent properties due to that complexity, in the same way you might think of biology as an emergent property of physics. It\'s not something you might predict for first principles, but once it\'s there, there\'s a lot you can say about it that\'s pretty interesting. And then from there, I\'m interested in trying to understand how do these things learn and how can you make them learn more efficiently. You can think of that\... A metaphor I like to use is like pharmaceuticals. Once you understand a biological organism, you can design an intervention that takes advantage of the patterns that you\'ve seen in order to get a certain behavior or a certain outcome.

\*\*0:01:00.8 Vael:\*\* Great. Cool. I missed maybe the second sentence there. I assume that was like large NLP systems or foundations or something?

\*\*0:01:08.1 Interviewee:\*\* Yes. So I work on deep learning, so I\'m interested in the neural networks as they are in practice.

\*\*0:01:12.7 Vael:\*\* Got it. Cool. All right. And then my next question is, what are you most excited about in AI? And what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:01:23.4 Interviewee:\*\* So the thing that I\'m most excited about is that this can do extraordinary things that we don\'t know how to program, but we know we can do. Let me try to elaborate on that a little bit. What I mean by that is there\'s a world of things that we don\'t know whether anything can accomplish, tasks that are sufficiently complicated or tasks where given the input data, you may not know whether you can predict the output data. But neural networks are closing the gap between things that we know are possible to do because we do them as humans all the time, but we don\'t know how to program, or there\'s no kind of discreet handwritten program that we can write easily that will describe this. And so I\'m really excited about the fact that we can take data and actually do something that starts to resemble things that require human complexity in order to do that. So to me, that\'s the exciting part. Self-diving cars are one example, though we\'re not that good at that, but even just a mere handwritten digit recognition like the MNIST task, which is the most basic machine learning task in the world, now that I\'m in the field, but when I first took a machine learning class and saw that task, my mind was blown that you could just, with this tiny little data set, learn how to do handwritten digit recognition. That\'s pretty profound, as far as I\'m concerned, and I\'ve never quite lost my fascination for that.

\*\*0:02:35.9 Interviewee:\*\* On the risk side, I\'m concerned about a few things. I\'m not an AGI believer or an AGI concerned person, so you can cross that one off. I think \[inaudible\] huge risks, I am always concerned about\... \[inaudible - very long\]. \[Vael saying what was missed\] \...Let me start over. It was me trying to search for the right words so you missed absolutely nothing.

\*\*0:03:47.2 Interviewee:\*\* Now that I found the right words though, the two things that scare me most, number one, the fact that the systems fail in ways that are unintuitive to humans or that we wouldn\'t be able to reason about intuitively. So the example I give is, you think an automobile driver is likely to drive more poorly at night or when they\'re tired or when they\'re impaired or what have you. It\'s really hard to have an intuition about when the Tesla is going to think of the bus as a cloud, and then slam into the back of it. And it\'s those rare failures when we use a system a lot of times that lead to really bad outcomes and lead to mistrust. At the end of the day, AI needs to be trustworthy and we need to, as humans, trust it, otherwise, it will never be useful, and we\'ll never accept it or terrible things will happen, and so I think a lot about the notion of what does it mean for it to be trusted; where are the gaps between where we are right now and what it would take to be trustworthy.

\*\*0:04:41.3 Interviewee:\*\* The other big concern I have is that I think we don\'t take the problem of understanding how these systems work seriously. It used to be that when we design a big complicated piece of software, think like Windows or PowerPoint or a browser or something like that, we assumed it would take 1000 engineers to build it, and then maybe let\'s say 50 or 100 to test it. Deep learning, we have this really incredible situation where \[inaudible - long\] \... the way that I would put it, the system kind of \[?verb\] on its own based on the data. And we\'re lazy in computer science, we like to think that, \"Oh, the system came to life on its own, so we should have this one hammer that we can bang against it and see whether it\'s good or not, or whether it\'s fair or not, or use the explainability bot to understand how to explain the system.

\*\*0:05:32.1 Interviewee:\*\* But in reality, I think the ratios are just going to be reversed. The software, if you\'re using\... 100 researchers may be able to build a very exciting piece of software. You\'re going to need 1000 or 2000 engineers to tear it apart down to the studs and try to understand how each individual piece of one specific system works. So I think the entire literature around explainability and understanding of deep learning is completely wrong-headed because people are looking for general solutions to a problem that doesn\'t have general solutions. It\'s like writing a program that will debug any program for you. There\'s just no such thing. It\'s not how this game works. So that\'s scares me a lot, that we\'re thinking about this completely the wrong way. And we do need some degree of explainability, we need some degree of understanding of how these systems work, but we\'re not going to get there from the way that we\'re currently thinking about this, and it\'s just going to take a lot more effort and resources than we\'re currently remotely considering giving to it.

\*\*0:06:24.5 Vael:\*\* Interesting. And it sounded like\... So how many engineers per researcher do we need, or per model do we need, do you think?

\*\*0:06:33.8 Interviewee:\*\* Well, this is going to be complicated. The 1000 and 100 are just kind of\... Every piece of software needs a different number of engineers, and three engineers can actually build a pretty big piece of software, but the amount of\... I\'ll go out there on a limb and say it\'s going to be a 10 to 1 ratio.

\*\*0:06:53.8 Interviewee:\*\* That\'s for every 10\... Let\'s call it 10 testers. Whatever we call them, whether it\'s engineers or researchers or what have you, let\'s say that 10 people need to try to understand the system or 100 people need to understand the system. For every \[inaudible\] building the system. \...Feel free to tell me I\'m wrong if \[inaudible\].

\*\*0:07:16.5 Vael:\*\* No, I\'m just trying to actually hear you, so it\'s 10 to\... Or 10 testers or 100 testers to one capabilities person making it happen. Wow. \...Are you still here? (Interviewee: \"Yep, still here.\")

\*\*0:07:34.0 Vael:\*\* Great, alright, so that\'s pretty extreme ratio. Do you think we\'ll get there? And what happens if we don\'t get there?

\*\*0:07:42.3 Interviewee:\*\* I think we\'ll have to get there. At the end of the day, we evaluate these systems by their capabilities. If you have an autopilot system and it crashes the plane a lot, at the end of the day, you\'re not going to use it and it\'s not going to be allowed to fly. In order to get the kinds of capabilities we\'re hoping for, and to get the degree of understanding of failures that we typically expect in any high assurance system, we\'re going to need to have\... If your self-driving car crashes enough times, someone will require you to tear that thing down to the studs to understand that, and you\'re going to have to do this regardless of whether you decide to do this proactively, or eventually your system won\'t be allowed on the road until you do. So I think there\'s going to be a huge amount of manual labor in understanding how these systems work, and I think that either through failures that lead to bans of certain uses of technology until it\'s better understood or until it\'s more resilient, or someone who\'s being proactive and actually wants to ensure their self-driving car works. Either way, you\'re going to end on this situation. No better way to put it.

\*\*0:08:48.5 Vael:\*\* Okay, that\'s super fascinating. Okay, so you think that we are going to need\... The way we\'re doing interpretability is not good right now, it needs to be specific to each system, and currently, we\'re probably doing much\... Yeah, we\'re not even going in the right direction. I kind of expect that self-driving cars will be deployed without this huge amount of intervention on it, but you say that for most systems or something, we will just continue to have failures until it becomes obvious from society that we need people to do interpretability and explainability type of work in this correct way.

\*\*0:09:22.0 Interviewee:\*\* I think so, I think that\'s a good summary. The only thing I\'ll add to this, and I spend a big chunk of my life doing AI policy, so I have to think about this stuff a lot. Is that the amount of work we put into the assurance is proportional to the risk that the system has if it fails. So my favorite example of this is\-- I\'ve done a lot of work in the past on facial recognition policy. Obviously a hot topic, facial bias issues, etc. So let me give you two examples. Google Photos has this clustering algorithm where it will basically find all the faces of grandma and cluster them so that you click on one picture of grandma, and you can find them all if you want to.

\*\*0:09:58.4 Interviewee:\*\* Suppose there\'s racial bias in that algorithm, and suppose that instead the police department is using a facial recognition system to try to identify the perpetrator of a crime via a driver\'s license database. These are two applications that might both have the same technology underneath, they might both have the same biases. We\'re going to be much more worried about one than the other because one could lead people to lose their freedom, and the other may lead to people rightly feeling offended and people rightly feeling hurt, but not someone being thrown in jail potentially. So we have to handle the consequences of the systems in line with their risks. I\'m a lot less concerned about interpretability of Google\'s facial recognition system than I am about a police\'s facial recognition system.

\*\*0:10:47.8 Vael:\*\* I see. And since they need to be treated differently anyway, according to your view\... Yeah, just do it\-- do the interpretability in accordance with their importance. That makes sense.

\*\*0:10:56.7 Interviewee:\*\* Exactly.

\*\*0:10:57.7 Vael:\*\* Yeah. Cool. Alright, so my next question is about future AI. So putting on a science fiction forecasting hat, say we\'re 50 plus years into the future. So at least 50 years in the future, what does that future look like?

\*\*0:11:11.4 Interviewee:\*\* I think it\'s honestly not going to look that different from the present, just a little bit more extreme. I don\'t think we\'ll be in an AGI world. I don\'t think we will have systems that rival human intelligence. I think we will be able to specify what we want out of systems in much higher level terms and actually get them to do this\-- \[I\'m\] thinking, like, a Roomba that is actually a useful intelligent Roomba. A Roomba that knows when to vacuum, knows when not to disturb you, that you can say, \"Hey, can you please do an extra touch-up on that area,\" and it\'ll go do it. I think the current wave of machine learning is great at pattern recognition; I don\'t think pattern recognition gets us to AGI. I think the technology is going to have to look fundamentally different, and I don\'t know that we\'re any closer now than we were 50 years ago in that respect, beyond that we know there are a lot of dead ends.

\*\*0:12:05.0 Interviewee:\*\* That\'s not to say what we\'re doing right now isn\'t exceedingly useful. And \[that\] we\'re not going to push this to the nth degree, and \[that\] we won\'t have the New York Times of the future where you click on a headline and it will generate an article for you based on your interests, your knowledge level, your background on the topic and the amount of time you have to read it; I think that\'s an application that might be 10 or 15 years in the future, if not sooner than that. But I don\'t think we\'ll be in a place where we\'re worried about giving robots rights or things like that. I don\'t think that a huge amount of pattern recognition will get us to cognition, and I don\'t think the current track we\'re on will get us there. That doesn\'t bother me personally because I don\'t really care about that. I\'m much more interested in what we can do for people now and in the near future, and not how we create intelligence. That\'s more of a scary question for me than an exciting question for me, but it\'s\... \[inaudible - short\]

\*\*0:13:00.2 Interviewee:\*\* \...thinks that, heading to an AGI world, even if\... And we may be in a world where we do have self-driving cars that actually work, even if it may not be here for 10 or 15 years, \'cause it\'s a hard problem. We may have actual smart digital assistants, but that\'s \[inaudible - short\] intelligent beings who are peers or \[hard to parse\] in that respect.

\*\*0:13:25.1 Vael:\*\* Interesting. I don\'t know if it will help for me to turn my video off, but I\'m going to try it to hope it gets a little bit less choppy. Great. Okay. So you\'re talking about within the frame of AGI, like lots of people presumably talk about AGI around you. What are people\'s opinions? What do you think of their opinions? Etcetera.

\*\*0:13:46.5 Interviewee:\*\* Oh, I think there are a lot of nut cases. I mean, there are also a lot of optimists and a lot of people who have read a lot of science fiction and want to bring that to life, which is great. And a lot of people who have spent too much time in San Francisco and are surrounded by peer groups who\... Basically, there\'s a lot of monoculture in San Francisco. And I say this having just come back to San Francisco, and as someone who refuses to move \[there\] despite \[\...\]. I have friends who, if I recall correctly, go and worship at the Church of the AGI or something like that, from what I understand, under the belief that eventually there will be artificial intelligences that are smarter and more powerful than us, and therefore they should get ahead of the game and start worshipping them now, since they\'ll be our overlords later.

\*\*0:14:32.4 Interviewee:\*\* I\'m a little more concerned with the here and now, and I think that the technological leaps between great pattern recognizes that we have today and that are way far away. Just because you\'re a nutcase doesn\'t mean you can\'t change the world. And there are plenty of examples of that, and it\'s really good to have that point of view constantly echoed in the community. But I don\'t worry about the end of the world because of AGI in the way that I think some of my friends do. I don\'t consider that to be the biggest existential risk that we have, far from it. And I don\'t think it\'s a risk that we should really be spending any time worrying about at the moment.

\*\*0:15:07.6 Vael:\*\* Got it. Is that because, like you said, there\'s other existential risks to prioritize, or you don\'t think this could be an existential risk?

\*\*0:15:15.1 Interviewee:\*\* I don\'t think this is\... In the very, very long tail of ultra low probability risks to civilization, this is so far out in the tail that it\'s not worth spending any time on, independent of the fact that there are also much greater risks.

\*\*0:15:29.6 Interviewee:\*\* It\'s not just a matter of priority. It\'s also a matter of\... it\'s not a good use of any resources.

\*\*0:15:34.8 Vael:\*\* Got it. That makes sense, yeah. Unfortunately, a lot of my questions are about AGI so you\... \[chuckle\] Here we go.

\*\*0:15:42.1 Interviewee:\*\* No, I\'m happy to give you a strong contrasting opinion to many that I\'m sure you\'ve heard. So come at it.

\*\*0:15:48.1 Vael:\*\* Lovely, lovely. Okay, cool. So how I\'m defining AGI here is like any sort of very general system that could, for example, replace all human jobs, current-day human jobs, whether or not we choose to or don\'t choose to do that. And the frame I usually take is like, 2012, deep learning revolution, here we are. We\'ve only been doing AI for like 70 years or something, and here we are 10 years later, and we have got systems like GPT-3, which have some weirdly general capabilities. But regardless of how you get there, because I can imagine that we hit some ceiling on the current deep learning revolution and we need to have paradigm shifts\-- my impression is generally that if we keep on pouring in the amount of human talent and have software\-- algorithmic improvements at the rate we\'ve seen, hardware improvements, etcetera, and just like, the driving human desire to continue to follow economic incentives to earn money and replace things and make life more convenient, which I think is what a lot of what ML is aimed at right now, that eventually we will get some sort of AGI system. I don\'t really know when. Do you think we will at some point get some very general system?

\*\*0:16:55.8 Interviewee:\*\* I think if you take the time to infinity and humanity lasts in time going to infinity, yes, we will. I do think we have one example of a truly general intelligence system, namely human beings. And eventually we will probably get to the point where we could replicate that intelligence manually, if we have to literally photocopy somebody\'s brain and the transistors at some point, when we get technology advanced enough for that. Do I think that will happen this century? No. Do I think it will happen next century? Probably not. Do I think it might happen in the ones after that? Maybe.

\*\*0:17:28.8 Interviewee:\*\* So the answer is yes, in the limit, but no in any kind of limit that you or I would think about or be able to conceptualize.

\*\*0:17:35.7 Vael:\*\* Yeah, that makes sense. So you think there\'s probably going to be a bunch more paradigm shifts needed before we get there?

\*\*0:17:42.8 Interviewee:\*\* I think at least one\-- it\'s hard to know how many paradigm shifts because it\'s hard to know what they are, but I do not think this paradigm is the right one. I think this paradigm is amazing, and I\'m really excited about the kinds of machines we can build. But you can be excited about the kinds of machines we can build and recognize the limits of those machines. In the same way that\-- how are we going to get to AGI in 10 years if we\'ve been pouring\... You were talking about economic incentives. How much investment do you think has gone into self driving cars over the past, let\'s say, 10 years?

\*\*0:18:14.2 Interviewee:\*\* Let\'s call it \$100 billion plus or minus. That\'s probably the largest single investment in AI technology for an application anywhere. Ever, very likely. And where are we today on self driving cars? They\'re 90% of the way there, \[inaudible - short\] 10 times as difficult as the first 90%. And I don\'t think you\'ll be seeing fully autonomous self-driving cars in the general case on general roads in the next 10 or 15 years. So the thought that there would be an AGI at that point, let alone in 50 years is completely nonsensical to me, personally.

\*\*0:18:56.0 Vael:\*\* Yeah, that makes sense. Especially since self-driving cars are, like, robotics and robotics is behind as well. But even GPT and stuff doesn\'t really have good grounding with anything that\'s happening in the world and how\--

\*\*0:19:09.2 Interviewee:\*\* GPT\'s capabilities are also wildly overstated. You can pull out a lot of good examples out of GPT if you really want to, and you can pull out a lot of crappy ones. But we\'re not going to just brute force large language models to get our way to general intelligence. That\'s BS you\'ll only hear from someone who works at OpenAI who wants their equity to be worth more, quite frankly. The only people who say this are the ones who have an economic incentive to say this, and the people who follow the hype. Otherwise, I don\'t really know of anyone who thinks GPT is the road to AGI, especially given that we can\'t scale up any bigger. I mean, this is something, this is my whole push right now, is that the only way that Nvidia is going to come out with new GPUs next week, and they are going to come out with new GPUs next week that will be twice as fast as the ones that came out two years ago, is if they doubled the amount of power. It\'s not like we\'re doubling the amount of hardware we have available.

\*\*0:20:02.4 Vael:\*\* Got it. Do you know how good optical or quantum computing will be? I know that those are in the pipeline.

\*\*0:20:08.9 Interviewee:\*\* They\'re in the pipeline. Quantum\'s been in the pipeline for a long time, and we\'re up to what, four qubits? Cool. Again, this is one of those cases where we\'ve been awful close to nuclear fusion for a long time. We\'re going to have nuclear fusion in the limit. I\'m a 100% certain of that. When we have that, call me.

\*\*0:20:32.7 Interviewee:\*\* It could be next year. I mean, we\'re very close to crossing that threshold, but we\'ve been really close to crossing that threshold for several decades. And so, trying to call the year that it\'s going to happen within a five, 10 or 50-year time horizon, it\'s really, really tough when it comes to these technologies. And I\'m in that same place about quantum. We are making progress on quantum and I\'m really excited about it. I\'m a little bit scared of it, but I\'m excited about it, but that doesn\'t mean\... Like, crossing that threshold is really difficult. The difference between 10 years and 50 years and a 100 years away may be very small improvements in technology, but it may take an exceedingly long time for us to accomplish. Optical, same thing. So I\'m not sitting here expecting that breakthroughs are just going to happen left and right. These breakthroughs often take a very long time; a lot of incremental advances and all sorts of other technical advances in other fields to make them happen. Material science especially in the case of quantum. And we may get there in five years, or we may get there in 50 or 100 years or longer, and it\'s hard to say.

\*\*0:21:36.6 Vael:\*\* That makes sense. So what would convince you that you should start working on AI alignment? It sounds like there\'s probably going to be some breakthrough that would make you think that it\'s important, but we\'re not necessarily anywhere near that breakthrough right now. Do you have an idea of what that might be?

\*\*0:21:56.9 Interviewee:\*\* Give me your personal definition of AI alignment.

\*\*0:22:00.0 Vael:\*\* Yeah. Well, actually I want your definition first. \[chuckle\]

\*\*0:22:03.3 Interviewee:\*\* So this is not a field that I follow that closely. The entire concept of alignment has really come to the fore in the past three or four months while I\'ve been trying to \[job-related task\]. So I haven\'t been paying as much attention, so I\'d actually appreciate your definition. I can tell you the kinds of people who I see talking about alignment and the kinds of papers that I\'ve seen, the paper titles that I\'ve seen go across my desk. But I couldn\'t give you a good definition even if \[inaudible - short\].

\*\*0:22:33.9 Vael:\*\* Yeah. So one of the definitions I use, and I\'ll give you a problem setting I usually think about as well\-- so, \[the\] definition is building models that represent and safely optimize \[inaudible\] specify human values. Alternatively, ensuring that AI behavior aligns with system designer intentions. And one of the examples I use for what an alignment problem would be is the idea that highly intelligent systems would fail to optimize exactly what their designers intended them to and instead do what we tell them to do. So the example of OpenAI, trying to\... Have that boat win a race and then it getting caught on like some little\-- but optimizing instead for a number of points and instead ending up in this little\-- side, collecting-points area, instead of winning the race. So doing what the designers told it to do instead of what it intended them to.

\*\*0:23:26.2 Interviewee:\*\* So, I mean, this sounds like a\... If you want my honest frank opinion.

\*\*0:23:34.8 Vael:\*\* Yeah.

\*\*0:23:35.3 Interviewee:\*\* A BS-ey rebranding of a simple fact of life in computing for\... Since the dawn of computing, the computer does what you tell it to do, not what you want it to do. (Vael: \"Yes.\")

\*\*0:23:47.5 Interviewee:\*\* And so, I don\'t\... I know a lot of people at OpenAI think they\'re very deep and profound for calling it AI alignment. \[inaudible - very long\].

\*\*0:25:02.7 Interviewee:\*\* So picking up where I was, there are probably a lot of folks at OpenAI, and I know exactly who they are, who think they\'re very deep and profound who are wondering about this question, but this is kind of the obvious fundamental thing that every first year programmer learns and is a question that everybody has been asking in every context whenever they develop a loss function for a neural network. Your loss function never reflects exactly the value that you want the system to carry exactly, even just the outcome. We optimize language models to be good at predicting the next word and yet they somehow also generate\... We want them to have properties that go above and beyond that, we want them to be able to even just transfer the representations for downstream tasks.

\*\*0:25:44.0 Interviewee:\*\* This is just a question of to what extent is the thing you\'re optimizing a system for going to actually align with the task that you want. I just, I don\'t see what\'s interesting or new about this question from a research perspective. I think of course it\'s an important one, but it\'s not a\... \[inaudible - long\] Saying, oh, we should worry about computer security. Well, yes, but there\'s no security robot that fixes all computer security. It\'s a complex context-specific problem. Again, people say things that are aligned with their economic incentives and that is certainly true for my friends at OpenAI but I don\'t see any profundity in this observation that\'s just the nature of computing.

\*\*0:26:38.1 Vael:\*\* Yeah. I think it has to be paired with the idea that if you have a very intelligent system that can plan ahead, that can model itself in the world, that it may have an incentive to preserve itself just as an agent pursuing any goal, because it doesn\'t want to decrease its chance of it succeeding at its goal. I think that has to be paired. Otherwise, it\'s not particularly special. But I do think creating an agent that in the far future, whenever AGI develops, that has an incentive to not be modified makes it much more dangerous than anything we\'ve seen previously.

\*\*0:27:16.2 Interviewee:\*\* I agree with that factor, when we get to a point where this becomes a problem. But I think you can see that the amount I care about this question is proportional to the amount that I think that AGI is or will be a concern in your lifetime, or your grandchildren\'s lifetime. And so I think there are a lot more fundamental basic questions. Like, we don\'t even understand why a neural network actually is able to recognize handwritten digits in the first place. And until we get these basic things down, we\'re never going to get to build the kind of systems that have these properties anyway.

\*\*0:27:44.7 Interviewee:\*\* So, I would make the analogy of, let me see, I\'m trying to think of the right metaphor\... I don\'t know, for the folks who worry about how we\'re going to communicate with the alien civilizations we inevitably come into contact with\-- we better build some space ships that can get us to space first before we start worrying. And then figure out whether there are alien civilizations out there before we start worrying about how we\'re going to communicate with them.

\*\*0:28:05.6 Vael:\*\* Yeah, this sounds like a very coherent worldview, I\'m like, yep, makes sense, is logical.

\*\*0:28:11.7 Interviewee:\*\* Yeah, I have strong opinions, and researchers are always incentivized to have strong opinions; it\'s what moves science forward. We argue and we disagree with each other, and we\'re all right, and we\'re all wrong. But these are my strong opinions. And if you were to chat with any of my friends at OpenAI, you\'d hear the opposite view and we could still go out for drinks and have a good time.

\*\*0:28:32.5 Vael:\*\* Great. So I think this line of questioning was originally developed by me asking what would you see\... What would you want to see in the world before you\'re like, \"Oh actually, people are right, this thing is coming earlier than I expected.\"

\*\*0:28:47.6 Interviewee:\*\* That\'s a good question. Honestly, I would want to see a\... I\'m thinking of examples of a task, so like a Turing test of some sort that would fit here. Obviously, the Turing Test is not a very effective test given that GPT-3 probably passes it with flying colors, despite the fact that GPT-3 is a very good language model. What I would look for is long-lived machine learning systems that can learn continually. We don\'t even have learning continually down. We have reinforcement learning systems that eventually become effective agents at solving one particular task; they can\'t move on to a second task, we don\'t have any general kind of learning process. Even the reinforcement learning agents that we have have to be really, really, specially hand-calibrated for a specific task, to the point where even the specific random seed you use is an important hyperparameter in determining whether the agent will succeed or fail at a given task.

\*\*0:29:50.0 Interviewee:\*\* If we\'re in that world, we\'re very far away from an agent that can learn many tasks or has a generalized learning process. This is I guess why people are excited about meta-learning in general. Again, I think we shouldn\'t worry about light speed travelling until we can get off the planet, but that\'s a whole different\... You know where I stand on that. So\... \[inaudible - short\] and improve themselves over time; reinforcement learning is kind of a shadow of what we would expect of a real system. A smart home assistant, where you can teach it new tasks by explaining it new tasks, and it will be able to figure out how to accomplish them. We don\'t have any kind of abstract or general learning capabilities right now. And we talk a very big game about things like AlphaFold or what have you, or any of the Alpha stuff coming out of DeepMind. But these are requiring lots and lots and lots of engineers to get a system to work on one very specific setting in ways that don\'t transfer to any other setting or any other task. There are principles we\... \[inaudible short\] being really hard trying a bunch of stuff. And that\'s a\... \[inaudible long\] real general learning process that will keep me up at night.

\*\*0:31:03.5 Vael:\*\* Got it. Okay. Interesting. So do you think you and your colleagues probably have the same information but are just reacting to it differently? Other people are also not seeing these very general systems or maybe\-- yeah, presumably, because you have access to the same information, but you\'re just interpreting it differently based on the incentives that your company or whatever is in?

\*\*0:31:29.9 Interviewee:\*\* I wouldn\'t always blame the incentives and the all, again, throwing some shade at San Francisco in particular. When you spend all of your waking hours and all of your sleeping hours around exclusively people who are working on the same \[inaudible\], same field. And when you work as they do not only in the tech industry, but especially in the very tight knit AI community, where if we all go to the same birthday parties when I\'m out there, and everybody goes to the same spas, and runs into each other on the street, despite the fact that San Francisco is a big city. When you live in that kind of monoculture, quite frankly, you\'ll lose touch with reality, to some extent. And I think, a lot of people start believing that\... \[inaudible very long\]

\*\*0:32:31.9 Vael:\*\* Okay, cool. My question was, although I missed that last bit, but my question was, how do people stay in touch with reality, how do they\...

\*\*0:32:43.7 Interviewee:\*\* I think you have to talk to people who are doing things other than building AI systems. You have to actually maybe interact with someone who works in healthcare or someone who works in finance, or someone who works in any other field besides AI or besides machine learning. And you have to remember that they are real human beings out there. I think this is just a general San Francisco problem. You see people, you see 20-somethings who are wealthy, producing systems that are convenient for them. We see this all the time, startups coming out of the Bay Area.

\*\*0:33:24.1 Interviewee:\*\* My experience in Boston and New York is it\'s a very different start up culture of healthtech, fintech, edtech, what have you, things that are quite frankly real and useful. So when I hang out with\... The times I\'ve been working at \[large tech company\] in California, for example, all of my friends are worried that the sky is falling for some reason. Maybe we\'re about to enter an AI winter or we\'re about to create AGI, it\'s usually one or the other. Depending on who you talk to and what mood they\'re in, and what\'s going on in their organization at that different point in time. But people are just so caught up in the inside baseball, they\'ve lost the bigger context for everything we\'ve accomplished, but also how far away we are from some of the things people talk about. And you gotta actually taste the real world from time to time, and remember that you don\'t just live in this little bubble of lots of neural networks and lots of people with PhDs working on neural networks.

\*\*0:34:19.7 Vael:\*\* And interacting with other people and seeing the wider world gives the context to believe more\... mainline views? Because it sounds like people end up in either direction, you said they believe that everything is about AI, everything\'s going to go badly: everything is going to go badly and too fast or everything\'s going too badly and too slow. And there\'s some regularity effect that happens if you\'re hanging out with other people?

\*\*0:34:50.4 Interviewee:\*\* It\'s almost the opposite of\... \[inaudible - short\] in that respect. I think it\'s partially because people will say things like, \"Well, there hasn\'t been a big advance or there hasn\'t been a big breakthrough in 18 months.\" And my reaction is, it\'s been a\... Yes, the transformer paper came out 18 months ago. Yeah, let\'s be patient, and the field does not always continually accelerate. No scientific field continually accelerates. We have periods of big progress and then periods of stagnation and paradigm shift. This is the structure of scientific revolution. If you want to\... For the extent to which you take that view of the world seriously, that does, in my impression, simply how science moves, in fits and starts, not at a steady pace. And in computers we\'re used to exponentials. That doesn\'t mean that these systems are going to get exponentially better over time. It may mean we had 10 really good years of progress based on a bunch of different factors all coming together, between big data being accessible, lots of computing accessible, and some improvements in deep learning, that all came together really nicely to give us a big burst of progress.

\*\*0:36:00.1 Interviewee:\*\* Maybe things will slow down a little bit. Or maybe there will be a new architecture in five years that will be another big burst, the way that convolutional neural networks were around the AlexNet breakthrough and then the way that transformers were. But this happens in fits and starts. But around \[large tech company\], partially because of \[large tech company\]\'s promotion process and partially because of the very internally competitive atmosphere that that fosters, people are always dour and feel like the sky is falling. And at OpenAI, the incentive is to continue the hype and build really big systems. It\'s partially in the culture, partially in the leadership, partially in the brand of the place and what it takes to make your stock go up in value. So I think people really are shaped by their environments in that respect, and you gotta get out and talk to other kinds of people and get other perspectives, and I don\'t think there\'s that much you can do when you\'re in a big organization like \[large tech company\], and there\'s not a lot you can do when you\'re in San Francisco and everybody you talk to. It\'s not \"what you do\". It\'s \"which tech company do you work for\" or more likely \"which AI project do you work on\".

\*\*0:37:04.4 Vael:\*\* I see, so this is probably, like you said, why you haven\'t moved and also presumably why you have friends who are not AI researchers.

\*\*0:37:12.4 Interviewee:\*\* Yeah, my office right now is at \[university\] \[non-science field\]. I just sit by a bunch of \[non-science field\] professors. I\'ve worked there at times in the past. I live in \[city\] right now, and they were kind enough to offer me a place to sit and work for a little bit. So I\'m hanging out with a bunch of \[non-science field\] faculty all day. It\'s a very different perspective on the world. They\'re worried about very mundane things compared to AGI coming and killing us all, or a quantum computing breakthrough that leads to the end of civilization when we train quantum neural networks or something like that. They\'re much more worried about bail reform, or very basic mundane day-to-day problems that are actually affecting people around them. It\'s hard to get caught up in the hype.

\*\*0:37:56.5 Vael:\*\* Yeah, and it seems like it\'s important to make sure that people are working on things that actually affect more than just their local environment. I think you mentioned something like that earlier, right?

\*\*0:38:05.9 Interviewee:\*\* Exactly. And there\'s also a little bit of\... as computer scientists, we love to calculate things, we love to measure things. Effective Altruism is one great example of this. We love to say, \"What is the way that I can put my dollar the furthest along?\" And I think sometimes people lose touch with the fact that there are humans out there. There are people right now, you ask questions like, well, either I can help the humans today or I can put all my resources toward saving the entirety of civilization. And civilization and the future is obviously\-- there are many more humans there than there are here today, so I\'ll worry about, on the 0.1% chance that we develop AGI and it comes to threaten us all, I\'ll worry about that problem, because in expectation, that will save more lives than what have you. And I think\--

\*\*0:38:51.5 Vael:\*\* \--Yeah, and this is the wrong way to think about things?

\*\*0:38:56.5 Interviewee:\*\* I don\'t think this is necessarily the wrong way to think about things, but I think it\'s a little bit\... Sometimes, it\'s more important to focus on what\'s right in front of you and what\'s tangible and what\'s here and now. Or the other way I\'ll put it is, MIT is pathologically bad as an institution at dealing with topics that can\'t be measured for exactly this reason. It\'s really hard to talk to people at MIT about values, unless a value can be measured or optimized.

\*\*0:39:25.0 Interviewee:\*\* And this is just, for a lot of folks who get empirical education and who are training systems all day, for whom this is a problem, they are just as much\... The people are just as much subject to this problem as the systems they build. So, if we want to talk about alignment, it\'s an issue not just of the systems, but of the people building the systems.

\*\*0:39:43.5 Vael:\*\* Alright, well, we are at time. Thank you so much, this is a very different opinion than the ones I\'ve received so far, and yeah, very well presented. Awesome.

\*\*0:39:52.6 Interviewee:\*\* Thank you, that\'s what I\'m here for. You know where to find me if you need to chat more. I\'m very exited to see what the outcome of the project is.

\*\*0:39:58.4 Vael:\*\* Great. Alright. Well, thank you so much.