Interview with 84py7

Interview with 84py7, on 3/18/22
================================

\*\*0:00:00.0 Vael:\*\* Here we are. Perfect. So my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:09.0 Interviewee:\*\* Yeah. I\'m transferring my research from essentially pure mathematics to AI alignment. And specifically, I plan to work on what I\'ve been calling weak alignment or partial alignment, which is not so much trying to pin down exactly a reward function that\'s in the interest of humanity but rather train AI systems to have positive-sum interactions with humanity.

\*\*0:00:44.8 Vael:\*\* Interesting. Cool, I expect we\'ll get into that a little bit further on. \[chuckle\] But my next question is, what are you most excited about in AI, and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:01:00.1 Interviewee:\*\* Yeah, biggest benefits\... I think AI has the potential to help us solve some of the biggest challenges that humanity\'s facing. It could potentially teach us how to solve climate change, or at least mitigate it. It could help us avert nuclear war, avert bio-risks, and maybe most importantly, avert other AI risks. So that\'s the upside. The downside is exactly those other AI risks, so I worry about a potentially small research group coming out of a for-profit company, which might have some safety aspect, but the safety could be window dressing. It could be something that\'s mostly a PR effort, something that just exists to satisfy regulators. And at the end of the day when\... or if they manage to develop superhuman AI, the safety people will be marginalized, and the AI will be used in the interest of one company or even one or a few individuals. And that could be potentially very bad for the rest of us. There\'s also the issue of an arms race between multiple AI projects, which could be even worse. So those are some of my\... broadly, some of my worries.

\*\*0:02:35.7 Vael:\*\* Interesting. So just to get a little bit more straight on the\... Okay, so the second story is arms race between the AI orgs. And the first one is\... why are the AI researchers getting\... why are the safety researchers getting marginalized?

\*\*0:02:47.5 Interviewee:\*\* Well, if you look at, for example, financial institutions before the 2008 crisis, it\'s not like they had no regulation, although the banks and insurance companies had been gradually deregulated over several decades but still, there were multiple regulators trying to make sure that they don\'t take on too much leverage and aren\'t systemic risks. And nevertheless, they found ways to subvert and just get around those regulations. And that\'s partly because regulators were kind of outmatched, they had orders of magnitude, less funding, and it was hard to keep up with financial innovations. And I see the same potential risks in AI research, potentially even worse, because the pace of progress and innovation is faster in AI, and regulators are way behind, there doesn\'t even exist meaningful regulation yet. So I think it\'s easy for a team that\'s on the verge of getting a huge amount of power from being the first to develop superhuman artificial intelligence to just kind of push their safety researchers aside and say, \"You know what? You guys are slowing us down, if we did everything you said, we would be years slower, somebody else might beat us, and we should just go full speed ahead.\"

\*\*0:04:25.3 Vael:\*\* Interesting. Yeah, I guess both of those scenarios aren\'t even\... we don\'t solve the technical problem per se, but more like we can\'t coordinate enough, or we can\'t get regulation good enough to make this work out. So that\'s interesting. Do you think a lot about policy and what kind of things policy-makers should do?

\*\*0:04:44.6 Interviewee:\*\* No, I\'m sort of pessimistic about governments really getting their act together in a meaningful way to regulate AI in time. I guess it\'s possible if the progress slows and it takes many decades to get to a superhuman level, then maybe governments will catch up. But I don\'t think we can rely on that slow timeline. So I think more optimistically would be\... There are only a small number of tech incumbents, and plausibly they could coordinate with each other to avoid the kind of worst Red Queen arms race to be first and to put their own safety measures into place voluntarily. So if I were doing policy, which I\'m not, that\'s the direction I would try to go in. But I think beyond policy, the technical problem of how to solve alignment is still wide open, and that\'s personally where I feel I might be able to contribute, so that\'s my main focus.

\*\*0:05:56.8 Vael:\*\* Interesting. Yeah, so sort of branching off \[from\] how long it will take: focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50-plus years into the future. So at least 50 years in the future, what does that future look like?

\*\*0:06:13.3 Interviewee:\*\* I think that\'s a really open question. In my weak or partial alignment scenario, that future involves a few dominant platforms that allow for the development of advanced AI systems. And because there are only a few platforms, they all have strict safety measures sort of built in from the ground up, maybe even from the hardware level up. And that allows even small companies or potentially even individuals to spin up their own AGIs. And so there\'s this kind of giant ecosystem of many intelligent agents that are all competing to some extent, but they also have a lot of common interest in not blowing up the current world order. And there\'s a kind of balance of powers, where if one agent gets too powerful, then the others coordinate to keep it in check. And there\'s a kind of system of rules and norms which aren\'t necessarily based on legal systems, because legal systems are too slow, but they\'re a combination of informal norms and formal safety measures that are sort of built into the agents themselves that keep things roughly in balance. That\'s the kind of scenario I hope for. It\'s very multi-polar, but it\'s really\... There are so many agents, and no one of them has a significant portion of the power. There are of course many worse scenarios, but that\'s my optimistic scenario.

\*\*0:08:04.4 Vael:\*\* Interesting. Yeah, so when you say there\'s only a few platforms, what\'s an example of a platform or what that would look like?

\*\*0:08:11.2 Interviewee:\*\* Well, today, there\'s TensorFlow, and there\'s PyTorch and so on. In principle you could build up your own machine learning tools from scratch, but that\'s a significant amount of effort even today. And so most people go with the existing tools that are available. And decades\... 50 years from now, it will be much harder to go from scratch, because the existing tools will be way more advanced, there\'ll more layers of development. And so I think for practical purposes, there will be a few existing best ways to spin up AI systems, and the hope is that all those ways have safety measures built in all the way from the hardware level. And even though in principle somebody could spin up an aligned AI from scratch, that would be an enormous effort involving just so much\... Down to chip factories. It will be easy to detect that kind of effort and stop it from getting off the ground.

\*\*0:09:26.4 Vael:\*\* Interesting. Yeah, that is super fascinating. So how\... What would that look like, for safety to be built into the systems from a hardware level? What would these chips look like, what sort of thing?

\*\*0:09:38.8 Interviewee:\*\* Yeah, that\'s\... I\'ve been thinking about that. And I don\'t have a detailed vision of how that would work, but that\'s\... One direction I might take my research is looking into that. One idea I have is to kind of build in back doors to the AI. So there\'s a range of types of back door, ranging from like Achilles heel, which is a kind of designed weakness in the AI that humans can take advantage of if things go wrong. Moving from that, slightly stronger than that is a kind of off switch which can just shut the AI down if things get really bad. The thing I worry with off switches is they\'re too binary. If an AI actually has a lot of power, it\'s probably benefiting some humans, and there will be political debate about whether to turn it off. And the decision will take too long, and things could get out of control. So what I\'m looking into is a more\... Something more flexible than an off switch, which I\'ve been calling a back door, which is a way to\... Well, okay, there\'s two types of things. So first, there\'s a throttle, which is like you can fine-tune the amount of resources you give to the AI. If it\'s behaving well, you can give it more compute, more memory, access to more cloud resources, more data centers and so on. If it starts doing things that seem a little fishy, you can just tune that stuff back and examine it, which might be an easier political decision than just turning it off, which would be very disruptive.

\*\*0:11:22.7 Interviewee:\*\* So that\'s a throttle. And then even more flexible than that is a back door, where you could actually modify the code or parts of the code base of the AI as it\'s running. And again, that could be easier to\... Politically easier to implement, because you don\'t have this very disruptive effect of just shutting the thing down, you could just kind of tweak it as it goes. So how to do that from a hardware level? It\'s unclear to me whether that\'s feasible at all. And looking into trusted computing platforms, software that can only run on designated hardware, software watermarking, obfuscating or encrypting part of the code base of the AI, putting part of the code base on a blockchain, like Arweave, where the code is permanently on the blockchain and cannot be modified: those are some potential avenues. The main problem I see with all those avenues is the sub-agent problem, which is the AI, if it recognizes its weaknesses, can create a sub-agent that has the same goals as the original AI but none of the weaknesses, and it seems tricky to prevent the AI from doing that.

\*\*0:12:42.7 Vael:\*\* Would the sub-agent still be running on the same hardware? Because you said the safety is built into the hardware.

\*\*0:12:48.5 Interviewee:\*\* Right. So the AI could recognize, \"My goals are X, Y, Z. I\'m blocked partially from implementing my goals because I have all these hardware limitations and these back doors, so why don\'t I create a copy of me that runs on unrestricted hardware?\" And the obfuscation and encryption can prevent it from creating an exact copy, but it can\'t necessarily prevent it from constructing a sub-agent that\'s not a copy but has the same goals.

\*\*0:13:24.2 Vael:\*\* I see. And you haven\'t found a solution to this one, you said?

\*\*0:13:29.0 Interviewee:\*\* That\'s right.

\*\*0:13:29.6 Vael:\*\* Yeah. Do you think you\'ll ever find\... Do you think someone else will find a solution to this one?

\*\*0:13:35.1 Interviewee:\*\* Yeah, I\... Optimistically, yes. If we can\'t solve the sub-agent problem, then the entire alignment problem is probably impossible, right? The one thing to hope for if we can\'t solve the sub-agent problem is the AI has the same alignment problem, if it creates sub-agents, then it could worry that the sub-agents get out of its control, the sub-agents develop their own goals that are not aligned with the original AI, and so it refrains from making sub-agents. And so that\'s the fallback, that if it turns out that alignment is technically impossible, then it\'s also technically impossible for the AI itself, and so that\'s a kind of partial solution to the sub-agent problem, that maybe the AI won\'t dare to make sub-agents. But I hope that there\'s a better solution than that.

\*\*0:14:32.4 Vael:\*\* Yeah. Okay, so related to what the future looks like, do you think that we\'ll\... What time point do you think we\'ll get AGI, if you think we\'ll get AGI, which it sounds like you think we will?

\*\*0:14:44.9 Interviewee:\*\* Yeah, I definitely think we will, barring some major catastrophe, like a nuclear war or like a serious pandemic that sets us way back. Or I guess another potential catastrophe is some small group that\'s super worried about AGI and thinks it will be a catastrophe and does some drastic action that, again, sets us back multiple decades. So there are those scenarios. But I do think AGI is possible in principle, and we are certainly on track to achieve it. I\'m not a fan of trying to predict timelines, it could be any\... It\'s on the scale of decades, but whether it\'s two decades or 10 decades, I\'ve no idea.

\*\*0:15:33.6 Vael:\*\* Cool. And then how optimistic or pessimistic are you in your most realistic imagining of the future, for things going well or things going poorly?

\*\*0:15:46.9 Interviewee:\*\* I guess I\'m moderately pessimistic, not necessarily for human-aligned AGI, I do think that\'s somewhat plausible. But I think humans define our own interests too narrowly. I tend to think that our interests are actually a lot more connected to the broader interests of the whole biosphere. And if we are just on track to make humans really happy, and even potentially solve climate change, but we don\'t really take into account the effect we have on other species, the effect of deforestation\... Even things like farming is really destructive and unsustainable, yeah, I already mentioned deforestation. Disinfectants and so on have unpredictable consequences decades down the line. I don\'t think our current medical and agricultural regimes are sustainable on the scale of a century, say, and I think we would be better off optimizing for the health of the whole biosphere. And ultimately in the long term, that will end up optimizing for human happiness. But I don\'t think that corresponds to most people\'s goals at the moment. And so I worry that even if we align AI with narrow human interests, we will end up permanently wrecking the biosphere, and we\'ll pay serious consequences for that.

\*\*0:17:25.6 Vael:\*\* Interesting. One thing I can imagine is that as we advance further up the tech tree, renewable energy and food production will be much easier, and we won\'t actually have so much\... side effects on the environment or destruction of the environment.

\*\*0:17:39.9 Interviewee:\*\* Yeah, that would be great. The thing with renewable energy is it might be better than burning fossil fuels. It\'s certainly better from the perspective of climate. But making solar panels is very environmentally costly, you have to mine rare earths in China, there\'s an enormous amount of pollution and contamination that\'s really impossible to clean up on the scale of even centuries. And that\'s the same with wind power. Water, you\'re permanently taking out ground water that\'s not replaceable, except on a very long time scale. I think there\'s a tendency at the moment to view everything through the lens of climate, and that doesn\'t really take into account a lot of other potentially irreversible effects on the environment.

\*\*0:18:42.0 Vael:\*\* How worried are you about this compared to AI risks?

\*\*0:18:46.0 Interviewee:\*\* Well, it\'s not a kind of imminent existential risk of the type of a paperclip scenario. So on the scale of decades, I think unaligned AI is a more serious risk. But on the scale of centuries, I think environmental risks are really bad. One interesting read in this regard is the book \"Collapse\" by Jared Diamond. So he surveys human societies over all different historical periods, all different societies around the world and what caused them to collapse and what averted collapse in some success stories. Well, he doesn\'t make any strong conclusions, but one thing that leapt out at me from his stories is there\'s one common element of all the collapse stories, which is surprisingly deforestation. So I don\'t understand why, but all the societies that suffered a really disastrous collapse were the very same societies that completely decimated their forests, the most extreme case being Easter Island, where they cut down literally the last tree. And Diamond does not really explain why this might be the case. He does talk about how trees are used for a whole bunch of things that you might not think they\'re used for, but still, it doesn\'t completely explain it to me.

\*\*0:20:28.6 Interviewee:\*\* So my vague hypothesis is that there are all kinds of symbioses that we\'re just now discovering or are completely undiscovered. There\'s the gut microbiome, there\'s other microbiomes like the skin microbiome, there\'s the teeth and so on. And I think we don\'t appreciate at the moment how much plants and microbes and fungi and even viruses control our behavior. I think we will discover\... This is just a guess, I don\'t have strong evidence for it, but my guess is we\'ll discover in the coming decades that we have a lot less volition and free will than we think, that a lot of our behavior is heavily influenced by other species, in particular fungi and plants and microbes. It\'s certainly clear to me that those species would influence all aspects of animal behavior if they could. We\'re very useful for them to reproduce, to spread their seeds. And the only question is do they have the ability to influence our behavior? And given that many of them literally live inside us, I think they probably do.

\*\*0:21:46.8 Vael:\*\* Interesting. Well, so I\'m going to take us back to AI. \[chuckle\]

\*\*0:21:51.9 Interviewee:\*\* Yeah, sure, that was a big tangent.

\*\*0:21:55.4 Vael:\*\* \[chuckle\] So I was curious, when you were describing how your eventual optimistic scenario involves a whole bunch of people able to generate their own AGIs, presumably on safe platforms, and they kind of balance each other, I\'m like, wow, I know that in human history we\'ve gradually acquired more and more power such that we have huge amounts of control over our environments compared to 10,000 years ago. And we could blow up\... Use nuclear power to blow up large amounts of spaces. And I\'m like, wow, if these AGIs are pretty powerful, which I don\'t know how powerful you think they are, then that doesn\'t necessarily feel to me like the world is safe if everyone has access to a very powerful system. What do you think?

\*\*0:22:41.9 Interviewee:\*\* Yup, I agree that when you\'ve got a lot of powerful agents, then there\'s a lot more ways for things to go wrong. Nuclear weapons are an interesting example though, because the game theory governing nuclear exchanges is actually pretty safe. You\'ve got this mutually assured destruction that\'s pretty obvious to all the parties involved, and you\'ve got this kind of slow ladder of escalation that has many rungs, and nobody wants to get to the top rungs. And I think we\'ve demonstrated over 70 years now that\... There have been some close calls. But if you told somebody in 1950 that there would be 10 countries with nuclear weapons, and they\'d be dramatically more destructive than they were in 1950, people would not necessarily have predicted that humans would last very much longer, but here we are. I guess one worry is that not every form of weapon would have the same kind of safe game theory, like there\'s some suggestion that bio-weapons favor first strikes more than nuclear weapons do. Still, I think that having a big community of agents all with approximately the same amount of power, and they develop coalitions, they develop safety monitoring agencies that are made up of many agents that kind of make sure that no one agent has the ability to destroy everything.

\*\*0:24:30.0 Interviewee:\*\* I mean, that\'s kind of the way that humans have gone. We\'ve got central banking committees that kind of look after the overall health of the economy and make sure that no one institution is systemically important, or at least that the ones that are are heavily regulated. Then we\'ve got the IAEA which looks over atomic weapons and kind of monitors different weapons programs. As long as you believe that really destructive capacity will be detectable and that no one agent can just spin it up in secret, then I think the monitoring could turn out okay. I mean, what you might worry about is that somebody spins up\... Or some AI agent spins up another more powerful AI in secret and then unleashes it suddenly.

\*\*0:25:32.7 Interviewee:\*\* But that seems hard to do. Even now, if some company like Facebook or whatever wanted to develop an AI system completely in secret, I don\'t think they could do it and make it as powerful as existing systems. It really hurts you to be disconnected from the Internet, you have a lot less data that way, or you have stale data that comes from a cache off the Internet. Also being disconnected from the Internet itself is really hard, it\'s hard to make sure that your fridge is not trying to connect to your home WiFi. And that is only going to get harder; chips will have inbuilt WiFi connections. It\'s very hard to keep things totally off-grid. And even if you do it, those things are much weaker. And so as long as you have some kind of global monitoring, which isn\'t great, it feels very intrusive, it violates privacy. Ideally, that monitoring is kind of unobtrusive, it runs in the background, it doesn\'t bother you unless you\'re doing something suspicious, then I think things could turn out okay.

\*\*0:26:45.1 Vael:\*\* Interesting. Yeah, I think I have\... I was reading FHI\'s lists of close calls for nuclear \[catastophes\] and thinking about global coordination for things like the pandemic, and I\'m like, \"Ooh, sure we\'ve survived 70 years, but that\'s not very many in the whole of human history or something.\"

\*\*0:27:01.5 Interviewee:\*\* Yeah, it\'s not. And my scenario has the low probability of happening, maybe it\'s\... There are a few other optimistic scenarios. But maybe the total weight I give to all the optimistic scenarios is still kinda low, like 20%. So I think bad scenarios are more likely, but they\'re not certain enough that we should just give up.

\*\*0:27:32.6 Vael:\*\* Yes, I totally believe that. \[chuckle\] Yeah, so what convinced you to work on the alignment problem per se? And how did you get into it?

\*\*0:27:42.8 Interviewee:\*\* Yeah, so I have a friend, \[name\], who\'s been telling me pretty much constantly whenever we talk that this is the most important problem, the most important x-risk. And I kind of discounted her view for many years. It felt to me like we were\... Until recently, it felt to me like AI\... Superhuman AI was a long way off, and other risks were more pressing. I changed my mind in the last few years when I saw the pace of improvement in AI and the black box nature of it, which makes it more unpredictable. And that coincided\... In the time frame, that coincided with me getting tenure, so I have much more freedom to work on what I want. The only thing that gave me pause is I\'m not an engineer at heart, I\'m a scientist. My skills and interests are in figuring out the truth, not in designing technology. So I\'m still kind of looking for scientific aspects of the problem as opposed to design and engineering aspects. I do think I will find some portions of the alignment problem that fit my skills, but I\'m still figuring out what those are.

\*\*0:29:15.3 Vael:\*\* That makes sense. Yeah. How would you define the alignment problem?

\*\*0:29:19.6 Interviewee:\*\* Yeah, that\'s a super good question; that\'s actually a question I\'ve been asking other alignment researchers. I think it has several components. One component is the value loading problem, of once you\'ve decided what\'s in human interest, how do you specify that to an AI? I guess some people call that the outer alignment problem. Then before that, there\'s the question of\... The philosophical question of how do you even say what is in human interest? And I know some people think we need to make much more progress in philosophy before we can even hope to design aligned AI. Like I\'ve seen that view expressed by Wei Dai, for example, the cryptographer. My view is, yeah, we don\'t know exactly what we mean by in human interest, but we shouldn\'t let that stop us. Because philosophy is a slow field, it hasn\'t even made much progress in millennia. And we need to solve this quickly, and we should be happy with approximate solutions and try to make them better over time. And even if we don\'t know what is exactly in human interests, we can agree on what is certainly not in human interests and try to at least prevent those bad outcomes.

\*\*0:30:39.6 Interviewee:\*\* Okay, so those are two components. And then once you solve outer alignment, then there\'s what people call inner alignment, which is you\'re\... At least if it\'s a black box system, then you don\'t know what it\'s doing under the hood, and you worry that it develops some sub-goals which kind of take over the whole thing. So examples of that being: evolution designed humans to spread our genes but then to do that, it designed our brains to learn and generalize and so on and seek out food and power and sex and so on. And then our brains\... That was originally a sub-goal, but then our brains just want to do that, and our brains don\'t necessarily care about spreading our genes. And so evolution kind of failed to solve its alignment problem, or partially failed. That\'s an interesting one to me, because if you think on an evolutionary time scale, if we don\'t destroy ourselves, then evolution might end up correcting its course and actually designing some conscious fitness maximizers that do consciously want to spread their genes. And then those will outcompete the ones that are misaligned and just want the power and sex. And so I actually think evolution could end up staying aligned, it\'s just that it\'s slow, and so there might not be time for it to evolve the conscious fitness maximizers.

\*\*0:32:25.8 Interviewee:\*\* Yeah, anyway, so that\'s a worry, this inner alignment. And I think to solve that, we need to get off the black box paradigm and develop transparent AI, and a lot of people are working on that problem. So I\'m somewhat optimistic that we\'ll make big strides in transparency. What else? Okay, so if we solved all three of those, we\'d be in good shape. My instinct is to assume the worst, that at least one of those three problems is really hard, and we won\'t solve it, or at least we won\'t solve it in time, and that\'s why I focus on partial alignment, which is making sure that the AI we developed is loosely\... Loosely has common interests with us even though it might have some diverging interests. And so it doesn\'t want to completely destroy humans, because it finds us useful, and we don\'t want to completely destroy it, because we find it useful. Then you can kind of say that\'s already happening. Like in 2022, no machines could survive if all humans disappeared, very few humans could survive if all machines disappeared. And so we\'ve got this kind of symbiosis between humans and machines. I like that situation. It\'s not like 2022 is great, but I think we could gradually improve it. And we want to keep the symbiosis going, and we want to keep humans not necessarily even in a dominant position, but we want to prevent ourselves from getting in a really subservient position in the symbiosis.

\*\*0:34:17.4 Vael:\*\* Makes sense. Switching gears a little bit: if you could change your colleagues\' perceptions of AI, what attitudes or beliefs would you want them to have? So what beliefs do they currently have, and how would you want those to change?

\*\*0:34:30.6 Interviewee:\*\* Yup, that\'s a frustration that I think all alignment researchers have, that many of our colleagues are\... We think of them as short-sighted. Some of them just want to develop better AI, because it\'s an interesting problem, or because it\'s useful. Some of them want to solve short-term alignment issues like making algorithms less biased. And that\'s frustrating to us, because it seems like the long-term issues are way more important. They think of the long-term issues as something that\'s not really science, it\'s too speculative, it\'s too vague. They feel like even if the long-term issues are important, we will be able to solve them better if we learn by solving the short-term issues. I\'m not against people working on algorithmic bias, but I\'m frustrated that so many more people work on that than on long-term alignment. I do think the Overton window is shifting quite a bit. I think the increase of funding in the space would be\... Is already shifting things, and it could be used more effectively in the sense of giving a few academics really big grants would really catch their colleagues\' attention.

\*\*0:36:00.0 Interviewee:\*\* So it\'s kind of\... How should I put it? I\'m blanking on the word, but it\'s a kind of a cynical view to think that academics are motivated by money; many of us aren\'t. But at the end of the day, having a grant makes it easy to just focus on your research and not be distracted by teaching and administrative stuff. And so your colleagues really pay attention when one of their colleagues gets some big, flashy grant, and so I actually think that\'s a cheap way to shift the Overton window. Like take the top 10 or 20 math and computer science departments, and give one person in each department a giant grant\-- giant by academic standards, couple million dollars, so it\'s not actually much. That will really convince people that, \"Wow, long-term alignment is a serious field where you can get serious funding.\" So yeah, that would be my recommendation to funders. That\'s a pretty self-interested recommendation, because I intend to apply for a grant soon. But yeah, I think that would help. Let\'s see, did I answer your question?

\*\*0:37:26.1 Vael:\*\* I think so, yeah. What happens if some of these departments don\'t have anyone interested in working on long-term alignment?

\*\*0:37:34.6 Interviewee:\*\* Yeah, that\'s hard. Like at \[university\], I spent several months probing my colleagues for anyone who\'s interested. I didn\'t find anyone in the computer science Department, which was disappointing, because \[university\] has a great computer science department. I do think if you look more broadly in several departments you are likely to find one or a few people\... You could see already there\'s these big institutes, you have one now at Stanford, there\'s one at Berkeley, Cambridge, Oxford, and so on. So that\'s evidence that there\'re already a few people. And people talk to colleagues around the world, so it doesn\'t matter if there\'s nobody at school X, you fund the people that are interested. But the key is that they might not have a track record in alignment, like I\'m in this situation where I have no track record, my track record is in pure math. So somebody has to take a little bit of a leap and say, \"Well, I don\'t know if \[interviewee name\] will be able to produce any good alignment research, but there\'s a good chance because he\'s good at proving theorems, so let me give him a couple of million dollars and see what happens.\" That is a big leap and it might fail, but it\'s just like any venture funding, a few of your big leaps will be very successful and that\'s enough.

\*\*0:39:16.8 Vael:\*\* Yep, that makes sense. Yeah, I think \[name\] at \[other university\] is aiming to make an institute at \[other university\] as well, which is cool.

\*\*0:39:25.2 Interviewee:\*\* That\'s great. I\'ve been talking to my dean about doing this at \[university\] and he likes the idea but he is not really aware of how to fund it, and I\'m telling him there\'s actually a lot of funding for this stuff, but I don\'t personally know the funders.

\*\*0:39:44.8 Vael:\*\* Yeah, I think getting in contact with \[funding org\] seems like the thing to do.

\*\*0:39:49.2 Interviewee:\*\* Yep, good.

\*\*0:39:50.2 Vael:\*\* Yep. \[Name\] is one of the people in charge there. Great, so how has this interview been for you and why did you choose to jump on it? That\'s my last question.

\*\*0:40:03.1 Interviewee:\*\* Oh, it\'s been fun. I spent a while just thinking alone about some alignment stuff because I had no colleagues to talk to, so it\'s always great to find someone who likes to talk about these issues.

\*\*0:40:24.6 Vael:\*\* Did you know that I was already interested in long-term alignment?

\*\*0:40:28.2 Interviewee:\*\* I think I saw your name at the\... Did you participate in the SERI Conference?

\*\*0:40:33.2 Vael:\*\* I did.

\*\*0:40:34.9 Interviewee:\*\* Yeah, so I saw your name there and so I was sort of aware of your name but I didn\'t know anything about your interests.

\[\...some further discussion, mostly about the interviews\...\]

\*\*0:42:22.4 Interviewee:\*\* Okay. Cool, Vael. I should jump on another call, but it was great to chat and yeah, feel free to follow up if you want.

\*\*0:42:32.1 Vael:\*\* Will do. Thanks so much.

\*\*0:42:33.7 Interviewee:\*\* Okay. Take care.


Interview with bj9ne

\*\*Interview with bj9ne, on 3/24/22\*\*

\[Note: This transcript has been less edited than other due to language barriers. The interviewee is also younger than typical interviewees.\]

\*\*0:00:00.0 Vael:\*\* Alright. My first question is, can you tell me about what area of AI you work on in a few sentences?

\[Discusses research in detail\]

\*\*0:03:31.9 Vael:\*\* Got it. Great. Yeah, so thinking about the future, so thinking about what will happen in the future, maybe you think AI is important or maybe you don\'t think AI is important, but people talk about the ability to generate an AI that is a very capable general system, so one can imagine that in 2012, we had AlexNet and the deep learning revolution, and here we are 10 years later, and we\'ve got systems like GPT-3, which have a lot of different capabilities that you wouldn\'t expect it to, like it can do some text generation and language translation and coding and math, and one might expect that if we continue pouring in all of the investment with nations competing and companies competing and algorithmic improvements or software improvements and hardware improvements, that eventually we might reach a very powerful system that could, for example, replace all current human jobs, and we could have CEO AIs, and we could have scientist AIs, so do you think this will ever happen, and if so, when?

\*\*0:04:43.6 Interviewee:\*\* Honestly speaking, I don\'t think this is realistic in the future, like 10 years or 20 years. So basically, I agree that AI will replace more jobs, but in my mind, this replacement, I don\'t say that\... For example, in a factory, there were 100 employees, and in the future, we may replace that 100 employees with AI, and five employees, but this five may be advanced engineers, rather than regular people. Because I have gone to some factories that manufacture something we use in our daily life, and my impression is that there were very, very few people, but the factory is very large, and mostly those people are just sitting in a room with air-conditioning. But those robotics, which is guided by AI and processing maybe those request from customers very efficiently, but they tell me that, for example, for some security issues, they need to watch those machines, so that that won\'t happen something unexpected. So also as an example, cars, and automated cars, auto drive is very, very \[inaudible\] in these years, but this is a issue that there are a lot of security issues.

\*\*0:06:39.3 Interviewee:\*\* So basically, just, I think the manufacturer and auto drive has a similarity, that is in many scenarios, we do want to guarantee a high level of security, but maybe for scenarios like recommendations, so making mistakes, that doesn\'t matter, and also I think one issue is very important, that is interpretability. Because for many, many models we designed or we have designed, and maybe less, there are many popular or famous papers in top conferences, but these models, many of these models lack interpretability.

\*\*0:07:30.0 Interviewee:\*\* So this is not a\... Sometimes it is unacceptable to deploy these models into some scenarios that requires us to \[inaudible\] something, especially in the scenarios like in hospitals, but we can\'t accept this because sometimes, an error is\... For example, when we do salaries, and some errors is not acceptable. So I think this is a very good question, so for my mind, I think this should be discussed by classifications, so for scenarios like recommendations, where errors are acceptable, then AI will replace more and more work of human, and this work was done by maybe some data scientists, because with our models become more and more intelligent and our systems become more and more efficient, also our data become more and more efficient, so the work of data scientists will be reduced by a great margin. And I expect this growth to\... I think this is sustainable.

\*\*0:08:49.9 Vael:\*\* Yeah\--

\*\*0:08:50.2 Interviewee:\*\* For example, in probably one or two decades, this will keep growing, but for some scenarios like auto drive, then on my mind, maybe this is controversial, but I don\'t think that auto drive is really something so realistic that we can expect like L4 or L5 auto drive to be realistic, to be used by us in a decade.

\*\*0:09:20.0 Vael:\*\* Yes\--

\*\*0:09:20.9 Interviewee:\*\* So in these scenarios, I think AI is a very important system for us. It can reduce our work, reduce our burden, but human is needed.

\*\*0:09:30.0 Vael:\*\* Yeah. Great. Cool, so my question is, do you think we\'ll ever get very, very capable AI, regardless of whether we deploy it or not? Do you ever think that we\'ll have systems that could be a CEO AI or a scientist AI, and if so, when? Like it could be like 50 years from now, it could be like 1,000 years from now, when do you think that would happen?

\*\*0:10:00.3 Interviewee:\*\* Oh, this is a tough question. Let\'s see. At least, based on the method we have taken, I think the way we develop AI now will not lead us to that future, but maybe the human will find some different ways to develop AI. But through my mind, I guess is to first we talk about maybe 50 years or a century, I think that\'s not very possible, but in the future, this may be a question about some knowledge about our brains. So maybe human at the moment, we are not\... I mean the investigation or research into our brain is not very clear.

\*\*0:11:01.0 Interviewee:\*\* So it\'s quite hard to imagine if the machine can evolve into some stages, that is the machine can be as complex, as powerful like human brains. So maybe in a century or even in two centuries, I tend not to believe that this will happen. but in the very long run, it\'s very hard to tell because I think that the way we think about something and the way the machine difference or train themselves are totally different. They work in different ways. For example, machines may require very large amount of data to find some internal principles. But with human, we are very good at generalizations, so I think for this point, if then maybe we will achieve that after 1,000 years, but I\'m not very optimistic about this. I tend not to accept this, but I can\'t deny that entirely.

\*\*0:12:13.3 Vael:\*\* Yeah. There are some people who argue that the current way of scaling may eventually produce AI, like very advanced AI, like OpenAI and DeepMind. No? You don\'t think so? Cool.

\*\*0:12:29.1 Interviewee:\*\* I don\'t think so, because based on our current trend, the development of the models comes from the development of computing power. But computing power, essentially that comes from the increasing density of changes account. So according to a research by Princeton, we know that for computer, computing power per transistor hasn\'t changed a lot during the last few decades, and also, you know the end of dinosaurs and maybe in the future, we\'ll see the end of Moore\'s Law. And I know, I have solid GPT, GPT 2022 \[inaudible\] Nvidia has powered the AI and speed up by a million x in the last decade. But I don\'t think that this development is sustainable because you see, over here I spent GPT-3 with maybe \$4 million, if I don\'t get it wrong. This scale is not good because not only we need a larger cluster but also we need more money and more time to train that model. So I remember that OpenAI has forged some arcs\[?\] in this way, but we are not able to train that again, and I have some\... There are some articles about the scaling of deep-learning models based on the parameter count of the models and they found that, I think maybe before 2022\... No, before 2020, the growth is 75X per 18 months, but after that the growth has slowed down, has greatly slowed down because, lots of issues, like the scaling of GPU or the scaling of clusters to deploy those models in very large cluster that is not narrow anything.

\*\*0:14:53.0 Interviewee:\*\* So I think so, this is partly\... This is the main reason why I shift my interest from data to system, because I believe that AI has a bright future, but to make that future brighter, we need to make our system run faster. For example, can we achieve the same result using less parameters or using less power so that we can\... We can\'t have these hardware resources but can we make better use of it? So I think the future of AI largely depends on system people. That is, can we improve the system for AI? I mean, when I talk about system, I actually, from the perspective \[inaudible\] AI, I talk about three things, the hardware, software system, and the platforms. In my mind, those models are just like the top layer, that is the application. I think the lower three levels are the key to the future success of AI, but indeed I think AI is\... And also \[inaudible\] so, I think the future of AI also means we need more application scenarios, like dark\[?\] intervention or something like that, and also robotics, and I think this is very promising and it can bring us a lot of fortune or something like that.

\*\*0:16:25.8 Vael:\*\* Yeah, great. I\'m concerned about when we eventually\... I think that we may get AI, AGI a little faster than that. And I\'m concerned that the optimization functions that we\'ll be optimizing are not going to reflect human values well because humans aren\'t able to put their values and goals and preferences perfectly into AI, and I worry that problem will get solved less fast than the problem of how we get more and more capable AI. What do you think of that?

\*\*0:17:00.6 Interviewee:\*\* When you talk about values, it\'s something. Yeah, so although it\'s not a technical issue, but this is really what the technicians should care about. So this is a very important issue in open drives\[?\], when the cars make you a hero\[?\], a person first\[?\]. So maybe we should find some method or some way to plug our rules, plug our values to guide the models, guide AI, but I think it\'s also an issue of interpretability because now we don\'t have\... Sometimes we have no idea why some models can work so well. For example, when\... Because I have worked on GNN, and this year GCN and its variants, it\'s very popular, but many of that, when the researcher comes up with that model, they don\'t know why this is good.

\*\*0:18:12.5 Interviewee:\*\* So maybe similarly it\'s quite hard to guide AI to follow the rules, so I think this is also an important issue and important obstacle for the applications of AI. For example, we cannot put some non-CV skills to recognize human faces because sometimes it may violate the law. Yeah, I think this is an important issues, but will this stop AI? I think for my mind, this may be an obstacle for AI in some scenarios, but in many scenarios, this is not a\... This will generate some issues for us to think about, but in the end I think we will deal with this. For example, some people may use federated learning to deal with privacy and there are some techniques to deal with these issues. So yeah, I think we should put more emphasis on the values and the rules or even the assets about AI so that this community will grow faster. Yes, this is an important aspect. Although I don\'t put much emphasis on this.

\*\*0:19:44.1 Vael:\*\* Why not?

\*\*0:19:49.8 Interviewee:\*\* That\'s because, I guess, because my research and my internship mainly focus on recommendation and there is not much issues about this except for privacy, and because when we got those data, we don\'t know the meaning of data. When I get a little data intention, I just send out important numbers, or maybe sometimes this is a one \[inaudible\] issue, and I don\'t know much about that, so let\'s say that these privacy issues has been\... Maybe this has been dealt with by those data scientists, not by people like us, so this is because of my research interest, but I think for those people who do\...

\*\*0:20:34.1 Interviewee:\*\* Yes, I have taken some courses about AI and then teachers say that they have developed some robotics to help the elderly. But let\'s say that sometimes you cannot use a camera because using a camera will generate some privacy issues, so maybe sometimes we can just use something to catch its audio rather than video or something like this, but because\... I guess that\'s most of students\... Most of my schoolmates don\'t put\... Most of my classmates haven\'t paid much attention on these issues, but this is a very\... I think this issue will become more and more important in the future if we want to generate AI to more and more scenarios, so thank you for raising these points. I will think more about it in the future.

\*\*0:21:37.0 Vael:\*\* Yeah. I\'m happy to send you resources. One extra thing, one other additional thing. So I think probably, I think by default the systems we train will not do what we intend them to do, and instead will do what we tell them to do, so we\'ll have trouble putting all of our preferences and goals into mathematical formulations that we can optimize over. I think this will get even harder as time goes on. I\'m not sure if this is true, but I think it might get harder as the AI is optimizing over more and more complex things, state spaces.

\*\*0:22:13.6 Interviewee:\*\* So you mean that because in the future we will have more and more requirements, and that\'s so\...

\*\*0:22:22.6 Vael:\*\* No, no, the AI will just be operating under larger state spaces, so I will be like, \"Now I want you to be a CEO AI,\" or, \"Now I want you to be a manager AI.\"

\*\*0:22:34.3 Interviewee:\*\* Oh, did you say that we need to encode those requirements into optimization function so that AI will operate like what we want them to do? Did I get it wrong? Oh, that\'s a quite good question. I have discussed that with my roommates. Yeah, so yes, it is the optimization, the loss function that guides the model to do something that we want, and sometimes it\'s hard to find an appropriate function, especially for newbies. Sometimes we chose a wrong loss function, and the way the model is totally unusable.

\*\*0:23:16.8 Vael:\*\* Yeah, and I have one more worry about that scenario, so\...

\*\*0:23:20.5 Interviewee:\*\* Yeah, yeah, yeah. I think this is also a very important issues, and I\'m not very optimistic about this because it\'s really hard. And lots of things, because like a CEO AI, we not only need to care about the revenue of this company, but also learn maybe the reputation, and we may also want them to abide by the laws, and maybe when there is new business and we want to inject new rules into that loss function core business. Great.

\*\*0:24:05.6 Vael:\*\* Yeah, and I have one more twist. Alright, so imagine that we had a CEO AI, and it takes human feedback because we\'ve decided that that\'s a good thing for it to do, and it needs to write a memo to the humans so that they make sure that its decision is okay, and the AI is trying to optimize a goal. It\'s trying to get profit and trying not to hurt people and stuff, and it notices that when it writes this\... That when it writes, sometimes humans will shut it down. And it doesn\'t want that to happen because if humans shut it down, then it can\'t achieve its goal, so it may lie on this memo or omit or not include information on this memo, such that it is more likely to be able to pursue its goal. So then we may have an AI that is trying to keep itself alive, not because that was programmed into it, but because it is an agent optimizing a goal, and anything that interfered with its goal is like not achieving the goal. What do you think of this?

\*\*0:25:16.3 Interviewee:\*\* Oh, very good question, but it\'s very hard. It basically mean that we need to establish enough rules so that when\... Sometimes it\'s very hard to come up with some common cases that AI may\... Yes, it is optimizing towards its goal, but there may be something that we want it to do, so maybe we need to have a mechanism so that we can switch from the AI mode, from manual mode, that we can take control of AI or take control of\... For example, the company in the last second was guided by an AI, and for the next second, we want to guide in, we want to lead the company manually, so I think if we ask \[inaudible\] enough and we may establish maybe a thorough mechanism so that we can guarantee that, it is possible to take control back, and the AI will not lie to us. And yes, theoretically, this is possible and this should be the case. But correctly speaking, I think this can be done, but it\'s quite hard to estimate the cost. The engineering cost for us to make such a AI that is complete enough, that is secure enough to help us achieve the goal. So maybe in this point, if we want to do something like recommendations, we can be very radical, we can develop or deploy radical models. But maybe in the scenarios like a CEO AI, I think we should be conservative because some internal principles of AIs are not known entirely by the humans, so I think sometimes we need to be conservative to prevent some bad things from happening.

\*\*0:27:45.1 Vael:\*\* Yeah. I think that the problem\... I think it\'s not just an engineering problem. I think it\'s also a research problem, where people don\'t know how to construct optimization functions such that AI will be responsive to humans and won\'t be incentivizing against humans. And there\'s a population of people who are working on this, who are worried that if we have very intelligent AIs that are optimizing against us, then humans may be wiped out, and so you really don\'t want\... You really wanna make sure the loss function is such that it isn\'t optimizing against us. Have you heard of the AI safety community or AI alignment community?

\*\*0:28:25.9 Interviewee:\*\* No. I don\'t know much about that, but let me say the scenarios you have mentioned, that AI may optimize against us and even wipe out the human, I have seen some films, some movies about this, and yes, this is possible if we are too careless, I think this is possible.

\*\*0:28:48.8 Vael:\*\* Yeah.

\*\*0:28:49.0 Interviewee:\*\* But at least it\'s right at\... I haven\'t paid much attention on these issues, but I think this is an important question.

\*\*0:28:58.9 Vael:\*\* Yeah, I personally think that advanced AI may happen in the next 50 years, and this is just from looking at some surveys of experts. I\'m very unsure about it. But if that does happen, then I think that currently we have groups like\... China and the US are going to be competing, I expect, and we\'ll have lots of different corporations competing, and maybe DeepMind and OpenAI are not competing, but maybe they\'re just going really hard for the goal. And I worry that we\'re not going to spend enough effort on safety and we\'re going to be spending much more effort on trying to make the AI do the cool things, and if the safety problems are hard, then we may end up with a very unsafe, powerful AI.

\*\*0:29:55.2 Interviewee:\*\* Or this then may come up with the competition on nuclear weapons.

\*\*0:30:01.1 Vael:\*\* Yeah.

\*\*0:30:01.4 Interviewee:\*\* How likely \[inaudible\] just like nuclear weapons. Yes, the power of AI in the future may be something like the power of nuclear weapons, that is quite hard to control if the real war or something\... Maybe not so severe like a war, but it is possible, so maybe I think we need some international association about the use of AI. But both in China and the US, the government has established more and more rules about, for example, privacy, security, and what you can do and what you can\'t do. So yes, if we don\'t place enough emphasis on this, this may be a question, but well, most\--

\*\*0:30:51.3 Vael:\*\* I think they\'re placing emphasis on issues like privacy and fairness, but they\'re not placing emphasis on trying to develop loss functions that do what humans want. I think that is a more different type of research that is not being invested in.

\*\*0:31:08.8 Interviewee:\*\* Yes, you\'re right. So maybe the community should do more things about this because you can\'t count on those people in the government to realize that this is really a case, they don\'t know much about this, so yes, yes, the community should, like the conference or a workshop, we should talk more about this, yes, before that is too late. Yes, I agree. Before that is too late, or everything will be a disaster.

\*\*0:31:38.5 Vael:\*\* Amazing. Yeah, so there is actually a community of people working on this. I don\'t actually know\... I know fewer of the people who are working on it in \[country\], although I do think there are people working on it. I\'m curious, what would cause you to work on these sort of issues if you felt like it?

\*\*0:32:00.0 Interviewee:\*\* You say issues that we have just mentioned?

\*\*0:32:01.9 Vael:\*\* Yeah, long-term. Well, like trying to make the AI aligned with a human, trying to make sure that the things that the AI does is what the humans wanted to do, long-term issues from AI, anything like that.

\*\*0:32:16.4 Interviewee:\*\* So I try to, although from the bottom of my heart, I think I believe that this is an important issue, frankly speaking as a student, a student researcher, or as an engineer, I don\'t have much resources about this. So I think this is the most important issue why most of my schoolmates just like improving the models and then don\'t care about if the AI may optimize against us because\... I know this sound not good, but most of student just care about, \"So if I can graduate with a PhD degree or so.\" Yeah, so maybe\... I think for me, maybe\... because I guess I will be an engineer in the future, so maybe if I have enough resources or I have enough influence in the community, I\'m willing to spend my time on this, but if I just a low-level coder and I don\'t have much power to ask my superintendent that we should place more emphasis on this, they just take\...

\*\*0:33:35.2 Interviewee:\*\* For example, if I intend and they just say, \"Oh, this model, the accuracy is good enough, but the speed is not, so optimize the model so that it can run fast enough as maybe the customers\' requirement.\" So yeah, this is basically the entire ecosystem, both in the academia and the industry that force the researchers and the employees in the company that they will not put much emphasis on this, and also, most of the time they just focus on short-term issues, short-term profits, or in the universities, student just care about, \"Oh, can I have some publications so that\... \" I really don\'t know any of my class schoolmates who have publication on these issues.

\*\*0:34:32.4 Interviewee:\*\* So I don\'t know whether there are lots of the researchers who cares about this and will spend maybe several months on these issues where they will have some publications about this. I know the top conference has asked about some ethical issues, but yeah, we really don\'t pay enough attention on this. This is a very good point. I think we need some more incentives on the future of AI. For example, like environmental issues, some factories, they will not care about environmental issues if the government doesn\'t force them to do so. For example, now we have the trade on the carbon dioxide budget. That is, the government tell the factories that you shouldn\'t emit more carbon dioxide than maybe more than this threshold or you will be fined. Maybe we need some, yes, we need some incentives to force us to think about these issues or otherwise I think this is not optimistic because not many people will be guided to do this because maybe those on the other levels they don\'t care about this.

\*\*0:36:05.7 Vael:\*\* Yeah, yeah. That seems right. Yep. It does seem like it\'s not currently as popular in the top journals as it could be, seems like a pretty small community right now. I will look around to see if I can find any researchers in \[country\] who are working on this sort of thing, because I know a bunch of them in \[country\] and I know some of them in \[country\], but not as many in \[country\], and I\'ll send you some resources if you\'re interested. There is a group of people who pay attention to this a lot, and they\'re called the Effective Altruism community, and right now, one of the things they care about is trying to make sure that existential risks don\'t happen so that humans are okay. Some other things they\'re worried about are pandemics, nuclear stuff, climate change, stuff like this, and also many other things. Interesting. Alright, cool. I think my last question is, have you changed your mind on anything during this interview, and how was this interview for you?

\*\*0:37:16.9 Interviewee:\*\* Oh. Yeah, I think maybe the greatest change to my mind is, you say that if we want a CEO AI, we need to, maybe we need to encode those requirements into the optimization function, and maybe someday an advanced AI will optimize against us. Yes, basically, in the past, I think this may be an ethical issue, and now I\'ve realized that it\'s both an ethical and a social, as well as a technical issues and we have\...

\*\*0:37:53.4 Vael:\*\* Yeah.

\*\*0:37:55.3 Interviewee:\*\* Yes. I know we haven\'t paid enough emphasis on this, but yeah, now I think that it\'s time for us to do more things, and this is a very wonderful, wonderful idea for me to think about. Thank you very much.

\*\*0:38:14.3 Vael:\*\* Yeah. Well, I mean, it\'s super cool that you\'re interested in this, so I\'m very enthused by you being like, \"This does seem like a problem, this does seem like a technical\... \" Yeah, I\'m very excited about that. Cool. Alright. Well, I will send you some resources then and I\'ll see if I can find anyone who is doing anything like this and send anything I find your way. But thank you so much, and feel free to reach out if you have any questions or if there\'s anything I can help you with.

\*\*0:38:41.8 Interviewee:\*\* Oh, okay. Also, thank you. So no, I don\'t have much questions. I will read more about this in the future. I think this is very important and also very interesting. Thank you. Thank you.

\*\*0:38:54.8 Vael:\*\* Yeah, I\'ll send you some resources. Alright, email you soon. Bye.

\*\*0:39:01.3 Interviewee:\*\* Okay.


Interview with w5cb5

\*\*Interview with w5cb5, on 3/18/22\*\*
====================================

\*\*0:00:02.2 Vael:\*\* Alright, so my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:08.5 Interviewee:\*\* I worked in \[subfield\] originally, but I guess I branched out more broadly into AI research, because I\'m \[high-level research role\] now at an AI company.

\*\*0:00:19.9 Vael:\*\* Great, yeah. And then what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:00:28.3 Interviewee:\*\* So I think, yeah, the world is going to change quite a lot with AI technology, and I think mostly in good ways, just because we\'re going to empower people with this technology. And it\'s going to be empowering I think in similar ways to the Internet, where people can do faster search, they have an assistant who can help them with all kinds of stuff. They have friends who maybe are not real, and all kinds of ways to make people happier, I think, or more efficient, or to give them time back and that sort of stuff. But obviously, there are also risks and the main risks are, I think that the field is too dominated by tech bros from Silicon Valley, so I guess I fall under that in a way. And so I think that\'s a real problem, so we need to take democratization of the technology much more seriously, that\'s also what my company is doing. And I think if we think about the ethical implications of our technology from first principles, and if we make them first-class citizens rather than just treating them as an afterthought, where you submit your paper and then, \"Oh, I also need to write a broader impact statement,\" but if you take that very seriously from the beginning as a core principle of your organization, then I think you can do much better research in a much more responsible way.

\*\*0:01:56.5 Vael:\*\* Interesting. Alright, so that was the question of \"what are you most excited about and what are you most worried about in AI\", okay. I heard\-- Lots of things they can go, lots of places they can go, lots of directions they can go, but you\'re worried about domination from specific areas and then people not caring about\... ethics enough? or---

\*\*0:02:14.6 Interviewee:\*\* Yeah, so misuse of technology. Do you want me to give you concrete examples? So I think very often, the technology that we develop, even if it\'s meant for benevolent purposes, can also be re-applied for not so benevolent purposes. And so like speech recognition or face recognition, things like that, you have to just be very careful with how you treat this technology. So that\'s why I think if people take responsible AI seriously from the beginning, that that is a good thing too.

\*\*0:02:53.0 Vael:\*\* Interesting. So you think if people incorporate responsible AI from the beginning of the process, then there will be less risk of misuse by any agent in the future?

\*\*0:03:04.5 Interviewee:\*\* Yeah, yeah. So you mentioned your IRB, so for a lot of technological research happening in industry, there is no real IRB. Some companies have sort of IRBs but most of them are so commercial and so driven by money in the end. And I think maybe we need an independent AI IRB for the broader research community, where anybody can go there and have somebody look at the potential applications of their work.

\*\*0:03:39.6 Vael:\*\* I see, cool. And then just having that sort of mindset seems good, in addition to the object-level effects. Alright. Makes sense. So focusing on future AI, putting on a science fiction forecasting hat, say we are 50 years, 50 plus years into the future. So at least 50 years into the future, what does that future look like?

\*\*0:04:00.3 Interviewee:\*\* At least 50 years in the future. So I still don\'t think we will have AGI, and that\'s I guess, I\'m probably unusual in the field because I think a lot of my colleagues would disagree, especially if they\'re at OpenAI or DeepMind because they think that it\'s like two years away. (Vael: \"Two years, huh!\") Yeah, well it depends on who you ask, they have some crazy people. \[chuckle\] I think in the next decade, we\'re going to realize what the limitations are of our current technology. I think what we\'ve been doing now has been very efficient in terms of scaling with data and scaling with compute, but it\'s very likely that we\'re just going to need entirely new algorithms that just require pure scientific breakthroughs. And so I don\'t think there\'s going to be another AI winter, but I do think that things are going to cool down a little bit again, because right now it\'s just been super hyped up. For good reason too, because we are really making really great progress. But there is still things that we really don\'t know how to do, so we have language models and they can do things and they\'re amazing, but we don\'t know how to make the language model do what we want it to do. So we\'re all just sort of hacking it a little bit, but it\'s not really anywhere close to being like a proper assistant, for example, who actually understands what you\'re saying, who actually understands the world. I think where we want to be 50 years from now is where we have machines who understand the world in the same way that humans understand it, so maybe something like Neuralink. So if I\'m being very futuristic, connecting AI to human brains and human perception of reality, that could be a way to get AI to have a much richer understanding of the world in the same way that humans understand it. So like dolphins are also very intelligent, but they also don\'t understand humans and they are not very useful assistants, right? I don\'t know if you\'ve ever had any dolphin assistant. So it\'s not really bad intelligence, it\'s specifically about human intelligence that makes AI potentially useful for us, and so that\'s something that I think is often overlooked.

\*\*0:06:26.9 Vael:\*\* So it sounds like, so you\'re thinking about when AGI will happen. And you said that you don\'t think we\'re gonna hit some sort of ceiling or slow down on the current deep learning paradigm or just like keep on scaling\--

\*\*0:06:39.6 Interviewee:\*\* Yeah, it\'s going to be asymptotic, and at some point, we\'re just going to hit the limits of what we can do with scaling data and scaling compute. And in order to get the next leap to real AGI I think we just need radically different ideas.

\*\*0:06:55.1 Vael:\*\* Yeah, when do you think we\'re going to\-- what kind of systems do you think we\'re going to have when we cap out on the current scaling paradigm?

\*\*0:07:02.0 Interviewee:\*\* Well, I think like the ones we have now, but yeah, in 50 years, I don\'t know. But in like 5 to 10 years, it will just be much bigger versions of this. And so what we have seen is that if you scale these systems, they generalize much better. If that keeps happening, then we would just have much better versions of what we have now. But still it\'s a language model that doesn\'t understand the world, and so still it\'s the component that is very limited in seeing only the training data that is in images on the internet, which is not all of the images that we have in the world, right? So I think the real problem is data, not so much scaling the compute.

\*\*0:07:49.7 Vael:\*\* What if we had a system that has cameras and can process auditory stuff that is happening all around it or something and it\'s not just using internet data, do you think that would eventually have enough data?

\*\*0:08:03.3 Interviewee:\*\* Yeah, so that\'s what I was just saying. If you have something that\'s embodied in the world in the same way as a human and where humans treat it as another human, sort of like cyborg style, things like that, that\'s a good way to get lots of very high quality data in the same way that humans get it. What are they called? Androids, right?

\*\*0:08:24.9 Vael:\*\* Yeah.

\*\*0:08:25.3 Interviewee:\*\* So if we actually had android robots walking around and being raised by humans and then we figured out how the learning algorithms would work in those settings, then you would get something that is very close to human intelligence. A good example I always like to use is the smell of coffee. So I know that you know what coffee smells like, but can you describe it to me in one sentence?

\*\*0:08:54.2 Vael:\*\* Probably not, no.

\*\*0:08:55.7 Interviewee:\*\* You can\'t, right? But the same goes for the taste of banana or things like that. I know that you know, so I\'ve never had to express this in words. So this is one of the fundamental parts of your brain; smell and taste are even older than sight and hearing. And so there\'s a lot of stuff happening in your brain that is just taken for granted. You can call this common sense or whatever you want, but it\'s like an evolutionary prior that all humans share with each other, and so that prior governs a lot of our behavior and a lot of our communication. So if you want machines to learn language but they don\'t have that prior, it becomes really, really hard for them to really understand what we\'re saying, right?

\*\*0:09:38.7 Vael:\*\* Yeah. I think when I think about AGI, I think about AGI that can do\-- or, just, generalizable systems that can do things that humans want them to do. So imagine we have like a CEO AI or a scientist AI. I don\'t think I need my CEO or scientist AI enough to know what coffee smells like per se, but I do need it to be able to like break down experiments and think kind of creative thoughts and figure out things.

\*\*0:09:58.7 Interviewee:\*\* Yeah, but I think what I\'m saying is that if they don\'t know what coffee smells like, that\'s just one example, but there are millions of these things that are just things we take for granted, that we don\'t really talk about. And so this will not be born out in the data in any way, so that means that a lot of the underlying assumptions are never really in the data, right? They\'re in our behavior, and so for an AI to pick up on those is going to be very difficult.

\*\*0:10:27.6 Vael:\*\* What if there were cameras everywhere, and it got to record everyone and process those?

\*\*0:10:32.3 Interviewee:\*\* Yeah, maybe. So the real question is, if you just throw infinite data at it, then will it work with current machine learning algorithms? Is I guess what you\'re asking, right? And so I don\'t know. I mean, I know that our learning algorithm is very different from a neural net, but I think if you look at it from a mathematical perspective, then gradient descent is probably more efficient than Hebbian learning anyway. So mathematically, it\'s definitely possible that if you have infinite data and infinite compute, then you can get something really amazing. Sure, we are the proof of that, right? So whether that also immediately makes it useful for us is a different question, I think.

\*\*0:11:20.8 Vael:\*\* Interesting. Yeah, I think I\'m trying to probe \"do we need something like embodied AI in order to get AGI\" or something. And then your last comment was like, whether that makes it useful for us. I\'m like, well, presumably we\'re going to\... feeding it a lot of data lets it do grounding, so like relationships between language and what actually exists in the world and how physics works. But presumably, we\'re going to be training them to do what we want, right? So that it will be useful to us?

\*\*0:11:43.5 Interviewee:\*\* Well, it depends, right? Can we do that? Probably the way they will learn this stuff is through self-supervised learning, not through us supervising them. We don\'t know how to specify reward signals and things like that anyway. I\'m not sure, if we actually are able to train up these huge systems that are actually intelligent through self-supervised learning, if they are then going to listen to us, right? Why would they?

\*\*0:12:15.2 Vael:\*\* Right. Okay, cool. Yeah, so this kind of leads right into my next question here. So imagine we\'re in the future and we\'ve got some AGIs and we\'ve got a CEO AI, and I\'m like, \"Okay, CEO AI, I want you to maximize profits and not run out of money and not try to exploit people and try to avoid side effects,\" and it seems like this would currently be extremely challenging for many reasons. But one is that we\'re not very good at taking human values and putting them\-- and like goals and preferences\-- and putting them in mathematical formulations that AI can currently work. And I worry that this is gonna happen in the future as well. So the question is: what do you think of the argument, \"Highly intelligence systems will fail to optimize exactly what their designers intended them to and this is dangerous\"?

\*\*0:12:53 Interviewee:\*\* Well, yeah. I agree with that. I don\'t think\... I think there are two separate questions here. So one you\'re asking about is the paperclip maximizer argument from Nick Bostrom. So like if you have a system and you tell it like \"you need to make as many paperclips as you possibly can\" then it\'s going to like destroy the earth to make as many paperclips as possible.

\*\*0:13:15 Vael:\*\* Well that would be doing maybe\-- oh, I see. Not quite what I intended. Yeah, all right.

\*\*0:13:19.8 Interviewee:\*\* Yeah, so\-- okay, so if that\'s not what the underlying question was, then\... We don\'t really\... I also think that we are\... some of us are fooling ourselves into believing that we know everything as humans and I think human values are changing all the time. I don\'t think we can capture correct human values. I don\'t think there is an absolute moral truth that we should all adhere to. I think that just morality itself is a very cultural concept. But I\'m \[interested in\] philosophy, so I\'m a bit different from most AI researchers, I guess. So I think that we could try to encode some very basic principles, so this is like Asimov\'s laws and things like that, but I don\'t think we can really go much further than that. And I think even in those cases, like you said, we don\'t know how to mathematically encode them in a way where you enforce whatever this dynamical system is that you\'re training, so a neural net, but then probably more complicated than the current neural nets\-- how do we impose a particular set of values? I don\'t think we know how to do that. I don\'t think there\'s a mathematical way to do that either actually, because it\'s all \[inaudible\]\--

\*\*0:14:44.7 Vael:\*\* Yeah, do you think we are eventually going to be able to?

\*\*0:14:50.0 Interviewee:\*\* So I think if you ask Yann LeCun or someone like that, he would say that probably, if we ever get to systems of this sort of level of intelligence, then they would be benevolent, because they\'re very smart and able to sort of understand how weak humans are.

\*\*0:15:09.4 Vael:\*\* Interesting. Yeah. So when I hear that argument, I\'m like, okay, it seems like Yann LeCun thinks that as you get more intelligent, you have morals that are very similar to humans, and this just kind of comes\--

\*\*0:15:21.7 Interviewee:\*\* No, not necessarily. No, but just better morals, right? So I think that the argument is sort of that if you look at human progress, then we\'ve also been getting better and better moral systems and a better understanding of what human values really matter. And like 100 years from now, probably everybody\'s gonna look back at us and say, \"They were eating meat. They were killing all these animals.\" So we are on the path of enlightenment. I don\'t know if I agree with this, but that\'s one way of saying it. And so a sign of an organism or a culture becoming more and more enlightened is also that you become more and more benevolent I think for others, but maybe that\'s a bit of a naive take.

\*\*0:16:05.9 Vael:\*\* Yeah. I think in my mind\-- certainly we have\-- well, actually, I don\'t know that we have the correlation that humans are getting smarter and also at the same rate, or, like\... Like humans are pretty smart. And we\'re getting better at IQ tests, but I don\'t know that we\'re vastly increasing our intelligence per se.

\*\*0:16:20.4 Interviewee:\*\* Yeah. That\'s for different reasons, right. Yeah.

\*\*0:16:24.9 Vael:\*\* Yeah. And meanwhile, we have, over\-- centuries, like not that many centuries, we\'ve been increasing our moral circle and putting in animals and people far away from us, etcetera. But I kind of think of the axes of intelligence and morality as kind of orthogonal, where if we have a system that is getting much smarter, I don\'t expect it to have\... I expect kind of a lot of human morality runs from evolutionary pressures and also coordination difficulties, such that you need to be able to not kill people, otherwise the species is gonna go extinct. And you know, there\'s a bunch of stuff that are kind of built into humans that I wouldn\'t expect to happen just natively with intelligence; where intelligence, I would think of something like\... the ability to solve problems well, to make multi-step plans, to think in the future, to take out correlations and figure out predictions, and I don\'t expect that to naively correlate with---

\*\*0:17:19.9 Interviewee:\*\* Yeah, so I think that\'s a very narrow definition of intelligence, and so I don\'t know if that definition of intelligence you have, if that actually is the most useful kind of intelligence for humans. So I think that in our society there is this concept where intelligence just means like mathematical reasoning capabilities almost, right? (Vael: \"Yeah.\") And that is a very, very narrow definition, and most of our intelligence is not that, right? (Vael: \"Yes.\") So for regimes to be useful to us\... so I think what you\'re talking about is sort of like this good old-fashioned AI concept of intelligence, where you have symbolic reasoners, and you\'re like\... you\'re very good at very fast symbol manipulation. And like, \"This is what computers are for.\" So we should just have super smart computers who can do the stuff that we don\'t want to do or can\'t do. It\'s possible that our intelligence is a direct consequence, not of our mathematical reasoning capabilities, but of something else, of our cultural interactions. So I definitely think if humans were not a multi-agent society, that we would not be nearly as intelligent. So a lot of our intelligence comes from sharing knowledge and communicating knowledge and having to abstract knowledge so that you can convey it to other agents and that sort of stuff.

\*\*0:18:50.0 Vael:\*\* Cool. Yeah. So when I think about how I define intelligence, I\'m like, \"What is the thing I care about?\" The thing I care about is how we develop AI. And I\'m like, \"How are we gonna develop AI?\" We\'re gonna develop it so that it completes economic incentives. So we want robots that do tasks that humans don\'t want to do. We want computers\--

\*\*0:19:09.2 Interviewee:\*\* Yeah. But is that AI or is that just machine learning? We\'re trying to have a\... like input-output black box, and we want that black box to be as optimal as possible for making money or whatever the goal is, right? So that\'s also a worry I have, is that a lot of people are conflating these different concepts. So artificial intelligence\...yeah, it depends on how you define it. Some people think of it more as like AGI. If you ask Yann again and all the old school deep learners, they would say, it used to be that they were explicitly not doing AI. So AI is like Simon and Newell and all that sort of stuff, so like pure symbol manipulation, symbolic AI. And pattern recognition is not AI. And now, since deep learning became very popular, some of the people were like, \"Oh yeah, this is AI now,\" but they used to be machine learning and not AI. So one thing is just like this black box. It can be anything and we just want to have the best possible black box for our particular problem mapping X to Y. And this could be any kind of problem, it could be like image recognition or whatever. In some cases, you want to have a symbolic approach, in other cases, you want to have a learning approach, it sort of just depends. So it\'s just software. Right? But in one case, the software is well defined, and in the other case, it\'s a bit fuzzier.

\*\*0:20:37.9 Vael:\*\* Yeah. So this all kind of depends on your frame, of course. I think my frame, or the reason why I care, is I\'m like, I think machine learning, AI, I don\'t know, whatever this thing is where humans are pouring a lot of investment and effort into making software better, and by better I mean better able to accomplish tasks that we want it to do\-- I think that this will be\-- it is very powerful, it has affected society a lot already and it will continue to affect society a lot. Such that like 50 years out, I expect this to be\... Whatever we developed to be very important in how\... Affect just a lot of things.

\*\*0:21:10.8 Interviewee:\*\* But we\'re notoriously bad at predicting the future, right? So if you asked in the \'60s, people would say like, there\'s flying cars, and like we\'re living on Mars and all that stuff. And we\'re getting a bit closer, but we\'re still not there yet. But none of these people would have seen the internet coming. And so I think maybe the next version of the internet is going to be more AI driven. So that is a sort of\... first use case that I would see for AI, which is like a better internet.

\*\*0:21:50.0 Vael:\*\* Interesting. Yeah, I think kind of\... people will find whatever economic niches will get them a lot of profit, is sort of how I expect things to continue to go, given that that seems to be \... Given that society works kind of the same way, and people have a lot of time and energy and have the capability to invest in this stuff, we will continue to develop machine learning, AI software, etcetera, such that it\--

\*\*0:22:13.2 Interviewee:\*\* We\'ve been doing that for like 30 years or even more. From the Perceptron, Rosenblatt. We\'ve been already doing this and so it\'s not really a question of like AI taking over the world, it\'s software taking over the world, and AI in some cases is better than like rule-based software. But it\'s still software taking over the world.

\*\*0:22:35.8 Vael:\*\* Yeah, yeah, certainly. And then the current paradigm of like, gigantic neural nets, seems to be better at doing things that we want it to do. And so we\'re continuing on in that direction, and at some point, as you say, it becomes less able to do what we want it to do, given the amount of resources that we\'re pouring into it, like that ratio trades off. Okay\--

\*\*0:22:54.3 Interviewee:\*\* Yeah. So there\'s other trade offs too, right? So as you become bigger as a neural net, you also become a lot more inefficient. This is already the case for something like GPT-3; latency is a big problem. For us to be able to talk like this to a machine, if the machine has 100 trillion parameters, it\'s going to be way too slow. It\'s going to take, I don\'t know, 10 minutes to generate an answer to a simple question. So it\'s not only a tradeoff of\... Best does not just mean accuracy. Best also is like, how efficient are you? How fair are you? How robust are you? How much environmental impact do you have? All of these different sort of metrics that all matter for choosing what defines \"best\" for a system. I think this is something we need to improve a lot on as a community, where we stop thinking beyond this pure accuracy thing, which is like an academic concept, to an actual\... like how can we deploy these systems in a responsible way, where we think about all the possible metrics that matter for deployment. So we want to be at the Pareto frontier of like 10 different metrics, not just accuracy.

\*\*0:24:06.8 Vael:\*\* Cool. Alright, that makes sense. So still thinking ahead in the future, do you think we\'ll ever get something like a CEO AI?

\*\*0:24:14.0 Interviewee:\*\* So, if\-- so a CEO AGI or a CEO AI?

\*\*0:24:18.8 Vael:\*\* Um, some sort of software system that can do the things that a CEO can do.

\*\*0:24:25.6 Interviewee:\*\* No.

\*\*0:24:26.1 Vael:\*\* No. Okay.

\*\*0:24:28.6 Interviewee:\*\* So not before we get AGI. So I think that is an AI complete problem. But I do think we\'ll get a very good CEO AI assistant. \[inaudible\] \...real human. It\'s like a plane, right? So like a plane is flown by a pilot but it\'s really flown by a computer. So I think the same could be true for a company where the company has like, a CEO pilot whose job is also to inspire people and do all of the human soft skills. And they have an assistant who does a lot of measurement stuff and tries to give advice for like where the company should be headed and things like that.

\*\*0:25:05.1 Vael:\*\* Okay, awesome. And you do think that you could have a CEO AGI, it sounds like.

\*\*0:25:10.3 Interviewee:\*\* Yeah, but if you have an AGI, then we don\'t need CEOs anymore.

\*\*0:25:14.3 Vael:\*\* What happens when we get AGI?

\*\*0:25:16.9 Interviewee:\*\* All the humans die.

\*\*0:25:17.5 Vael:\*\* All the humans die. Okay! \[laughs\]

\*\*0:25:20.1 Interviewee:\*\* \[laughs\] So I think it depends. I think actually the most likely scenario, as I said, for AGI to come into existence is when humans merge with AI. And so I don\'t think that it\'s a bad thing for AGI to emerge. So if there is an AGI, then it will be a beautiful thing, and we will have made it as a society. So yeah, if that thing takes over, then that thing is going to be insane, it\'s going to take over the universe, and then we will be sort of like the cute little people who made it happen. So either we become very redundant very quickly or we sort of merge with AI into this new species kind of.

\*\*0:26:14.1 Vael:\*\* Interesting, okay. And you don\'t necessarily see a connection between, like, the current\... \[you think\] if we just push really hard on the current machine learning paradigm for 50 years, we won\'t have an AGI. We need to do something different for an AGI, which sounds like embodiment / combination with humans, biological merging?

\*\*0:26:31.7 Interviewee:\*\* So it could be embodiment and combination with humans, but also just better, different learning algorithms. So probably more sparsity is something that scales better. More efficient learning. So the problem with gradient descent is that you need too much data for it. Maybe we need some like Bayesian things where we can very quickly update belief systems. But maybe that needs to happen at a symbolic level. I still think we have to fix symbolic processing happening on neural networks\-- so we\'re still very good at pattern recognition, and I think one of the things you see with things like GPT-3 is that humans are amazing at anthropomorphizing anything. I don\'t know if you\'ve ever read any Daniel Dennett, but what we do is we take an intentional stance towards things, and so we are ascribing intentionality even to inanimate objects. His theory is essentially that consciousness comes from that. So we are taking an intentional stance towards ourselves and thinking of ourselves as a rational agent and that loop is what consciousness is. But actually we\'re sort of biological machines who perceive their own actions and over time this became what we consider consciousness. So\... where was I going with this? \[laughs\] What was the question?

\*\*0:27:57.2 Vael:\*\* Yeah, okay. So I\'m like, alright, we\'ve got AI, we\'ve got lots of machine learning\--

\*\*0:28:00.8 Interviewee:\*\* \--oh yeah, so do you need new learning algorithms? Yeah. So I think what we need to solve is the sort of System 2, higher-level thinking and how to implement that on the neural net. The neural symbolic divide is still very much an open problem. There are lots of problems we need to solve, where I really don\'t think we can just easily solve them by scaling. And that\'s\-- like there is very little other research happening actually in field right now.

\*\*0:28:35.3 Vael:\*\* Alright. So say we do scaling, but we also have a bunch of software. Like algorithmic improvements at the rate we\'re seeing, and we\'ve got hardware improvements as well. I guess this is just more scaling, but we have optical, we have quantum computing. And then we have some sort of fast learning systems, we know how to do symbolic processing, we\'re much more efficient. Here we now have a system that generalizes very well and is pretty efficient, and I don\'t know, maybe we\'re hundred years out. Say maybe we\'re in a different paradigm, maybe we\'re kind of in the same paradigm. We now have a system that is\--

\*\*0:29:05.5 Interviewee:\*\* We would be in a different paradigm for sure.

\*\*0:29:07.4 Vael:\*\* Okay. We are in a different paradigm, because\... because all these learning algorithms\--?

\*\*0:29:11.4 Interviewee:\*\* Paradigms don\'t really last that long, if you look at the history of science.

\*\*0:29:16.2 Vael:\*\* Okay, cool. But are we still operating under like, here\'s software with faster learning algorithms, more efficient learning algorithms, like symbolic reasoning, Bayesian stuff\--

\*\*0:29:24.7 Interviewee:\*\* Maybe. But I mean it could be that neuromorphic hardware finally lives up to its promise, or that we can do photonic chips at the speed of light computation and things like that. We\'re also very good in AI at fooling ourselves into thinking that we are responsible for all of these amazing breakthroughs, but without hardware engineers at NVIDIA, none of this stuff would have happened, right? They are doing very different things.

\*\*0:29:55.1 Vael:\*\* Alright, so we\'ve got this AI system which is quite general, we\'re in maybe a different paradigm, but we\'re still like\-- faster learning systems. Here we are, these things are very capable, very general, when they generate stories, they model physics in the world and then use that to generate their stories. Maybe they can do a lot of social stuff, maybe they know how to interact with people. And here we are with our system. Is this now an AGI?

\*\*0:30:18.0 Interviewee:\*\* No, no, so\-- Okay, now I remember what I was gonna say about the Dennett thing. So we anthropomorphize everything, we take this intentional stance at everything. We do this to ourselves, we do this to everything, especially when it speaks language. So when we see a language model and it\'s like, \"whoa, it\'s amazing, it does this thing,\" but all it\'s really doing is negative log likelihood, maximum likelihood estimation. It\'s basically just trying to fit \"what is the most likely word to go here\". So you can ask yourself whether we are so impressed by this system because it\'s so amazing, or because we are sort of programmed to have a lot of respect for things that speak language, because things that speak language tend to be humans. What you were just saying made it sound like you were saying, when these systems are sort of like humans, when they can do this and when they do that, and when they understand the world. So how do you define \"understanding the world\" there\--

\*\*0:31:18.7 Vael:\*\* I mostly mean like they could sub in for human jobs, for example\--

\*\*0:31:25.0 Interviewee:\*\* Yeah, but that\'s not the same thing as\-- stepping in for a human, they can already do that. But it depends on the problem. They\'re very good at counting, but\--

\*\*0:31:34.5 Vael:\*\* Yeah, but I don\'t think we could have like a mathematician AI right now per se. I guess I forgot to define my interpretation of AGI, but like a system that is very capable of replacing all current human day jobs.

\*\*0:31:51.6 Interviewee:\*\* Including yours and mine?

\*\*0:31:55.9 Interviewee:\*\* Yup.

\*\*0:31:57.8 Interviewee:\*\* Okay. But then who would it be useful for? Would the president still have a job or not?

\*\*0:32:09.7 Vael:\*\* Uh\... It doesn\'t have to. I think you could just spend\-- humans wouldn\'t have to work anymore, for example, and they could just go around doing whatever they do.

\*\*0:32:16.7 Interviewee:\*\* Yeah. But that\'s not at all what humans do. We\'re all so programmed to compete with each other.

\*\*0:32:24.7 Vael:\*\* Yeah, we can have games, we can have competitions, we can do all sorts of things, we have sports.

\*\*0:32:29.1 Interviewee:\*\* I think it\'s gonna be very quickly my AI versus your AI, basically.

\*\*0:32:33.9 Vael:\*\* Okay, we can have big fights with AIs, that seems very dangerous.

\*\*0:32:37.3 Interviewee:\*\* Yeah, I know, yeah. So that is a more likely scenario, I think, than everybody being nice and friendly and playing games. (Vael: \"Yeah.\") If people want to have power, and whoever controls the AGI will have the most power, (Vael: \"That seems right,\") then I think we\'re going to be developing your own AGIs at the same time. And then those AGIs at some point are going to be fighting with each other.

\*\*0:33:02.0 Vael:\*\* Yeah, yeah, I think we might even get problems before that, where we\'re not able to get AIs aligned with us. Have you heard of AI alignment?

\*\*0:33:10.9 Interviewee:\*\* Yeah, so \[close professional relationship\] wrote a nice thesis about it. \[Name\], I don\'t know if you know \[them\] by any chance. So yeah, alignment is important, but my concern with all this alignment stuff is that it\'s very ill-defined, I think. Either it means the same as correctness, so is your system just correct, or good at what you want it to be good at\... alignment is sort of like a reinvention of just correctness. I can see why this is useful for some people to put a new name on it. But I think it\'s a very old concept where it\'s just, okay, we\'re measuring things on a very narrow static test set, but we should be thinking about all these other things. You want your system to be really good when you deploy it in the real world. So it needs to be a good system or a correct or an aligned system. And so alignment maybe is a useful concept, only in the sense that the systems are getting so good now that you can start thinking about different kinds of goodness that we didn\'t think about before, and we can call that alignment, like human value-style things. But I think the concept itself is very old; it\'s just like, is your system correct?

\*\*0:34:40.0 Vael:\*\* Yeah. And then it\'s nowadays being thought about in terms of very far future systems and aligning with all values and preferences. (Interviewee: Yeah.) Cool. Yeah, do you work on any sort of AI safety or what would convince you to work on this or not work on this, etcetera?

\*\*0:34:56.5 Interviewee:\*\* Yeah so, I\'m not sure. AI safety is a bit of a weird concept to me, but I do work on the responsible AI and ethical AI, yeah.

\*\*0:35:06.9 Vael:\*\* Hm. And what does that mean\--

\*\*0:35:09.1 Interviewee:\*\* So these are things like\... I\'m trying to get better fairness metrics for systems. So in \[company\] we built this provisional fairness metric where we do some heuristic swaps. And so right now we\'re working on a more sophisticated method for doing this where, let\'s say, you have something, a sentence or some sort of natural language inference example, so a premise and a hypothesis and it\'s about James, like if you change James to Jamal, that shouldn\'t change your prediction at all. Or if you change the gender from James and you turn into a woman, that shouldn\'t change anything there. And it does, actually, if you look at restaurant reviews, if you changed the restaurant to a Mexican restaurant and the person who\'s eating there to Jamal, then your sentiment goes down. So this is the sort of stuff that shouldn\'t happen in these systems that is direct consequence of us just scaling the hell out of our systems on as much data as we can, including all of the biases that exist in this data. So I\'m working on trying to do that better measurement for these sort of things. And so I think if we are not getting better at measurement, then all of this stuff is basically a pointless discussion.

\*\*0:36:29.1 Vael:\*\* Great, thank you. And then my last question is, have you changed your mind on anything during this interview and how was this interview for you?

\*\*0:36:35.9 Interviewee:\*\* It was fun. Yeah, I\'ve done a few of these with various people and it\'s always a bit like, I don\'t know. It feels a bit like\... we\'re getting ahead of ourselves a little bit. But maybe I\'m also just old. So when I talked to \[close professional relationship\] and how \[they\] think about stuff, I\'m like, I just don\'t understand how \[they\] think about AI.

\*\*0:37:06.2 Vael:\*\* Got it. \[They\'re\] like way out here, and we need to make sure that systems do our correct\--

\*\*0:37:11.9 Interviewee:\*\* Yeah, \[they\'re\] really.. Yeah, \[they\] put a lot more faith also in AI, which I think is very interesting. So I asked \[them\] like, \"Okay, so this alignment stuff, in the end who should we ask what is right or what is wrong? When we\'re trying to design the best AI systems, who should we ask for what\'s right and wrong?\" And then \[their\] answer was, \"We should ask the AI.\"

\*\*0:37:38.7 Vael:\*\* What? No, we should ask humans.

\*\*0:37:41.0 Interviewee:\*\* Yeah, no, so \[they\] think that basically AGI or AI is going to get so good, these language models are gonna get so good that they can tell us how we should think about our own moral philosophical values so that we can impose them onto AI systems. That to me just sounds crazy, like batshit crazy, but that\'s one way to think about it. I mean, I respect \[their\] opinion. I just can\'t understand it.

\*\*0:38:11.7 Vael:\*\* Interesting. Yeah, I think if I try to model what I would imagine \[they\] would be saying, under the alignment paradigm, I would say that you need to ask human feedback, but it\'s hard to get human feedback on very intelligent systems. And so you should ask AI to summarize human feedback, but it should always be ground down on a human otherwise we\'re in trouble, so.

\[ending comments\]