Irina Rish - Out-of-distribution generalization-by Towards Data Science-video_id QjXFN4UWZCg-date 20220309

# Irina Rish on Out-of-distribution generalization by Jeremie Harris on the Towards Data Science Podcast

During training, AIs will often learn to make predictions based on features that are easy to learn, but deceptive.

Imagine, for example, an AI that’s trained to identify cows in images. Ideally, we’d want it to learn to detect cows based on their shape and colour. But what if the cow pictures we put in the training dataset always show cows standing on grass?

In that case, we have a \_spurious correlation\_ between grass and cows, and if we’re not careful, our AI might learn to become a grass detector rather than a cow detector. Even worse, we could only realize that’s happened once we’ve deployed it in the real world and it runs into a cow that isn’t standing on grass for the first time.

So how do you build AI systems that can learn robust, general concepts that remain valid outside the context of their training data?

That’s the problem of out-of-distribution generalization, and it’s a central part of the research agenda of Irina Rish, a core member of the Mila— Quebec AI Research institute, and the Canadian Excellence Research Chair in Autonomous AI. Irina’s research explores many different strategies that aim to overcome the out-of-distribution problem, from empirical AI scaling efforts to more theoretical work, and she joined me to talk about just that on this episode of the podcast.

Here were some of my favourite take-homes from the conversation:

- To Irina, GPT-3 was an “AlexNet moment” for AI alignment and AI safety research. For the first time, we had built highly capable AIs without actually understanding their behaviour, or knowing how to steer it. As a result, Irina thinks that this is a great time to get into AI alignment research.
- Irina thinks that out-of-distribution generalization is an area where AI capabilities research starts to merge with AI alignment and AI safety research. Getting systems to learn robust concepts is not only helpful for ensuring that they have rich representations of the world (which helps with capabilities), but also helps ensure that accidents don’t happen by tackling the problem of spurious correlations.
- Irina has researched several strategies aimed at addressing the out-of-distribution sampling problem. One of them involves using the invariance principle: the idea that the features we want our AI models to learn from are going to be consistent (invariant) regardless of the environments our data come from. Consider for example the case of cow detection I mentioned earlier: the features we want our AI to lock onto (the shape and colour of cows, for example) are consistent across different environments. A cow is still a cow whether it’s in a pasture, indoors or in the middle of a desert. Irina is exploring techniques that allow AIs to distinguish between features that are invariant and desirable (like cow shape and colour) and features that are variable and unreliable predictors (like whether or not there’s grass on the ground).
- Another approach Irina sees as promising is scaling. We’ve talked about scaling on the podcast before — but in a nutshell, it’s the idea that current deep learning techniques can more or less get us to AGI as-is, if only they’re used to train large enough neural networks with huge enough datasets, and an equally massive quantity of compute power. In principle, scaling certain kinds of neural nets could allow AIs to learn so much about their training data that their performance is limited only by the irreducible noise of the data itself.
- That possibility raises another question: is there too much noise in language data (which was used to train GPT-3, and the first generation of massively scaled foundation models) for AIs trained on language alone to reach human-level capabilities across the board? It’s possible, Irina thinks — and that’s why she’s excited about the trend towards multi-modal learning: the practice of training AIs on multiple data types at the same time (for example, on image, text and audio data). The hope is that by combining these input data types, an AI can learn to transcend noise limits that may exist in any one data type alone.


Reframing superintelligence _ Eric Drexler _ EA Global - London 2018-by Centre for Effective Altruism-video_id MircoV5LKvg-date 20190314

# Eric Drexler Reframing Superintelligence - EA Forum

\_When people first began to discuss advanced artificial intelligence, existing AI was rudimentary at best, and we had to reply on ideas about human thinking and extrapolate. Now, however, we've developed many different advanced AI systems, some of which outperform human thinking on certain tasks. In this talk from EA Global 2018: London, Eric Drexler argues that we should use this new data to rethink our models for how superintelligent AI is likely to emerge and function.\_

## The Talk

I've been working in this area for quite a while. The chairman of my doctoral committee was one Marvin Minsky. We had some discussions on AI safety around 1990. He said I should write them up. I finally got around to writing up some developed versions of those ideas just very recently, so that's some fairly serious procrastination. Decades of procrastination on something important.

For years, one couldn't talk about advanced AI. One could talk about nanotechnology. Now it's the other way around. You can talk about advanced AI, but not about advanced nanotechnology. So this is how the Overton window moves around.

What I would like to do is to give a very brief presentation which is pretty closely aligned with talks I've given at OpenAI, DeepMind, FHI, and Bay Area Rationalists. Usually I give this presentation to a somewhat smaller number of people, and structure it more around discussion. But what I would like to do, still, is to give a short talk, put up points for discussion, and encourage something between Q&A and discussion points from the audience.

Okay so, when I say "Reframing Superintelligence," what I mean is thinking about the context of emerging AI technologies as a process rolling forward from what we see today. And asking, "What does that say about likely paths forward?" Such that whatever it is that you're imagining needs to emerge from that context or make sense in that context. Which I think reframes a lot of the classic questions. Most of the questions don't go away, but the context in which they arise, the tools available for addressing problems, look different. That's what we'll be getting into.

Once upon a time, when we thought about advanced AI, we didn't really know what AI systems were likely to look like. It was very unknown. People thought in terms of developments in logic and other kinds of machine learning, different from the deep learning that we now see moving forward with astounding speed. And people reached for an abstract model of intelligent systems. And what intelligent systems do we know? Well, actors in the world like ourselves. We abstract from that very heavily and you end up with rational, utility-directed agents.

Today, however, we have another source of information beyond that abstract reasoning, which applies to a certain class of systems. And information that we have comes from the world around us. We can look at what's actually happening now, and how AI systems are developing. And so we can ask questions like, "Where do AI systems come from?" Well, today they come from research and development processes. We can ask, "What do AI systems do today?" Well, broadly speaking, they perform tasks. Which I think of, or will describe, as "performing services." They do some approximation or they do something that someone supposedly wants in bounded time with bounded resources. What will they be able to do? Well, if we take AI seriously, AI systems will be able to automate asymptotically all human tasks, and more, at a piecemeal and asymptotically general superintelligent level. So we said AI systems come from research and development. Well, what is research and development? Well, it's a bunch of tasks to automate. And, in particular, they're relatively narrow technical tasks which are, I think, uncontroversially automate-able on the path to advanced AI.

So the picture is of AI development moving forward broadly along the lines that we're seeing. Higher-level capabilities. More and more automation of the AI R&D process itself, which is an ongoing process that's moving quite rapidly. AI-enabled automation and also classical software techniques for automating AI research and development. And that, of course, leads to acceleration. Where does that lead? It leads to something like recursive improvement, but not the classic recursive improvement of an agent that is striving to be a more intelligent, more capable agent. But, instead, recursive improvement where an AI technology base is being advanced at AI speed. And that's a development that can happen incrementally. We see it happening now as we take steps toward advanced AI that is applicable to increasingly general and fast learning. Well, those are techniques that will inevitably be folded into the ongoing AI R&D process. Developers, given some advance in algorithms and learning techniques, and a conceptualization of how to address more and more general tasks, will pounce on those, and incorporate them into a broader and broader range of AI services.

So where that leads is to asymptotically comprehensive AI services. Which, crucially, includes the service of developing new services. So increasingly capable, increasingly broad, increasingly piecemeal and comprehensively superintelligent systems that can work with people, and interact with people in many different ways to provide the service of developing new services. And that's a kind of generality. That is a general kind of artificial intelligence. So a key point here is that the C in CAIS, C in Comprehensive AI Services does the work of the G in AGI. Why is it a different term? To avoid the implication... when people say AGI they mean AGI agent. And we can discuss the role of agents in the context of this picture. But I think it's clear that a technology base is not inherently in itself an agent. In this picture agents are not central, they are products. They are useful products of diverse kinds for providing diverse services. And so with that, I would like to (as I said, the formal part here will be short) point to a set of topics.

They kind of break into two categories. One is about short paths to superintelligence, and I'll argue that this is the short path. The topic of AI services and agents, including agent services, versus the concept of "The AI" which looms very large in people's concepts of future AI. I think we should look at that a little bit more closely. Superintelligence as something distinct from agents, superintelligent non-agents. And the distinction between general learning and universal competence. People have, I think, misconstrued what intelligence means and I'll take a moment on that. If you look at definitions of good from the 1960s, ultra-intelligence and more recent Bostrom and so on (I work across the hall from Nick) on superintelligence the definition is something like "a system able to outperform any person in any task whatsoever." Well, that implies general competence, at least as ordinarily read. But there's some ambiguity over what we mean by the word "intelligence" more generally. We call children intelligent and we call senior experts intelligent. We call a child intelligent because the child can learn, not because the child can perform at a high level in any particular area. And we call an expert who can perform at a high level intelligent not because the expert can learn - in principle you could turn off learning capacity in the brain - but because the expert can solve difficult problems at a high level.

So learning and competence are dissociable components of intelligence. They are in fact quite distinct in machine learning. There is a learning process and then there is an application of the software. And when you see discussion of intelligent systems that does not distinguish between learning and practice, and treats action as entailing learning directly, there's a confusion there. There's a confusion about what intelligence means and that's, I think, very fundamental. In any event, looking toward safety-related concerns, there are things to be said about predictive models of human concerns. AI-enabled solutions to AI-control problems. How this reframes questions of technical AI safety. Issues of services versus addiction, addictive services and adversarial services. Services include services you don't want. Taking superintelligent services seriously. And a question of whether faster development is better.

And, with that, I would like to open for questions, discussion, comment. I would like to have people come away with some shared sense of what the questions and comments are. Some common knowledge of thinking in this community in the context of thinking about questions this way.

## Discussion

\_Question\_: Is your model compatible with end-to-end reinforcement learning?

\_Eric\_: Yes.

To say a little bit more. By the way, I've been working on a collection of documents for the last two years. It's now very large, and it will be an FHI technical report soon. It's 30,000 words structured to be very skim-able. Top-down, hierarchical, declarative sentences expanding into longer ones, expanding into summaries, expanding into fine-grained topical discussion. So you can sort of look at the top level and say, hopefully, "Yes, yes, yes, yes, yes. What about this?" And not have to read anything like 30,000 words. So, what I would say is that reinforcement learning is a technique for AI system development. You have a reinforcement learning system. It produces through a reinforcement learning process, which is a way of manipulating the learning of behaviors. It produces systems that are shaped by that mechanism. So it's a development mechanism for producing systems that provide some service. Now if you turn reinforcement learning loose in the world open-ended, read-write access to the internet, a money-maximizer and did not have checks in place against that? There are some nasty scenarios. So basically it's a development technique, but could also be turned loose to produce some real problems. "Creative systems trying to manipulate the world in bad ways" scenarios are another sector of reinforcement learning. So not a problem per se, but one can create problems using that technique.

\_Question\_: What does asymptotic improvement of AI services mean?

\_Eric\_: I think I'm abusing the term asymptotic. What I mean is increasing scope and increasing level of capability in any particular task to some arbitrary limit. Comprehensive is sort of like saying infinite, but moving toward comprehensive and superintelligent level services. What it's intended to say is, ongoing process going that direction. If someone has a better word than asymptotic to describe that I'd be very happy.

\_Question\_: Can the tech giants like Facebook and Google be trusted to get alignment right?

\_Eric\_: Google more than Facebook. We have that differential. I think that questions of alignment look different here. I think more in terms of questions of application. What are the people who wield AI capabilities trying to accomplish? So there's a picture which, just background to the framing of that question, and a lot of these questions I think I'll be stepping back and asking about framing. As you might think from the title of the talk. So picture a rising set of AI capabilities: image recognition, language understanding, planning, tactical management in battle, strategic planning for patterns of action in the world to accomplish some goals in the world. Rising levels of capability in those tasks. Those capabilities could be exploited by human decision makers or could, in principle, be exploited by a very high-level AI system. I think we should be focusing more, not exclusively, but more on human decision makers using those capabilities than on high-level AI systems. In part because human decision makers, I think, are going to have broad strategic understanding more rapidly. They'll know how to get away with things without falling afoul of what nobody had seen before, which is intelligence agencies watching and seeing what you're doing. It's very hard for a reinforcement learner to learn that kind of thing.

So I tend to worry about not the organizations making aligned AI so much as whether the organizations themselves are aligned with general goals.

\_Question\_: Could you describe the path to superintelligent services with current technology, using more concrete examples?

\_Eric\_: Well, we have a lot of piecemeal examples of superintelligence. AlphaZero is superintelligent in the narrow domain of Go. There are systems that outperform human beings in playing these very different kinds of games, like Atari games. Face recognition recently surpassed human ability to map from human speech to transcriptive words. Just more and more areas piecemeal. A key area that I find impressive and important is the design of neural networks at the core of modern deep learning systems. The design of and learning to use appropriately, hyperparameters. So, as of a couple of years ago, if you wanted a new neural network, a convolutional network for vision, or some recurrent network, though recently they're going for convolution networks for language understanding and translation, that was a hand-crafted process. You had human judgment and people were building these networks. A couple of years ago people started in these, this is not AI in general but it's a chunk that a lot of attention went into, getting superhuman performance in neural networks by automated, AI-flavored like, for example, reinforcement learning systems. So developing reinforcement learning systems that learn to put together the building blocks to make a network that outperforms human designers in that process. So we now have AI systems that are designing a core part of AI systems at a superhuman level. And this is not revolutionizing the world, but that threshold has been crossed in that area.

And, similarly, automation of another labor-intensive task that I was told very recently by a senior person at DeepMind would require human judgment. And my response was, "Do you take AI seriously or not?" And, out of DeepMind itself, there was then a paper that showed how to outperform human beings in hyperparameter selection. So those are a few examples. And the way one gets to an accelerating path is to have more and more, faster and faster implementation of human insights into AI architectures, training methods, and so on. Less and less human labor required. Higher and higher level human insights being turned into application throughout the existing pool of resources. And, eventually, fewer and fewer human insights being necessary.

\_Question\_: So what are the consequences of this reframing of superintelligence for technical AI safety research?

\_Eric\_: Well, re-contexting. If in fact one can have superintelligent systems that are not inherently dangerous, then one can ask how one can leverage high-level AI. So a lot of the classic scenarios of misaligned powerful AI involve AI systems that are taking actions that are blatantly undesirable. And, as Shane Legg said when I was presenting this at DeepMind last Fall, "There's an assumption that we have superintelligence without common sense." And that's a little strange. So Stuart Russell has pointed out that machines can learn not only from experience, but from reading. And, one can add, watching video and interacting with people and through questions and answers in parallel over the internet. And we see in AI that a major class of systems is predictive models. Given some input you predict what the next thing will be. In this case, given a description of a situation or an action, you try to predict what people will think of it. Is it something that they care about or not? And, if they do care about it, is there widespread consensus that that would be a bad result? Widespread consensus that it would be a good result? Or strongly mixed opinion?

Note that this is a predictive model trained on many examples, it's not an agent. That is an oracle that, in principle, could operate with reasoning behind the prediction. That could in principle operate at a super intelligent level, and would have common sense about what people care about. Now think about having AI systems that you intend to be aligned with human concerns where, available for a system that's planning action, is this oracle. It can say, "Well, if such and such happened, what would people think of it?" And you'd have a very high-quality response. That's a resource that I think one should take account of in technical AI safety. We're very unlikely to get high-level AI without having this kind of resource. People are very interested in predicting human desires and concerns if only because they want to sell you products or brainwash you in politics or something. And that's the same underlying AI technology base. So I would expect that we will have predictive models of human concerns. That's an example of a resource that would reframe some important aspects of technical AI safety.

\_Question\_: So, making AI services more general and powerful involves giving them higher-level goals. At what point of complexity and generality do these services then become agents?

\_Eric\_: Well, many services are agent-services. A chronic question that arises, people will be at FHI or DeepMind and someone will say, "Well, what is an agent anyway?" And everybody will say, "Well, there is no sharp definition. But over here we're talking about agents and over here we're clearly not talking about agents." So I would be inclined to say that if a system is best thought of as directed toward goals and it's doing some kind of planning and interacting with the world I'm inclined to call it an agent. And, by that definition, there are many, many services we want, starting with autonomous vehicles, autonomous cars and such, that are agents. They have to make decisions and plan. So there's a spectrum from there up to higher and higher level abilities to do means-ends analysis and planning and to implement actions. So let's imagine that your goal is to have a system that is useful in military action and you would like to have the ability to execute tactics with AI speed and flexibility and intelligence, and have strategic plans for using those tactics that are superintelligent level.

Well, those are all services. They're doing something in bounded time with bounded resources. And, I would argue, that that set of systems would include many systems that we would call agents but they would be pursuing bounded tasks with bounded goals. But the higher levels of planning would naturally be structured as systems that would give options to the top level decision makers. These decision makers would not want to give up their power, they don't want a system guessing what they want. At a strategic level they have a chance to select, since strategy unfolds relatively slowly. So there would be opportunities to say, "Well, don't guess, but here's the trade off I'm willing to make between having this kind of impact on opposition forces with this kind of lethality to civilians and this kind of impact on international opinion. I would like options that show me different trade-offs. All very high quality but within that trade-off space. And here I'm deliberately choosing an example which is about AI resources being used for projecting power in the world. I think that's a challenging case, so it's a good place to go.

I'd like to say just a little bit about the opposite end, briefly. Superintelligent non-agents. Here's what I think is a good paradigmatic example of superintelligence and non-agency. Right now we have systems that do natural language translation. You put in sentences or, if you had a somewhat smarter system that dealt with more context, books, and out comes text in a different language. Well, I would like to have systems that know a lot to do that. You do better translations if you understand more about history, chemistry if it's a chemistry book, human motivations. Just, you'd like to have a system that knows everything about the world and everything about human beings to give better quality translations. But what is the system? Well, it's a product of R&D and it is a mathematical function of type character string to character string. You put in a character string, things happen, and out comes a translation. You do this again and again and again. Is that an agent? I think not. Is it operating at a superintelligent level with general knowledge of the world? Yes. So I think that one's conceptual model of what high-level AI is about should have room in it for that system and for many systems that are analogous.

\_Question\_: Would a system service that combines general learning with universal competence not be more useful or competitive than a system that displays either alone? So does this not suggest that agents might be more useful?

\_Eric\_: Well, as I said, agents are great. The question is what kind and for what scope. So, as I was saying, distinguishing between general learning and universal competence is an important distinction. I think it is very plausible that we will have general learning algorithms. And general learning algorithms may be algorithms that are very good at selecting algorithms that are good at selecting algorithms for learning a particular task and inventing new algorithms. Now, given an algorithm for learning, there's a question of what you're training it to do. What information? What competencies are being developed? And I think that the concept of a system being trained on and learning about everything in the world with some objective function, I don't think that's a coherent idea. Let's say you have a reinforcement learner. You're reinforcing the system to do what? Here's the world and it's supposed to be getting competence in organic chemistry and ancient Greek and, I don't know, control of the motion of tennis-playing robots and on and on and on and on. What's the reward function, and why do we think of that as one task?

I don't think we think of it as one task. I think we think of it as a bunch of tasks which we can construe as services. Including the service of interacting with you, learning what you want, nuances. What you are assumed to want, what you're assumed not to want as a person. More about your life and experience. And very good at interpreting your gestures. And it can go out in the world and, subject to constraints of law and consulting an oracle on what other people are likely to object to, implement plans that serve your purposes. And if the actions are important and have a lot of impact, within the law presumably, what you want is for that system to give you options before the system goes out and takes action. And some of those actions would involve what are clearly agents. So that's the picture I would like to paint that I think reframes the context of that question.

\_Question\_: So on that is it fair to say that the value-alignment problem still exists within your framework? Since, in order to train a model to build an agent that is aligned with our values, we must still specify our values.

\_Eric\_: Well, what do you mean by, "train an agent to be aligned with our values." See, the classic picture says you have "The AI" and "The AI" gets to decide what the future of the universe looks like and it had better understand what we want or would want or should want or something like that. And then we're off into deep philosophy. And my card says philosophy on it, so I guess I'm officially a philosopher or something according to Oxford. I was a little surprised. "It says philosophy on it. Cool!" I do what I think of as philosophy. So, in a services model, the question would instead be, "What do you want to do?" Give me some task that is completed in bounded time with bounded resources and we could consider how to avoid making plans that stupidly cause damage that I don't want. Plans that, by default, automatically do what I could be assumed to want. And that pursue goals in some creative way that is bounded, in the sense that it's not about reshaping the world; other forces would presumably try to stop you. And I'm not quite sure what value alignment means in that context. I think it's something much more narrow and particular.

By the way, if you think of an AI system that takes over the world, keep in mind that a sub-task of that, part of that task, is to overthrow the government of China. And, presumably, to succeed the first time because otherwise they're going to come after you if you made a credible attempt. And that's in the presence of unknown surveillance capabilities and unknown AI that China has. So you have a system and it might formulate plans to try to take over the world, well, I think an intelligent system wouldn't recommend that because it's a bad idea. Very risky. Very unlikely to succeed. Not an objective that an intelligent system would suggest or attempt to pursue. So you're in a very small part of a scenario space where that attempt is made by a high-level AI system. And it's a very small part of scenario space because it's an even smaller part of scenario space where there is substantial success. I think it's worth thinking about this. I think it's worth worrying about it. But it's not the dominant concern. It's a concern in a framework where I think we're facing an explosive growth of capabilities that can amplify many different purposes, including the purposes of bad actors. And we're seeing that already and that's what scares me.

\_Question\_: So I guess, in that vein, could the superintelligent services be used to take over the world by a state actor? Just the services?

\_Eric\_: Well, you know, services include tactical execution of plans and strategic planning. So could there be a way for a state actor to do that using AI systems in the context of other actors with, presumably, a comparable level of technology? Maybe so. It's obviously a very risky thing to do. One aspect of powerful AI is an enormous expansion of productive capacity. Partly through, for example, high-level, high quality automation. More realistically, physics-limited production technology, which is outside today's sphere of discourse or Overton window.

Security systems, I will assert, could someday be both benign and effective, and therefore stabilizing. So the argument is that, eventually it will be visibly the case that we'll have superintelligent level, very broad AI, enormous productive capacity, and the ability to have strategic stability, if we take the right measures beforehand to develop appropriate systems, or to be prepared to do that, and to have aligned goals among many actors. So if we distribute the much higher productive capacity well, we can have an approximately strongly Pareto-preferred world, a world that looks pretty damn good to pretty much everyone.

\_Note: for a more thorough presentation on this topic, see Eric Drexler's\_ [\_other talk\_](https://www.effectivealtruism.org/articles/ea-global-2018-paretotopian-goal-alignment/) \_from this same conference.\_

\_Question\_: What do you think the greatest AI threat to society in the next 10, 20 years would be?

\_Eric\_: I think the greatest threat is instability. Sort of either organic instability from AI technologies being diffused and having more and more of the economic relationships and other information-flow relationships among people be transformed in directions that increase entropy, generate conflict, destabilize political institutions. Who knows? If you had the internet and people were putting out propaganda that was AI-enabled, it's conceivable that you could move elections in crazy directions in the interest of either good actors or bad actors. Well, which will that be? I think we will see efforts made to do that. What kinds of counter-pressures could be applied to bad actors using linguistically politically-competent AI systems to do messaging? And, of course, there's the perennial states engaging in an arms race which could tip into some unstable situation and lead to a war. Including the long-postponed nuclear war that people are waiting for and might, in fact, turn up some day. And so I primarily worry about instability. Some of the modes of instability are because some actor decides to do something like turn loose a competent hacking, reinforcement-learning system that goes out there and does horrible things to global computational infrastructure that either do or don't serve the intentions of the parties that released it. But take a world that's increasingly dependent on computational infrastructure and just slice through that, in some horribly destabilizing way. So those are some of the scenarios I worry about most.

\_Question\_: And then maybe longer term than 10, 20 years? If the world isn't over by then?

\_Eric\_: Well, I think all of our thinking should be conditioned on that. If one is thinking about the longer term, one should assume that we are going to have superintelligent-level general AI capabilities. Let's define that as the longer term in this context. And, if we're concerned with what to do with them, that means that we've gotten through the process to there then. So there's two questions. One is, "What do we need to do to survive or have an outcome that's a workable context for solving more problems?" And the other one is what to do. So, if we're concerned with what to do, we need to assume solutions to the preceding problems. And that means high-level superintelligent services. That probably means mechanisms for stabilizing competition. There's a domain there that involves turning surveillance into something that's actually attractive and benign. And the problems downstream, therefore, one hopes to have largely solved. At least the classic large problems and now problems that arise are problems of, "What is the world about anyway?" We're human beings in a world of superintelligent systems. Is trans-humanism in this direction? Uploading in this direction? Developing moral patients, superintelligent-level entities that really aren't just services, and are instead the moral equivalent of people? What do you do with the cosmos? It's an enormously complex problem. And, from the point of view of having good outcomes, what can I say? There are problems.

\_Question\_: So what can we do to improve diversity in the AI sector? And what are the likely risks of not doing so?

\_Eric\_: Well, I don't know. My sense is that what is most important is having the interests of a wide range of groups be well represented. To some extent, obviously, that's helped if you have in the development process, in the corporations people who have these diverse concerns. To some extent it's a matter of politics regulation, cultural norms, and so on. I think that's a direction we need to push in. To put this in the Paretotopian framework, your aim is to have objectives, goals that really are aligned, so, possible futures that are strongly goal-aligning for many different groups. For many of those groups, we won't fully understand them from a distance. So we need to have some joint process that produces an integrated, adjusted picture of, for example, how do we have EAs be happy and have billionaires maintain their relative position? Because if you don't do that they're going to maybe oppose what you're doing, and the point is to avoid serious opposition. And also have the government of China be happy. And I would like to see the poor in rural Africa be much better off, too. Billionaires might be way up here, competing not to build orbital vehicles but instead starships. And the poor in rural Africa of today merely have orbital space capabilities convenient for families, because they're poor. Nearly everyone much, much better off.


NeurIPSorICML_a0nfw-by Vael Gates-date 20220318

# Interview with AI Researchers NeurIPSorICML\_a0nfw by Vael Gates

\*\*Interview with a0nfw, on 3/18/22\*\*

\*\*0:00:02.3 Vael:\*\* Alright. So my first question is, can you tell me about what area of AI you work on in a few sentences?

\*\*0:00:08.7 Interviewee:\*\* Yes. More recently I\'ve been working on AI for mathematical reasoning and applications in navigation, mathematical navigation.

\*\*0:00:18.1 Vael:\*\* Great. And then, what are you most excited about in AI and what are you most worried about? In other words, what are the biggest benefits or risks of AI?

\*\*0:00:25.8 Interviewee:\*\* The biggest benefit of AI as a tool that now we have and that sort of works, is that it just enabled a lot of new applications when we are able to define goals for it. The broad story that I\'m pursuing recently is like, suppose you have an AI expert for a certain domain, then how can you leverage that expert to teach people to be better at the same domain? And the reason why we\'re working more on mathematical education, it\'s exactly because the mathematical domains are easier to define as a formal problem. Training solvers for interesting mathematical domains is not possible with AI tools, that \[hard to parse\] not be so much 10 years ago. In a lot of other fields, it\'s also been the case that we as humans knew what we wanted and wrote down a bunch of heuristics to accomplish certain goals. But we were always sure that those weren\'t the best, and now AI is allowing us to replace a lot of those by just better algorithms.

The main worry is exactly when we don\'t know exactly what the goal is, or when we don\'t know how to specify it. Or in a lot of cases, in research, people have been stuck with these proxy tasks that are supposed to represent some kind of behavior that we want, but a lot of people are just unsure of what exactly what we want out of these systems. It\'s a worry for me that a lot of people are spending a lot of time and resources on those problems without knowing exactly what to expect. I come from sort of a systems background, so I\'m more used to people having proxy tasks that are very directly related to the actual task. For example, in compilers, usually one goal is, make programs faster, and then you can create benchmarks. \"Okay, here is the set of programs, let\'s run your compiler and it optimize\[?\] and see how long it takes.\" And you can argue about whether those programs are representative of what actual programs that people will run it on, if the benchmark that I have is representative or not, but the goal was very clear, it\'s to make things faster. Now in language for example, in NLP, we have a lot of these tasks about language understanding and benchmarks that try to capture some form of understanding. But it\'s not entirely clear what the goal is, if we solve this benchmark\-- and this is being revised all the time. People propose a benchmark that\'s hard for current-day models, then a few months later, someone comes up with a solution. Then people say, \"Oh, but that\'s actually not exactly what we wanted \'cause look, the model doesn\'t do this other thing.\" And then we enter this cycle of refining exactly what\'s the problem and developing models, but without a clear goal in a lot of cases.

\*\*0:03:47.7 Vael:\*\* Got it. Okay, so that question was, what are you most excited about AI and what are you most worried about in AI? Could you summarize the culmination of all that?

\*\*0:03:57.1 Interviewee:\*\* Yeah. It\'s centered around goals. So AI lets us pose richer and new kinds of goals more formally and optimize for those. When those are clear, those are cases where I\'m very excited about. When they\'re not clear, then I\'m worried about it.

\*\*0:04:16.3 Vael:\*\* Got it. Cool, that makes sense. Awesome. Alright, so focusing on future AI, putting on a science fiction forecasting hat, say we\'re 50 plus years into the future. So at least 50 years in the future, what does that future look like?

\*\*0:04:32.8 Interviewee:\*\* It can look like\... a lot of\... it can go in a lot of ways, I think. I think, in one possible future\-- do you want me to just list the possible futures?

\*\*0:04:48.1 Vael:\*\* I think I\'m most interested in your realistic future, but also like optimistic, pessimistic; it\'s a free-form question.

\*\*0:04:57.8 Interviewee:\*\* Okay. Yeah, maybe optimistically what would happen is that AI lets us solve problems that are important for society and that we just don\'t have the right tools to at this point. I think a lot of exciting applications in\... In language, we can imagine some interesting applications where a lot of computer interfaces become much easier, just for doing complex tasks, to specify a natural language. Like programming right now gives a lot of power to people, but it also takes a lot of time to learn, so it would be awesome to enable a lot more people to automate tasks using natural language. So in one future, a lot of those applications will be enabled and things will be great. In another future, which is actually\-- it\'s probably going to be a combination of these things, but a very likely future, which is already kind of happening, but maybe at a smaller scale than is possible, is that people will start replacing existing tools and systems with AI for not very clear reasons and get not very clear outcomes. And then we don\'t really know exactly how they\'re misbehaving in a lot of cases. And the thing is that the incentives for deploying the systems are at odds with broader societal goals in a lot of cases. So, for example, private companies like Google and Facebook, they have all the incentives possible to deploy these systems to optimize their metrics that ultimately correlate with revenue. I don\'t know.

Facebook has a lot of metrics which ultimately relate to both how much time people spend on the platform, and that translates to how much money they make. And they basically have thousands of engineers trying different combinations of features and things to suggest users to do, and like very small tweaks a lot of times to try to optimize for those through metrics. And a lot of that process became much faster with AI because now you can take much more fine-grain decisions for individual users. And sure, we can look at the metrics and see that, \"Yeah, they keep improving, that\'s a new tool that now exists and enables them to optimize for this goal.\" At the same time, it\'s not exactly to clear to me what is happening, that AI is choosing things for individual users. It might be like closing people off in their bubbles, it might be\... all the things we know about, but also potentially a lot of other things, other behaviors. Exactly because they\'re not the same behavior for everyone, that also makes it harder to study and understand. So I like the future, is that AI will replace a lot of these systems, but that will involve people losing control of exactly what it is doing. And that will probably come before us having a good understanding of what\'s going on, just because there is already an incentive to deploy those systems, like a financial incentive.

\*\*0:08:40.8 Vael:\*\* Yeah, interesting. You\'re making a lot of the arguments that I often make later in this interview. Alright. This next thing is, so people talk about the promise of AI, by which they mean many things, but one thing that I think they mean, and that I\'m talking about here, is that the idea of having a very general, capable system, such that the systems would have cognitive capacities that we could use to replace all current-day human jobs. Which we could or could not, but the cognitive capacities do that. And I usually think about this in the frame of like, here we have\... in 2012, we have AlexNet and the deep learning revolution, and then 10 years later, here we are, and you\'ve got systems like GPT-3, which have some weirdly emergent capabilities like text generation and language translation and math and coding and stuff. And so one might expect that if we continued pouring all the human effort that\'s been going into this, with nations competing and corporations competing and lots of young people and lots of talent and algorithmic improvements at the same rate we\'ve seen and hardware improvements, maybe we\'ll get optical or quantum computing, then we might scale to very general systems. Or we might not, and we might hit some sort of ceiling and need to paradigm shift. But my question is, regardless of how we get there, do you think we\'ll ever have very general systems like a CEO or a scientist AI, and if so, when?

\*\*0:11:00.9 Interviewee:\*\* Yeah. The thing that has been happening with these large language models, that they objectively become more and more capable, but we at the same time reassess what we consider to be general intelligence. That will probably keep happening for quite a while, I would say at least like 10-15 years, just because\...I don\'t think we know exactly what\... Human intelligence is still very poorly understood in a lot of ways. It\'s not clear that higher intelligence is possible without some of the shortcomings that human intelligence has. So I think that cycle of getting more powerful systems but also realizing some of their shortcomings and saying, \"Oh, maybe\...\" Basically shifting the goal and what is it that we consider general intelligence will probably keep happening.

\*\*0:12:07.1 Vael:\*\* Yeah, my question is, do you think that we\'ll get systems like a CEO AI or a scientist AI or be able to replace most cognitive jobs at some point, and when that will be?

\*\*0:12:18.6 Interviewee:\*\* Yeah. Let\'s see. We\'ll probably have systems that can act like a CEO or a scientist. Whether they\'ll have the same goals that a CEO or a scientist have, like a human CEO or a human scientist have in the real world, that is a different question, which is not that clear to me.

\*\*0:12:47.0 Vael:\*\* Interesting. Yeah. So there\'s the question of whether AIs develop like consciousness, for example. So I\'m assuming not consciousness, and I\'m like, \"Okay, say we have a system that can do multi-step planning, it can do models of other people modeling it, it\'s interacting with humans by text and video, or whatever,\" and I\'m like, \"Alright, I need you to solve cancer for me, AI,\" or, \"I want you to run this company,\" and we put in its goal, optimization function so that it just does that at some point. Yeah.

\*\*0:13:24.4 Interviewee:\*\* Yeah. You\'re right. So you asked about the cognitive capability. I think that will probably be there in like 30-40 years, would be my estimate. For at least a subset of what it means to run a company. Now, my question is, if we have a system that does that, is it also possible for the system to simply obey everything we tell it to do? Because, of course, a person is capable of doing that, but you wouldn\'t expect to tell the person, \"Okay, go do that.\" They might complain. They might say \"No, I\'m in a bad mood today,\" or something like that.

\*\*0:14:03.3 Vael:\*\* Yeah, and I don\'t even think you could tell any given human\... Like if someone is like, \"Alright, Vael, go be the CEO of a company,\" I\'d be like, \"What? I don\'t know how to do that.\"

\*\*0:14:09.9 Interviewee:\*\* Exactly, exactly. Yeah.

\*\*0:14:12.0 Vael:\*\* Yeah, yeah. Interesting. Okay, so that leads into my next set of questions. So imagine we have a CEO AI and I\'m like, \"Alright, CEO AI, I wish for you to maximize profits, and try not to run out of money and try not to exploit people and try to avoid side effects.\" And so currently, this would be very technically challenging for a number of reasons. One reason is that we aren\'t very good at taking human values and preferences and stuff and putting them into mathematical formulations such that AI can optimize over. And so I worry that AI in the future will continue to do this, and then we\'ll have AI that continue to do what we tell them to do instead of what we intend them to do, but as our expressions get more and more ambiguous, and higher scale and operating on more of the world. And so what do you think of the argument, \"Highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous?\"

\*\*0:15:22.1 Interviewee:\*\* Yeah, if I understand what you\'re saying, that is very clear to me. That\... specifying\... If you just hope to give a short description of what you want, you\'ll most likely fail.

\*\*0:15:35.9 Vael:\*\* Even a long definition would be fine. I just want there to be any system of putting what we want into the AI.

\*\*0:15:46.0 Interviewee:\*\* Yeah, so I think there might be a safe way to do that, but it might require a lot of interaction and will be hard. I have thought a little bit about that, because I was thinking about the question of how do we\... So if we want to use these systems for interfacing with people, then we have to let them handle ambiguity in a way that\'s similar to how people do. And in cognitive science, there is a lot of knowledge about the mechanisms that people use to resolve ambiguity, like there is context, prior knowledge, common ground, and all that. Some of that could be replicated now to fully align the way that AI resolves ambiguity with how people do it. We\'ll also probably take a lot of work in understanding how people do it, which is not there yet.

\*\*0:16:50.3 Vael:\*\* Interesting. Interesting. Do you think that we\'ll have to know enough about human psychology? Like, what fields need to advance in order for us to have systems that do what we intend them to do?

\*\*0:17:06.1 Interviewee:\*\* Yeah, if you\'re talking about aligning a system with human values, part of that problem is understanding human values and like how people agree on values. And there\'s just so much that we assume that we don\'t have to tell people, it\'s assumed that they were created in the same way and exposed to the same contexts. All that goes away with AI, or at least most of it. And I\'m not an expert in these fields, so I might not know the exact names. I guess pragmatic reasoning is one thing that we have a lot of high-level insights on how it works, but to operationalize it into an NLP system, for example, we\'re still very far away from that.

\[Person\] worked on this model for pragmatic reasoning, which can explain very neatly some very simple cases. Like, you have a few words and a few people, and some words describe these people. You have a three-by-three matrix that predicts exactly what people would guess the word is referring to with very high accuracy. But just extending that to a sentence and the three people to be an image is a very complicated problem. So I think that is something that would need to advance a lot. Other fields of psychology that try to understand ambiguity like of cognitive psychology. Yeah. I don\'t know their names, but I think\... Oh well, yeah. I guess another one in linguistics that is relevant for this is human convention formation. Like when we are talking to a person over time, we develop these partner-specific conventions. That\'s very natural, enables very efficient communication. And if you\'re writing a description of what you want in a long form to an AI\-- Oh, if you\'re doing so for a person, you assume that the person is picking up on what we want and forming the same conventions over time. So at first you might say, \"I want to optimize the revenues for XYZ company.\" And then later you might say \"the company,\" dropping the XYZ, but you assume that\-- the only company that I mentioned was XYZ, so the only interpretation possible for a company is attribute the XYZ company. Well, that\'s because humans form conventions that way, but if you have kind of an adversary AI, then suddenly that\'s not an assumption you can build off of anymore. So the ways in which humans communicate with conventions, with ambiguity and all the computational tools that we need to do that are open problems, if that makes sense.

\*\*0:20:05.3 Vael:\*\* Yeah. That does seem right. Thanks. Alright, so this next question is still about the CEO AI. So imagine we have the CEO AI that is capable of multi-step planning and has a model of itself in the world. So it\'s modeling other people modeling it because that seems really important in order to have a CEO. So it\'s making these plans for the future and it\'s trying to optimize for profit with the constraints I\'ve mentioned. And it\'s noticing that some of its plans fail because the humans shut it down. And so we built into this AI and its loss function that it has to get human approval for things because it seems like a basic safety measure. (Interviewee: \"Sorry, it has to get what?\") It has to get approval for any action from humans, like a stakeholder-type thing. And so the humans have asked for a one-page memo from the AI just to discuss what it\'s doing. And so the AI is thinking about what to put in this memo, and it\'s like, \"Maybe I should omit some relevant information that the humans would want because that would reduce the chance that the AIs \[note: I meant to say \"humans\"\] would shut me down and increase the likelihood of my plans succeeding, of optimizing my goal.\" And so in this case, we\'re not building self-preservation into the AI, what we\'re doing is having an AI that\'s an agent that is optimizing over a goal. And so instrumentally, it has the goal of self-preservation. So what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous?\"

\*\*0:21:30.7 Interviewee:\*\* Yeah. I think if we get systems that have that capability to do the things that we said in the beginning, like planning, in the most extreme form, then I agree with this statement.

\*\*0:21:44.6 Vael:\*\* Interesting. Do you think we\'ll ever\-- hm, so\... We were talking about whether you thought that we would get to that point. What was the answer to that?

\*\*0:21:54.9 Interviewee:\*\* The answer is that I think so, although I\'m not sure to what extent. I\'m not sure if the most powerful possible version will actually happen in 30-40 years. But I think there\'s, there\'s like a small chance, but\-- it\'s not impossible.

\*\*0:22:11.1 Vael:\*\* Interesting. So yeah, within our lifetimes. Yeah, so I\'m worried about the possibility that we don\'t get the alignment problem right per se. And we end up with systems that are now optimizing against humans, which seems bad, especially if they are smarter than humans. Yeah. How likely do you think this sort of scenario would happen?

\*\*0:22:34.7 Interviewee:\*\* The scenario of having super-capable AI that is---

\*\*0:22:38.3 Vael:\*\* That has an instrumental incentive to do self-preservation or other instrumental incentives like power-seeking or acquiring resources or improving itself.

\*\*0:22:48.1 Interviewee:\*\* Yeah, I think it is small but not zero.

\*\*0:22:57.9 Vael:\*\* Yeah. How bad do you think that would be if such a thing happened?

\*\*0:23:03.4 Interviewee:\*\* Yeah, I think pretty bad.

\*\*0:23:05.9 Vael:\*\* Pretty bad. Yeah. Like what would happen?

\*\*0:23:10.4 Interviewee:\*\* Yeah, it\'s not exactly clear to me how that would look like. But it is a little scary to think about humans losing control in some ways. The worry is exactly that we can\'t\... like if we\'re not under control, we can\'t know exactly what comes.

\*\*0:23:30.6 Vael:\*\* Yeah. That seems true. Have you heard of the\-- What does AI safety mean to you? That\'s my first question.

\*\*0:23:39.3 Interviewee:\*\* AI safety. I understand it as a field that\... When I think of AI safety, the first defining problem that comes for me is AI alignment. This question, \"How do you specify your goals in a way that aligns with human values?\" and all that.

\*\*0:24:00.6 Vael:\*\* That makes sense. Yeah. When did you start learning about AI alignment or when did you start caring about it?

\*\*0:24:05.9 Interviewee:\*\* Ah, it was mostly when I came to the \[position\] at \[university\], because I really wasn\'t that much of an AI person before. I kind of incidentally got into AI because it gave me tools to think about problems that were hard before. And I happened to get a bit more involved with the AI safety community. \[Friend\'s name\], you might know how him. (Vael: \"Ah! Nice.\") We\'re very close friends. We lived together for almost three years, and had a lot of conversations.

\[short removed segment, discussing situation and friend\]

\*\*0:25:09.2 Vael:\*\* Yeah. Some visions that I have, why it could be bad, is if we have an AI that\'s like very powerful, then if it\'s incentivized to get rid of humans, then maybe it\'s intelligent enough to do synthetic biology against humans, or nanotechnology or whatever, or maybe just increase the amount of pollution. I don\'t know. I just feel like there\'s ways in which you could make the environment uninhabitable for humans via like putting things in the water or in the air. It\'d be pretty hard to do worldwide, but seems maybe possible if you\'re trying. Yeah. Also we\'ve got like misuse and AI-assisted war, and like maybe if we put AI in charge of like food production or manufacturing production, then we can have a lot of correlated supply chain failures. That\'s another way, but I don\'t know if that would kill everyone of course, but that could be quite bad.

\*\*0:26:12.5 Interviewee:\*\* Yeah, that is something that I worry more about in the short-term, which is AI in the hands of\... Even currently, AI can already be pretty bad with like spreading misinformation. That certainly happened here in \[country\], but in \[other country\] in the last elections, where this really bad president got elected, he used a lot of misinformation campaigns funded by a lot of people. And it\'s super hard to track because it\'s on WhatsApp and it\'s like the double-edged sword, the sword of privacy. It aligns with things that we care about, but which also allows these campaigns to run at massive scale and not be tracked. Even currently, AI personalizing wordings and messages to different people at large scale could be really pretty \[hard to parse, \"good\"?\].

\*\*0:27:20.1 Vael:\*\* Got it. Do you work on AI alignment stuff?

\*\*0:27:26.0 Interviewee:\*\* So currently, I\'m working more on the education side of research. I still keep in the back of my head some of the problems about the ambiguity thing. We did have one paper last year that started building some of those ideas but I\'m not actively working on that. But it\'s still like a realm of problems that I think is important for a number of reasons.

\*\*0:27:56.2 Vael:\*\* Yeah. That makes sense. What would cause you to work on it?

\*\*0:28:04.4 Interviewee:\*\* Sorry, what what would make me work on it?

\*\*0:28:08.3 Vael:\*\* Yeah, like in what circumstance would you be like: Whelp, it is now five year later, it is now one year later, and I happen to be somehow working on some element of the AI alignment problem. Or AI safety or something. How did that happen?

\*\*0:28:27.8 Interviewee:\*\* Yeah, it\'s interesting. From my trajectory, I have always tried to work on the thing that I feel is not being done and that I\'m in a certain position to do. And that has over time shifted towards education for a number of reasons. And I don\'t know exactly where that comes from, but I still feel that I\'m more able to contribute there than with other AI problems. Although I guess that kind of thing is hard to assess, but it\'s still my feeling on when deciding what things to work on.

\*\*0:29:19.9 Vael:\*\* Hm, so it sounds like there\'s maybe something like\... Okay, so I don\'t quite know if it\'s a matter of whether the education bit is neglected by the world, or whether it\'s like, you feel like it would be a better fit per se, or both?

\*\*0:29:35.6 Interviewee:\*\* Yeah, it\'s the combination of both. Like if it\'s neglected. It is important, important in the short-term. And I think there are like things that I could do in the next five years. If the plan that I have for the \[position\] I have works out, I think it would be great. Like we would have tools that people would be able to use, and they need technical advances that are in the intersection of things that I care about or know for very weird reasons. But I feel like I\'m in a position to make progress there and very immediately. So yeah, I just feel very urged to see how that goes.

\*\*0:30:21.3 Vael:\*\* Yeah. Makes sense. Yeah, do you ever think you\'ll go get into the AI alignment thing or you\'ll just be like, \"Woo, seems good for someone to work on it?\" I get that\-- that\'s an impression that I get.

\*\*0:30:32.6 Interviewee:\*\* Uh huh, yeah. I think I\'m getting a taste of it through \[friend\]. We actually wrote this project together two months ago. Okay, I actually met with \[friend\] to discuss this a few times but I have no idea where it\'s exactly came from, I think from someone at LessWrong that posted like a challenge. So it\'s a program called ELK, which I forgot even what ELK means, but I met with \[friend\] and we ended up coming up with some ideas, and he submitted it. \[\...\]

\*\*0:31:16.9 Interviewee:\*\* So I guess I\'m getting a little bit of flavor of the kind of work that that entails. I still don\'t feel exactly comfortable in doing it myself just because it requires this very hypothetical and future reasoning that I\'m not used to, not that I couldn\'t. But I understand that thinking about that kind of problem today requires you to think of systems that we can\'t use today and we can\'t run and test today. And I\'m very used to ideas that I can write down the code and see how they work and then make progress from there. So I don\'t know, my opinion and comfort might change.

\*\*0:32:18.5 Vael:\*\* Yeah, I also encountered this where I was like, \"I don\'t wanna think about far-future programs or far-future things, where there\'s no feedback loops and it\'s very hard to tell what\'s happening and there\'s only a limited number of people, and then\-- now there\'s money, but there wasn\'t necessarily very much money before\-- and there\'s not really journals, and everything\'s pre-pragmatic.\" And then, I think I was won over by the argument that, I don\'t know, I think existential risk is probably the thing that I\'ll just work on for most of my life because it seems like really important, and like, probable. But then I tried technical research for a while, and I thought it was not my cup of tea. And then I got into AI community building, which is what I\'m kind of currently doing, and trying to get more people to become AI technical researchers. But not me! It\'s been a good fit in some sense.

\*\*0:33:10.1 Interviewee:\*\* I see, that makes sense.

\*\*0:33:11.1 Vael:\*\* Yeah, and I think it\'s pretty interesting as well. Well, the current thing I\'m doing is talking to AI researchers and being like, \"What do you think of things? What about this argument?\" \...Cool. Well, we\'ve got a little bit of time left, so, see if I can ask some of my more unusual questions, which are\... How much do you think about policy and what do you think about policy related to AI?

\*\*0:33:34.6 Interviewee:\*\* Yeah, I think I\'ve\... I think a lot about policy in general, about just related to AI. I\'m currently at this phase in my life where I think a lot of the immediate problems that we have, unfortunately in a sense, we\'ll have to resolve by good politics, and I\'m not exactly sure yet how to do that. Because one other thing that I think about, when I think about AI safety, is that one thing is to have the technical ability to do certain things. The other is to get the relevant players to implement them. So, if there is an AI safety mechanism that can like limit the reach of Google\'s super AI that will do all things, but it also means that it will decrease revenue to some extent\... I\'m not sure the decision that they will make which would be to implement the safe version. Think about VC\'s that are sitting in these boards and pushing decisions for the company. I\'m not sure if all of them actually care about this and realize that it\'s a problem. So I think part of the solution is also on the political side, of like us as a society sitting down and deciding, \"What do we gotta do?\" More than just the technical proposal. I\'ve decided that this year, I will try to have some political involvement in \[country\], which is a place where I can. I\'m currently in the planning phase, but I\'ve committed to doing something this year, because we have elections this year.

\*\*0:35:39.2 Vael:\*\* Interesting, yeah. I think AI governance is very, very important. I think the community\-- the AI safety community also, like\... or, I don\'t know, the Effective Altruism community \[hard to parse\] a lot in their listings. Alright, question: if you could change your colleagues\' perceptions of AI, what attitudes or beliefs would you want them to have? So what beliefs do they currently have and how would you want those to change?

\*\*0:36:05.9 Interviewee:\*\* I\'m not sure if this is the kind of attitude that you\'re talking about, but one attitude that I really don\'t like and is very prevalent in papers is that people kind of self-identify with the method that they\'re proposing so much that they call it \"our method\". In the sense that the goal should not necessarily be to show off better numbers, and show a curve where the color that\'s associated with you is like higher than the other. But it should really be to understand what\'s going on, right? If we have a system that has some sort of behavior that\'s desirable, that\'s awesome, and we should also look at how it maybe doesn\'t. Like people include a lot of cases in their papers where, \"Oh look, the model is doing this.\" But then when we actually try out the model, it\'s very easy to find cases where it doesn\'t or it fails in weird ways. And the reluctance to put that comes from this attachment from like, \"Oh, this is like me, but in form of a method that is written in this paper.\" And I wish that people didn\'t have that attitude, that they were more detached and scientific about it.

\*\*0:37:24.9 Vael:\*\* Yeah, makes sense. Cool. And then my last question is, why did you choose this interview, and how has this interview been for you?

\*\*0:37:32.6 Interviewee:\*\* Yeah, it\'s been really fun. I had thought of some of the questions before, and not of others. Yeah, the exercise that I maybe haven\'t done that much is to think super long-term, like 50-100 years in the future. I thought about it some but I don\'t think I have the right tools yet to think about them, so I get in this hard-to-process state. Yeah, so I came mostly out of curiosity, I didn\'t know exactly what to expect. If you don\'t mind sharing, but if you do, that\'s okay, how did you pick the people to interview?

\*\*0:38:28.3 Vael:\*\* Yeah, so this is a pretty boring answer, but I originally selected people who\'d submitted papers to NeurIPS or ICML in 2021, and like yeah, then some proportion of people replied back to me.

\*\*0:38:44.5 Interviewee:\*\* I see.

\*\*0:38:47.2 Vael:\*\* Yeah, great, cool. That makes a lot of sense. Thanks you so much for your time. I will send you the money and some additional resources if you\'re curious about it. I\'m sure \[friend\] has already shown you some, but if you\'re curious. And thanks so much for doing this.

\*\*0:39:00.7 Interviewee:\*\* Yeah, no, definitely. Thank you for the invite, this was fun.

\*\*0:39:05.1 Vael:\*\* All right. Bye.

\*\*0:39:05.8 Interviewee:\*\* Bye.


Sino-Western cooperation in AI safety _ Brian Tse _ EA Global - San Francisco 2019-by Centre for Effective Altruism-video_id 3qYmLRqemg4-date 20190829

# Brian Tse Sino-Western cooperation in AI safety - EA Forum

\_International cooperation is essential if we want to capture the benefits of advanced AI while minimizing risk. Brian Tse, a policy affiliate at the\_ [\_Future of Humanity Institute\_](https://www.fhi.ox.ac.uk/)\_, discusses concrete opportunities for coordination between China and the West, as well as how China’s government and technology industry think about different forms of AI risk.\_

\_A transcript of Brian’s talk, which we have edited lightly for clarity, is below. You can also watch it on\_ [\_YouTube\_](https://www.youtube.com/watch?v=3qYmLRqemg4&list=PLwp9xeoX5p8MqGMKBZK7kO8dTysnJ-Pzq&index=8&t=5s) \_or read it on\_ [\_effectivealtruism.org\_](https://effectivealtruism.org/articles/brian-tse-sino-western-cooperation-in-ai-safety)\_.\_

## The Talk

It has been seven decades since a nuclear weapon has been detonated.

![](https://images.ctfassets.net/ohf186sfn6di/1ZoSlAdq1BLLV2u3X2gwRX/6f45b25b638e071a29624c15b054b7b4/Tse1.png)

For almost four decades, parents everywhere have not needed to worry about their children dying from smallpox.

![](https://images.ctfassets.net/ohf186sfn6di/39RqVyvhzwg4gVXYHJX5Zu/b5225eb44a1a2b74551a7672e03133c4/TSe2.png)

The ozone layer, far from being depleted to the extent once feared, is expected to recover in three decades.

![](https://images.ctfassets.net/ohf186sfn6di/6ERXuBj7tDEMnOaUss0259/da778454b8bddf2462e077850778b909/Tse3.png)

These events — or non-events — are among humanity’s greatest achievements. They would not have occurred without cooperation among a multitude of countries. This serves as a reminder that international cooperation can benefit every country and person.

Together, we can achieve even more. In the next few decades, AI is poised to be one of the most transformative technologies. In the Chinese language, there is a word, “wēijī,” which is composed of two characters: one meaning danger and the other opportunity.

![](https://images.ctfassets.net/ohf186sfn6di/3DxtZ8X7nQcLbMx2L59qoU/2cc5e45059559bc05abc333dafd5a5d9/Tse5.png)

Both characters are present at this critical juncture. With AI, we must seek to minimize dangers and capture the upsides. Ensuring that there is robust global coordination between stakeholders around the world, especially those in China and the West, is critical in this endeavor.

So far, the idea of nations competing for technological and militarist supremacy has dominated the public narrative.

![](https://images.ctfassets.net/ohf186sfn6di/7be6c6zbbLM1s4z5ke6b2z/6192b8c68de6390b9f98f972adc8f7c7/Tse6.png)

When people talk about China and AI, they always invoke the country's ambition to become the world leader in AI by 2030. In contrast, there is very little attention paid to China’s call for international collaboration in security, ethics, and governance of AI, which are areas of mutual interest. I believe it is a mistake to think that we must have either international cooperation or international competition. Today, some believe that China and the U.S. are best described as strategic adversaries.

I believe we must deliberately use new concepts and terms that capture the two countries’ urgent need to cooperate — not just their drive to compete.

![](https://images.ctfassets.net/ohf186sfn6di/2kCYHULhiJne66glDPGQU9/636099b850e41e7b281ed255fc1f6af2/Tse7.png)

[Joseph Nye](https://en.wikipedia.org/wiki/Joseph\_Nye), well-known for coining the phrase “soft power,” has suggested that we use “cooperative rivalry” to describe the relationship. Graham Allison, the author of [\_Destined For War\_](https://www.amazon.com/Destined-War-America-Escape-Thucydidess-ebook/dp/B01IAS9FZY), has proposed the word “coopertition,” allowing for the simultaneous coexistence of competition and cooperation.

In the rest of my talk, I'm going to cover three areas of AI risk that have the potential for global coordination: accidents, misuse, and the race to develop AI.

![](https://images.ctfassets.net/ohf186sfn6di/2kFFTHdAbjywRkLicjbzNb/dedb9ac8c05d8ff5381ac3d6308a3de0/Tse8.png)

For each of these risks, I will talk about their importance and feasibility for coordination. I will also make some recommendations.

## The risk of AI accidents

As the deployment of AI systems has become more commonplace, the number of AI-related accidents has increased. For example, on May 6, 2010, the Dow Jones Industrial Average experienced a sudden crash of $1 trillion known as the “[Flash Crash](https://en.wikipedia.org/wiki/2010\_Flash\_Crash).”

![](https://images.ctfassets.net/ohf186sfn6di/1rG4RIhcytxpLCSSLMAOqa/b2d163215231d0d9390cfeae5833a7b2/Tse9.png)

It was partly caused by the use of high-frequency trading algorithms. The impact immediately spread to other financial markets around the world.

As the world becomes increasingly interdependent, as with financial markets, local events have global consequences that demand global solutions. The participation of \[the Chinese technology company\] [Baidu](https://en.wikipedia.org/wiki/Baidu) in the [Partnership on AI](https://www.partnershiponai.org/) is an encouraging case study of global collaboration.

![](https://images.ctfassets.net/ohf186sfn6di/1gj1V6eO47Xrmd0r85ZzHs/4c426b0f2673bb439f4cc24a2b9783fe/Tse10.png)

In a press release last year, Baidu said that the safety and reliability of AI systems is critical to their mission and was a major motivation for them to join the consortium. The \[participating\] companies think autonomous vehicle safety is an issue of particular importance.

China and the U.S. also seem to be coordinating on nuclear security. One example is the [Center of Excellence on Nuclear Security in Beijing](http://www.chinadaily.com.cn/world/2016xivisitczech/2016-04/02/content\_24248877.htm), which is by far the most extensive nuclear program to receive direct funding from both the U.S. and Chinese governments.

![](https://images.ctfassets.net/ohf186sfn6di/5RGogAMdz2TdPJhCF2AAL1/0aabe35cf3dfa2ec2dbf2c94f6ba2d70/Tse11.png)

It focuses on building a robust nuclear security architecture for the common good. A vital feature of this partnership is an intense focus on exchanging technical information, as well as reducing the risk of accidents.

It is noteworthy that, so far, China has emphasized the need to ensure the safety and reliability of AI systems. In particular, the [Beijing AI Principles](https://www.baai.ac.cn/blog/beijing-ai-principles) and the [Tencent Research Institute](https://www.tencent.com/en-us/abouttencent.html) have highlighted the risks of [AGI systems](https://en.wikipedia.org/wiki/Artificial\_general\_intelligence).

With our current understanding of AI-related accidents, I believe Chinese and international stakeholders can collaborate in the following ways:

![](https://images.ctfassets.net/ohf186sfn6di/DC2KV5iNBDioXIusptFyd/f5a1360e3dd8facfe81b1938eaed181b/Tse13.png)

1\. Researchers can attend the increasingly popular AI safety workshops at some of the major machine learning conferences.

2\. Labs and researchers can measure and benchmark the safety properties of reinforcing learning agents based on efforts by organizations and safety groups like that of [DeepMind](https://deepmind.com/).

3\. International bodies, such as [ISO](https://www.iso.org/news/ref2336.html), can continue their efforts to set technical standards, especially around the reliability of machine learning systems.

4\. Lastly, alliances such as the [Partnership on AI](https://www.partnershiponai.org/) can facilitate discussions on best practices (for example, through \[the Partnership’s\] [Safety-Critical AI Working Group](https://www.partnershiponai.org/wp-content/uploads/2018/07/Safety-Critical-AI\_-Charter.pdf)).

## The risk of AI misuse

Even if we can mitigate the unintended accidents of AI systems, there is still a possibility that they'll be misused.

![](https://images.ctfassets.net/ohf186sfn6di/4aUajWe2ILsEyBAwNU7QBg/cc39fa462cdf7b93933f828233ac47f2/Tse14.png)

For example, earlier this year, [OpenAI](https://openai.com/) decided not to release the training model of [GPT-2](https://openai.com/blog/better-language-models/), which \[can generate language on its own\], due to concerns that it might be misused to impersonate people, create misleading news articles, or trick victims into revealing their personal information. This reinforces the need for global coordination; malicious actors from anywhere could have gained access to the technology behind GPT-2 and deployed it in other parts of the world.

In the field of cybersecurity, there was a relevant case study of the global response to security incidents.

![](https://images.ctfassets.net/ohf186sfn6di/5rRWmUfPFqxVKSNXGUfUCd/59cc69464a545d851ad1668b6087606f/Tse15.png)

In 1989, [one of the first computer worms](https://en.wikipedia.org/wiki/WANK\_(computer\_worm) attacked a major American company. The incident prompted the creation of the international body [FIRST](https://www.first.org/) to facilitate information-sharing and enable more effective responses to future security incidents. Since then, FIRST has been one of the major institutions in the field. It currently currently lists ten American and eight Chinese members, including companies and public institutions.

Another source of optimism is the growing research field of \[adversarial images\]. These are small input samples that have been moderated slightly to cause machine learning classifiers to misclassify them \[[e.g., mistake a toy turtle for a gun](https://www.theverge.com/2017/11/2/16597276/google-ai-image-attacks-adversarial-turtle-rifle-3d-printed)\].

![](https://images.ctfassets.net/ohf186sfn6di/lJbmOVEwzZv1hXL2NAJr4/a53ea2be66331f8d7ddfe5b1fc821a47/Tse16.png)

This issue is highly concerning, because \[adversarial images\] could be used to attack a machine learning system without the attacker having access to the underlying model.

Fortunately, many of the leading AI labs around the world are already working hard on this problem. For example, [Google Brain organized a competition on this research topic](https://arxiv.org/pdf/1804.00097.pdf), and the team from China’s Tsinghua University won first place in both the “attack” and “defense” tracks of the competition.

Many of the Chinese AI ethical principles also cover concerns related to the misuse of AI . One promising starting point of coordination between Chinese and foreign stakeholders, especially the AI labs, involves publication norms.

![](https://images.ctfassets.net/ohf186sfn6di/4QBUJ1V0ErUUomuohKULGk/40d8ca1ef8b554c8fecf438721dae469/Tse18.png)

Following the \[controversy around\] OpenAI’s GPT-2 model, the Partnership on AI organized a seminar on the topic of research openness. There was no immediate consideration of whether the AI community should restrict research openness. However, they did agree that if the AI community moves in that direction, review parameters and norms should be standardized across the community (presumably, on a global level).

## The risk of competitively racing to develop AI

The third type of risk that I'm going to talk about is the risk from racing to develop AI.

![](https://images.ctfassets.net/ohf186sfn6di/2fpkY3dsOB7gOWMpvxDco3/2af5c51ca0daa08a461c36ce865d3e97/Tse19.png)

Under competitive pressure, AI labs might put aside safety concerns in order to stay ahead. Uber’s self-driving car crash in 2018 illustrates this risk.

![](https://images.ctfassets.net/ohf186sfn6di/1LWMvi8tnoS3SFgPXZbD3q/e6d879a980bf6424be747400bc59535f/Tse20.png)

When it happened, commentators initially thought that the braking system was the culprit. However, further investigation showed that the victim was detected early enough for the emergency braking system to have worked and prevented the crash.

So what happened? It turned out that the engineers intentionally turned off the emergency braking system because they were afraid that its extreme sensitivity would make them look bad relative to their competitors. This type of trade-off between safety and other considerations is very concerning, especially if you believe that AI systems will become increasingly powerful.

This problem is going to be even more heuristic in the context of international security. We should seek to draw lessons from historical analogs.

![](https://images.ctfassets.net/ohf186sfn6di/2tbajnqeEQvQ4etm0A3N94/3293a000d93e4954da70500bd95011ea/Tse21.png)

For example, the report “[Technology Roulette](https://s3.amazonaws.com/files.cnas.org/documents/CNASReport-Technology-Roulette-Final.pdf)” by Richard Danzig discusses the norm of “no first use” and its contribution to stability during the nuclear era. Notably, China was the first nuclear-weapon state to adopt such a policy back in 1964, with varying degrees of success. Other nations have also used the norm to moderate the proliferation and use of various military technologies, including blinding lasers and offensive weapons from outer space.

Now, with AI as a general-purpose technology, there is a further challenge: How do you specify and verify that certain AI technologies haven’t been used? On a related note, the Chinese nuclear posture has been described as a defense-oriented one. The question with AI is: Is it technically feasible for parties to differentially improve defensive capabilities, rather than offensive capabilities, thereby stabilizing the competitive dynamics? I believe these are still open questions.

Ultimately, constructive coordination depends on the common knowledge that there is this shared risk of a race to the bottom with AI. I'm encouraged to see increasing attention paid to the problem on both sides of the Pacific.

![](https://images.ctfassets.net/ohf186sfn6di/5qMHgDr06grzXgXl5wVR0U/71da82e0ca96e1e457c551a48ab78b3d/Tse22.png)

For example, [Madame Fu Ying](https://en.wikipedia.org/wiki/Fu\_Ying), who is chairperson of the National People’s Congress Foreign Affairs Committee in China and an influential diplomat, has said that Chinese technologists and policymakers agree that AI poses a threat to humankind. At the World Peace Forum, she further emphasized that the Chinese believe we should preemptively cooperate to prevent such a threat.

The [Beijing AI Principles](https://www.baai.ac.cn/blog/beijing-ai-principles), in my view, provide the most significant contribution from China regarding the need to avoid a malicious AI race. And these principles have gained support from some of the country’s major academic institutions and industry leaders. It is my understanding that discussions around the [Asilomar AI Principles](https://futureoflife.org/ai-principles/), the book [\_Superintelligence\_](https://www.amazon.com/dp/B00LOOCGB2/) by Nick Bostrom, and warnings from Stephen Hawking and other thinkers have all had a meaningful influence on Chinese thinkers.

Building common knowledge between parties is possible, as illustrated by the [Thucydides Trap](https://foreignpolicy.com/2017/06/09/the-thucydides-trap/).

![](https://images.ctfassets.net/ohf186sfn6di/1Vzb0VkstLArpEzq3OyryU/ef32769dd3ea9b85336a31269a11b57b/Tse24.png)

Coined by the scholar Graham Allison, The Thucydides Trap describes the idea that rivalry between an established power and a rising power often results in conflict. This thesis has captured the attention of leaders in both Washington, D.C. and Beijing. In 2013, President Xi Jinping told a group of Western visitors that we should cooperate to escape from the Thucydides Trap. In parallel, I think it is important for leaders in Silicon Valley — as well as in Washington, D.C. and Beijing — to recognize this collective problem of a potential AI race to the precipice, or what I might call “the Bostrom Trap.”

With this shared understanding, I believe the world can move in several directions. First, there are great initiatives, such as the [Asilomar AI Principles](https://futureoflife.org/ai-principles/), which can help many of the signatories \[adhere to\] the principle of arms-race avoidance.

![](https://images.ctfassets.net/ohf186sfn6di/pxytr1hFG0skvo5KfSsYW/5f81f2483fefc4fe864fa93597295fb1/Tse25.png)

Expanding the breadth and depth of this dialogue, especially between Chinese and Western stakeholders, will be critical to stabilize expectations and foster mutual trust.

Second, labs can initiate AI safety research collaborations across borders.

![](https://images.ctfassets.net/ohf186sfn6di/5YtD8WaEd5XvT8eapOQf2R/7cb760847d9b77f3c509645f89301746/Tse26.png)

For example, labs could collaborate on some of the topics laid out in the seminal paper “[Concrete Problems Of AI Safety](https://arxiv.org/abs/1606.06565),” which was itself a joint effort from multiple institutions.

Lastly — and this is also the most ambitious recommendation — leading AI labs could consider adopting the policies in the [OpenAI Charter](https://openai.com/charter/).

![](https://images.ctfassets.net/ohf186sfn6di/5NQpoh8WeGaajjsWpmqS0J/82c00dbe6484071b2fad6b3943e8751b/Tse27.png)

The charter claims that if a value-aligned, safety-conscious project comes close to building AGI technology, OpenAI will stop competing and start assisting with that project. This policy is an incredible public commitment, as well as a concrete mechanism in trying to reduce these undesirable \[competitive\] dynamics.

Throughout this talk, I have not addressed many of the complications involved in such an endeavor. There are considerations such as industrial espionage, civil/military fusion, and civil liberties. I believe each of those topics deserve a nuanced, balanced, and probably separate discussion, given that I will not be able to do proper justice to them in a short presentation like this one. That said, on the broader challenge of overcoming political tension, I would like to share a story.

Some believe the [Cuban Missile Crisis](https://en.wikipedia.org/wiki/Cuban\_Missile\_Crisis) had a one-in-three chance of resulting in a nuclear war between the U.S. and the Soviet Union. After the crisis, President J. F. Kennedy was desperately searching for a better way forward.

![](https://images.ctfassets.net/ohf186sfn6di/64tMEIgVMRJ7yccKWLl3Rq/9970f46aa5c66f674cd5ca5c57700bef/Tse29.png)

Before he was assassinated, in one of his most significant speeches about international order, he proposed the strategic concept of a world safe for diversity. In that world, the U.S. and Soviet Union could compete rigorously, but only peacefully, to demonstrate whose value and system of governance might best serve the needs of citizens. This eventually evolved into what became “[détente](https://en.wikipedia.org/wiki/D%C3%A9tente),” a doctrine that contributed to the easing of tension during the Cold War.

In China, there is a similar doctrine, which is “harmony in diversity.” \[Brian says the word in Mandarin.\]

![](https://images.ctfassets.net/ohf186sfn6di/2061oezRrXIAkHDrjjpNAD/a8c794137426b41121695aa224f7c9e8/Tse30.png)

The world must learn to cooperate in tackling our common challenges, while accepting our differences. If we were able to achieve this during the Cold War, I believe we should be more hopeful about our collective future in the 21st century. Thank you.

## Q&A

\*\*Nathan Labenz \[Moderator\]:\*\* I think the last time I saw you was just under a year ago. How do you think things have gone over the last year? If you were an attentive reader of the \_New York Times\_, you would probably think things are going very badly in US/China relations. Do you think it's as bad as all that? Or is the news maybe hyping up the situation to be worse than it is?

\*\*Brian:\*\* It is indeed worrying. I will add two points to the discussion. One: we're not only thinking about coordination between governments. In my talk, I focused on state-to-state cooperation, but I mentioned a lot of potential areas of collaboration between AI labs, researchers, academia and civil society. And I believe that the incentive and the willingness to cooperate between those stakeholders are there. Second, my presentation was meant to be forward-looking and aspirational. I was not looking at the current news. I was thinking that if in five to 10 years, or even 20 years, AI systems become increasingly advanced and powerful — which means there could be tremendous upsides for everyone to share, as well as downsides to worry about — the incentive to cooperate, or at least aim for “coopertition,” should be there.

It could be interesting to think about game theory. I won’t go into the technical details. But the basic idea is that if there are tremendous upsides and also shared downsides for some number of parties, then it is more likely that those parties will be willing to cooperate instead of just compete.

\*\*Nathan:\*\* A question from the audience: Do you think that there's any way to tell right now whether the U.S. or the West (however you prefer to think about that), has an edge over China in developing AI? And do you think that there are political or cultural differences that contribute to that, if you think such a difference exists?

\*\*Brian:\*\* Just in terms of the potential for developing capable systems? We are not talking about safety and ethics, right?

\*\*Nathan:\*\* You can interpret the question \[how you like\].

\*\*Brian:\*\* Okay. I will focus on capabilities. Currently, it is quite clear to me that China is nowhere near the U.S. in terms of overall AI capabilities. People have argued at length. I would add a few things.

If you look at the leadership structure of Chinese AI companies — for example, [Tencent](https://www.tencent.com/en-us/abouttencent.html) — and some of the recent developments, it seems like the incentive to develop advanced and interesting theoretical research is not really there. Chinese AI companies are much more focused on products and near-term profit.

One example I would give is the Tencent AI lab director, Dr. Tong Zhang, who was quite interested in ideas relevant to AGI and worked at Tencent for two years. He decided to leave the AI lab earlier this year and is now going back to academia. He is joining the Hong Kong University of Science and Technology as a faculty member. Even though he didn't explicitly mention the reason \[for his departure\], people think that the incentive to develop long-term, interesting research is not there at Tencent or, honestly, at many of the AI companies.

Another point I will raise is this: If you look at some of the U.S. AI labs — for example, [FAIR](https://engineering.fb.com/ai-research/fair-fifth-anniversary/) or [Google Brain](https://x.company/projects/brain/) — the typical structure is that you have two research scientists and one research engineer on a team. The number could be greater, but the ratio is usually the same. But the ratio of research scientists to research engineers is the opposite for Chinese AI companies. There, you have one research scientist and two research engineers, which implies that they are much more focused on putting their research ideas into practice and applications.

\*\*Nathan:\*\* That's a surprising answer to me because I think that the naive, “\_New York Times\_ reader” point of view would be that the Chinese government is way better than the U.S. government in terms of long-term planning and priority-setting. If you agree with that, how do you think that translates into a scenario where the Chinese mega companies are maybe not doing as much as the American companies?

\*\*Brian:\*\* I think the Chinese model is still interesting from a long-term, mega-project perspective. But there is variance in terms of what type of mega projects you are talking about. If you're talking about railways, bridges, or infrastructure in general, the Chinese government is incredibly good at that. You can construct a lot of buildings in just days, and I think that it takes the U.S., UK, and many other governments years. But they are engineering projects. We're not talking about Nobel Prize-winning types of projects. I think that's really the difference.

There is some analysis on where the top AI machine learning researchers are working and all of them are in the U.S. But if you look at pretty good researchers — potentially Alan Turing Prize-winning researchers — then yes, China has a lot of them. I think we have to be very nuanced in terms of looking at what types of scientific projects we are talking about, and whether it's mostly about scientific breakthroughs or engineering challenges.

\*\*Nathan:\*\* Fascinating. A bunch of questions are coming in. I'm going to do my best to get through as many as I can. One question is about the general fracturing of the world that seems to be happening, or bifurcation of the world, into a Chinese sphere of influence (which might just be China, or maybe it includes a few surrounding countries), and then the rest of the world. We're seeing Chinese technology companies getting banned from American networks, and so on. Do you think that that is going to become a huge problem? Is it already a huge problem, or is it not that big of a problem after all?

\*\*Brian:\*\* It's definitely concerning. My main concern is the impact on the international research community. \[In my talk\], I alluded to the international and interconnected community of research labs and machine-learning researchers. I believe that community will still be a good mechanism for coordinating on different AI policy issues —they would be great at raising concerns through the [AI Open Letter Initiative](https://futureoflife.org/ai-open-letter/), collaborating through workshops, and so on.

But this larger political dynamic might affect them in terms of Chinese scientists’ ability to travel to the U.S. What if they just can't get Visas? And maybe in the future, U.S. scientists might also be worried about getting associated with Chinese individuals. The thing I'm worried about is really this channel of communication between the research communities. Hopefully, that will change.

\*\*Nathan:\*\* You're anticipating the next question, which is the idea that individuals are maybe starting to become concerned that if they appear to be on either side of the China/America divide — if they appear too friendly — they'll be viewed very suspiciously and might suffer consequences from that. Do you think that is already a problem, and if so, what can individuals do to try to bridge this divide while minimizing the consequences that they might suffer?

\*\*Brian:\*\* It's hard to provide a general answer. It probably depends a lot on the career trajectories of individuals and other constraints.

\*\*Nathan:\*\* There’s a question about the Communist Party. The questioner assumes that the Communist Party has final say on everything that's going on in China. I wonder if you think that's true, and if it is, how do we work within that constraint?

\*\*Brian:\*\* In terms of international collaboration and what might be plausible?

\*\*Nathan:\*\* Is there any way to make progress without the buy-in of the Communist Party, or do you need it? And if you need it, how do you get it?

\*\*Brian:\*\* I think one assumption there is that it is bad to have involvement from the government. I think we need to try to avoid that — I can just smell the assumptions when people ask these types of questions. It is not necessarily true. I think there are ways that the Chinese government can be involved meaningfully. We just need to be thinking about what those spaces are.

Again, one promising channel would be AI safety conferences through academia. If Tsinghua University is interested in organizing an AI safety conference with potential buy-in from the government, I think that's fine, and I think it's still a venue for research collaboration. The world just needs to think about what the mutual interests are and, honestly, the magnitude of the stakes.

\*\*Nathan:\*\* At a minimum, the Communist Party has at least demonstrated awareness of these issues and seems to be thinking about them. I think we're a little bit over time already, so maybe just one last question. Do you see this competition/cooperation dynamic and potentially this race to the precipice dynamics getting repeated across a lot of things? There's AI, and obviously in an earlier era there was nuclear rivalry, which hasn't necessarily gone away either. We also saw this news item of the first [CRISPR-edited babies](https://www.sciencemag.org/news/2019/08/top-stories-untold-story-2018-s-crispr-babies-china-s-gene-edited-crops-and-new), and that was a source of a lot of concern for people who thought, "We're losing control of this technology." So, what's the portfolio of these sorts of potential race-dynamic problems?

\*\*Brian:\*\* I think these are relevant historical analogs, but what makes AI a little bit different is that AI is a general-purpose technology, or omni-use technology. It's used across the economy. It's a question of political and economic \[importance\], not just international security. It's not just a nuclear weapon or a space weapon. It’s everywhere. It's more like electricity in the industrial revolution.

One thing that I want to add, which is related to the previous question, is the response from Chinese scientists to the gene-editing incident. Many people condemned the behavior of the scientist \[responsible for the gene editing\] because he didn't \[comply fully\] with regulations and was just doing it at a small lab in the city. But what you can see there is this uniformity of an international response to the incident; the responses from U.S. scientists, UK scientists, and Chinese scientists were basically the same. There was an open letter to \_Nature\_, with hundreds and hundreds of Chinese scientists saying that this behavior is unacceptable.

What followed was that the Chinese government wanted to develop better regulations for gene editing and \[explore\] the relevant ethics. I think this illustrates that we can have a much more global dialogue about ethics and safety in science and technology. And in some cases, the Chinese government is interested in joining this global dialogue, and takes action in its domestic policy.