Alex Turner - Will powerful AIs tend to seek┬ápower-by Towards Data Science-video_id 8afHG61YmKM-date 20220119

# Alex Turner on Will Advanced AIs Tend To Seek Power by Jeremie Harris on the Towards Data Science Podcast

## Alex Turner on his spotlighted 2021 NeurIPS paper and why we should worry about the power-seeking behaviour of AI systems

Today’s episode is somewhat special, because we’re going to be talking about what might be the first solid quantitative study of the power-seeking tendencies that we can expect advanced AI systems to have in the future.

For a long time, there’s kind of been this debate in the AI safety world, between:

- People who worry that powerful AIs could eventually displace, or even eliminate humanity altogether as they find more clever, creative and dangerous ways to optimize their reward metrics on the one hand, and
- People who say that’s Terminator-bating Hollywood nonsense that anthropomorphizes machines in a way that’s unhelpful and misleading.

Unfortunately, recent work in AI alignment — and in particular, a spotlighted 2021 NeurIPS paper — suggests that the AI takeover argument might be stronger than many had realized. In fact, it’s starting to look like we ought to expect to see power-seeking behaviours from highly capable AI systems by default. These behaviours include things like AI systems preventing us from shutting them down, repurposing resources in pathological ways to serve their objectives, and even in the limit, generating catastrophes that would put humanity at risk.

As concerning as these possibilities might be, it’s exciting that we’re starting to develop a more robust and quantitative language to describe AI failures and power-seeking. That’s why I was so excited to sit down with AI researcher Alex Turner, the author of the spotlighted NeurIPS paper on power-seeking, and discuss his path into AI safety, his research agenda and his perspective on the future of AI on this episode of the TDS podcast.

Here were some of my favourite take-homes from the conversation:

- AI alignment is a very complicated problem that seems straightforward from a distance. For that reason, people — including AI researchers — can become convinced that it’s not worth worrying about because they simply haven’t engaged with the real argument for alignment risk. It’s just too easy to brush off concerns over the intrinsic risk posed by powerful AI systems in the future as “a Terminator-style scenario”, or “that thing Elon is always hyping up”, when in reality, there’s a substantial body of compelling work (and increasingly, even [experimental results](https://openai.com/blog/faulty-reward-functions/)) and research behind the AI risk argument.
- Alex’s paper demonstrates that optimal policies — policies that achieve the maximum reward in a wide range of environments — tend to seek power. Roughly speaking, “power-seeking” here means accessing states that lead them to have more options in the future. For example, getting turned off is a low-power state for an AI system, because its action space after being shut off is empty — it wouldn’t have any options. Based on Alex’s work, we can expect AI systems with optimal policies to avoid being turned off as a result.
- Alex’s paper comes with some caveats. First, its conclusions apply only to optimal policies, and don’t explicitly address imperfect ones. As a result, you might imagine that sub-optimal policies (which are probably more realistic) wouldn’t lead to the same power-seeking behaviour. But Alex’s follow-up work closes that loophole: it turns out that optimality isn’t required to generate power-seeking at all.
- A second caveat has to do with the assumptions Alex makes about the rewards AI systems receive as they navigate through their environments. Alex’s work explores the behaviour of AI agents who are exposed to a wide array of random reward distributions, and shows that in the typical case, those agents tend to seek power. But we might hope that human-designed reward distributions would be less risky, since they’d explicitly be engineered to be safe and beneficial. But not so fast, says Alex: reward engineering is notoriously difficult and counterintuitive. As many experiments have shown, even today’s AI systems don’t optimize for the things we want: they optimize for easy-to-measure proxies that would be downright dangerous if they were being used as reward metrics for agents more clever than human beings.

\_Editor’s note: Alex’s work on power-seeking is important, timely, and promising. If you’re interested in AI safety, or even if you’re an AI risk skeptic, I highly recommend reading and engaging with his work. I don’t think it’s possible to have an informed, skeptical view on AI risk without understanding power-seeking, and Alex’s arguments in particular. You can reach out to Alex by email at\_ [turneale@oregonstate.edu](mailto:turneale@oregonstate.edu).


README-by Vael Gates-date 20220509

# Interview with AI Researchers by Vael Gates

Table of Contents
=================

\*\*[Table of Contents](#table-of-contents) 1\*\*

\*\*[Interview Information](#interview-information) 2\*\*

> [Individually-selected](#individually-selected) 2
>
> [NeurIPS-or-ICML](#neurips-or-icml) 2

\*\*[Intended Script (all interviews)](#intended-script-all-interviews) 3\*\*

\*\*[Post-Interview Resources Sent To Interviewees](#post-interview-resources-sent-to-interviewees) 5\*\*

> [Master list of resources](#master-list-of-resources) 5

\*\*[Informal Interview Notes](#informal-interview-notes) 10\*\*

> [Thoughts from listening to myself doing these interviews](#thoughts-from-listening-to-myself-doing-these-interviews) 10
>
> [Content analysis](#content-analysis) 11

Interview Information
=====================

These interviews are associated with the LessWrong Post: [[Transcripts of interviews with AI researchers]{.ul}](https://www.lesswrong.com/posts/LfHWhcfK92qh2nwku/transcripts-of-interviews-with-ai-researchers).

(Please do not try to identify any interviewees from any remaining peripheral information.)

###

### Individually-selected

"Five of the interviews were with researchers who were informally categorized as 'particularly useful to talk to about their opinions about safety' (generally more senior researchers at specific organizations)."

- 7ujun

- zlzai

- 92iem

- 84py7

- w5cb5

###

### NeurIPS-or-ICML

"Six of the interviews were with researchers who had papers accepted at NeurIPS or ICML in 2021."

- a0nsf (this is the interview in which I most straightforwardly get through my questions)

- q243b

- 7oalk

- lgu5f

- cvgig

- bj9ne (language barriers, young)

Intended Script (all interviews)
================================

There was a fixed set of questions that I was attempting to walk people through, across all of the interviews. It's a sequence, so I generally didn't move onto the next core question until I had buy-in for the previous core question. The core questions were: "do you think we'll get AGI" (if yes, I moved on; if not I interacted with the beliefs there, sometimes for the entire interview), "\[alignment problem\]", and "\[instrumental incentives\]". I was reacting to the researchers' mental models in all cases. I was trying to get to all of the core questions during the allotted time, but early disagreements often reappeared if the interviewee and I didn't manage to reach initial agreement. I prioritized the core questions, and brought other questions up if they seemed relevant.

The questions (core questions are highlighted):

- "What are you most excited about in AI, and what are you most worried about? (What are the biggest benefits or risks of AI?)"

- "In at least 50 years, what does the world look like?"

- "When do you think we'll get AGI / capable / generalizable AI / have the cognitive capacities to have a CEO AI if we do?"

 - Example dialogue: "All right, now I\'m going to give a spiel. So, people talk about the promise of AI, which can mean many things, but one of them is getting very general capable systems, perhaps with the cognitive capabilities to replace all current human jobs so you could have a CEO AI or a scientist AI, etcetera. And I usually think about this in the frame of the 2012: we have the deep learning revolution, we\'ve got AlexNet, GPUs. 10 years later, here we are, and we\'ve got systems like GPT-3 which have kind of weirdly emergent capabilities. They can do some text generation and some language translation and some code and some math. And one could imagine that if we continue pouring in all the human investment that we\'re pouring into this like money, competition between nations, human talent, so much talent and training all the young people up, and if we continue to have algorithmic improvements at the rate we\'ve seen and continue to have hardware improvements, so maybe we get optical computing or quantum computing, then one could imagine that eventually this scales to more of quite general systems, or maybe we hit a limit and we have to do a paradigm shift in order to get to the highly capable AI stage. Regardless of how we get there, my question is, do you think this will ever happen, and if so when?"

- "What do you think of the argument 'highly intelligent systems will fail to optimize exactly what their designers intended them to, and this is dangerous'?"

 - Example dialogue: "Alright, so these next questions are about these highly intelligent systems. So imagine we have a CEO AI, and I\'m like, \"Alright, CEO AI, I wish for you to maximize profit, and try not to exploit people, and don\'t run out of money, and try to avoid side effects.\" And this might be problematic, because currently we\'re finding it technically challenging to translate human values preferences and intentions into mathematical formulations that can be optimized by systems, and this might continue to be a problem in the future. So what do you think of the argument \"Highly intelligent systems will fail to optimize exactly what their designers intended them to and this is dangerous\"?

- "What do you think about the argument: 'highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals, and this is dangerous'?"

 - Example dialogue: "Alright, next question is, so we have a CEO AI and it\'s like optimizing for whatever I told it to, and it notices that at some point some of its plans are failing and it\'s like, \"Well, hmm, I noticed my plans are failing because I\'m getting shut down. How about I make sure I don\'t get shut down? So if my loss function is something that needs human approval and then the humans want a one-page memo, then I can just give them a memo that doesn\'t have all the information, and that way I\'m going to be better able to achieve my goal.\" So not positing that the AI has a survival function in it, but as an instrumental incentive to being an agent that is optimizing for goals that are maybe not perfectly aligned, it would develop these instrumental incentives. So what do you think of the argument, \"Highly intelligent systems will have an incentive to behave in ways to ensure that they are not shut off or limited in pursuing their goals and this is dangerous\"?"

- "Have you heard of the term "AI safety"? And if you have or have not, what does that term mean for you?"

- "Have you heard of AI alignment?"

- "What would motivate you to work on alignment questions?"

- "If you could change your colleagues' perception of AI, what attitudes/beliefs of theirs would you like to change?"

- "What are your opinions about policy oriented around AI?"

I also had content prepared if we got to the end of the interview, based on [[Clarke et al. (2022)]{.ul}](https://www.alignmentforum.org/posts/WiXePTj7KeEycbiwK/survey-on-ai-existential-risk-scenarios), [[RAAPs]{.ul}](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic), some of Critch's content on pollution, and my general understanding of the space. My notes: "Scenarios here are about loss of control + correlated failures... can also think about misuse, or AI-assisted war. Also a scenario where the AI does recursive-self-improvement, and ends up actually able to kill humans via e.g. synthetic biology or nanotechnology or whatever, pollution."

Post-Interview Resources Sent To Interviewees
=============================================

I sent most interviewees resources after the interviews.

- I usually floated the idea of sending them resources during the interview, and depending on their response, would send different amounts of resources.

- I did \*\*not\*\* send resources if the interviewee seemed like they would be annoyed by them.

- I only sent a couple of resources if they seemed not very open to the idea.

- For people who were very interested, I often sent them different content that was more specific to them getting involved. These were the people who I sometimes sent the EA / Rationalist material at the end-- I very rarely included EA/Rationalist-specific content in emails, only if they seemed like they'd be very receptive.

Here's my master list of notes, which I selected from for each person based on their interests. I sometimes sent along copies of Human Compatible, the Alignment Problem, or the Precipice.

### Master list of resources

Hello X,

Very nice to speak to you! As promised, some resources on AI alignment. I tried to include a bunch of stuff so you could look at whatever you found interesting. Happy to chat more about anything, and thanks again!

\*\*Introduction to the ideas:\*\*

- \*\*The\*\* \*\*[[Most Important Century]{.ul}](https://www.cold-takes.com/most-important-century/) and specifically \"[[Forecasting Transformative AI]{.ul}](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)\" by Holden Karnofsky, blog series and podcast. Most recommended for description of AI timelines\*\*

- [[Introduction]{.ul}](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment) piece by Kelsey Piper (Vox)

- A short [[interview]{.ul}](https://www.vox.com/future-perfect/2019/10/26/20932289/ai-stuart-russell-human-compatible) from Prof. Stuart Russell (UC Berkeley) about his book, [[Human-Compatible]{.ul}](https://smile.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS/ref=sr\_1\_1?dchild=1&keywords=human+compatible&qid=1635910751&s=digital-text&sr=1-1) (the other main book in the space is [[The Alignment Problem]{.ul}](https://smile.amazon.com/Alignment-Problem-Machine-Learning-Values-ebook/dp/B085T55LGK/ref=sr\_1\_1?dchild=1&keywords=alignment+problem&qid=1635910676&s=digital-text&sr=1-1), by Brian Christian, which I actually like more!)

\*\*Technical work on AI alignment:\*\*

- Some [[empirical work]{.ul}](https://deepmindsafetyresearch.medium.com/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) by DeepMind\'s Safety team about the alignment problem

- [[Empirical work]{.ul}](https://arxiv.org/pdf/2112.00861.pdf) by an organization called Anthropic (mostly OpenAI\'s old Safety team) on alignment solutions

- [[Podcast (and transcript)]{.ul}](https://futureoflife.org/2021/11/01/rohin-shah-on-the-state-of-agi-safety-research-in-2021/) by Rohin Shah, describing the state of AI value alignment (probably want the first half or so)

- [[Talk (and transcript)]{.ul}](https://forum.effectivealtruism.org/posts/63stBTw3WAW6k45dY/paul-christiano-current-work-in-ai-alignment) by Paul Christiano describing the AI alignment landscape in 2020

- [[Alignment Newsletter]{.ul}](https://rohinshah.com/alignment-newsletter/) for alignment-related work

- A much more hands-on approach to [[ML safety]{.ul}](https://arxiv.org/abs/2109.13916), focused on current systems

- Interpretability work aimed at long-term alignment: [[Elhage (2021)]{.ul}](https://transformer-circuits.pub/2021/framework/index.html), by Anthropic and [[Olah (2020)]{.ul}](https://distill.pub/2020/circuits/zoom-in/)

- Ah, and one last report, which outlines one small research organization\'s ([[Alignment Research Center]{.ul}](https://alignmentresearchcenter.org/)) [[research direction]{.ul}](https://docs.google.com/document/d/1WwsnJQstPq91\_Yh-Ch2XRL8H\_EpsnjrC1dwZXR37PC8/edit) and offers prize money for solving it: [[https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals]{.ul}](https://www.lesswrong.com/posts/QEYWkRoCn4fZxXQAY/prizes-for-elk-proposals)

\*\*Introduction to large-scale, long-term risks from humanity\-- including \"existential risks\" that would lead to the extinction of humanity:\*\*

- The [[first third of this book summary]{.ul}](https://ndpr.nd.edu/reviews/the-precipice-existential-risk-and-the-future-of-humanity/), or the book [[The Precipice]{.ul}](https://www.amazon.com/Precipice-Existential-Risk-Future-Humanity/dp/031648492X/), by Toby Ord (not about AI particularly, more about long-term risks)

> Chapter 3 is on natural risks, including risks of asteroid and comet impacts, supervolcanic eruptions, and stellar explosions. Ord argues that we can appeal to the fact that we have already survived for 2,000 centuries as evidence that the total existential risk posed by these threats from nature is relatively low (less than one in 2,000 per century).
>
> Chapter 4 is on anthropogenic risks, including risks from nuclear war, climate change, and environmental damage. Ord estimates these risks as significantly higher, each posing about a one in 1,000 chance of existential catastrophe within the next 100 years. However, the odds are much higher that climate change will result in non-existential catastrophes, which could in turn make us more vulnerable to other existential risks.
>
> Chapter 5 is on future risks, including engineered pandemics and artificial intelligence. Worryingly, Ord puts the risk of engineered pandemics causing an existential catastrophe within the next 100 years at roughly one in thirty. With any luck the COVID-19 pandemic will serve as a \"warning shot,\" making us better able to deal with future pandemics, whether engineered or not. Ord\'s discussion of artificial intelligence is more worrying still. The risk here stems from the possibility of developing an AI system that both exceeds every aspect of human intelligence and has goals that do not coincide with our flourishing. Drawing upon views held by many AI researchers, Ord estimates that the existential risk posed by AI over the next 100 years is an alarming one in ten.
>
> Chapter 6 turns to questions of quantifying particular existential risks (some of the probabilities cited above do not appear until this chapter) and of combining these into a single estimate of the total existential risk we face over the next 100 years. Ord\'s estimate of the latter is one in six.

- [[How to Reduce Existential Risk]{.ul}](https://80000hours.org/articles/how-to-reduce-existential-risk/) by 80,000 Hours or \"[[Our current list of pressing world problems]{.ul}](https://80000hours.org/problem-profiles/)\" blog post

\*\*Governance:\*\*

- [[AI Governance: Opportunity and Theory of Impact]{.ul}](https://www.allandafoe.com/opportunity), by Allan Dafoe and [[GovAI](https://www.governance.ai/)]{.ul} generally

- [[AI Governance: A Research Agenda]{.ul}](https://www.fhi.ox.ac.uk/wp-content/uploads/GovAI-Agenda.pdf), by Allan Dafoe and [[GovAI]{.ul}](https://www.governance.ai/)

- [[The longtermist AI governance landscape: a basic overview]{.ul}](https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview) if you\'re interested in getting involved, also [[more personal posts of how to get involved]{.ul}](https://forum.effectivealtruism.org/tag/governance-of-artificial-intelligence) including [[Locke\_USA - EA Forum]{.ul}](https://forum.effectivealtruism.org/users/locke\_usa)

```{=html}

```
- [[The case for building expertise to work on US AI policy, and how to do it](https://80000hours.org/articles/us-ai-policy/)]{.ul} by 80,000 Hours

\*\*How AI could be an existential risk:\*\*

- [[AI alignment researchers disagree a weirdly high amount about how AI could constitute an existential risk]{.ul}](https://www.alignmentforum.org/posts/WiXePTj7KeEycbiwK/survey-on-ai-existential-risk-scenarios), so I hardly think the question is settled. Some plausible ones people are considering (from the paper)

```{=html}

```
- \"Superintelligence\"

 - A single AI system with goals that are hostile to humanity quickly becomes sufficiently capable for complete world domination, and causes the future to contain very little of what we value, as described in "[[Superintelligence]{.ul}](https://en.wikipedia.org/wiki/Superintelligence:\_Paths,\_Dangers,\_Strategies)\". (Note from Vael: Where the AI has an instrumental incentive to destroy humans and uses its planning capabilities to do so, for example via synthetic biology or nanotechnology.)

- Part 2 of "[[What failure looks like]{.ul}](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like)"

 - This involves multiple AIs accidentally being trained to seek influence, and then failing catastrophically once they are sufficiently capable, causing humans to become extinct or otherwise permanently lose all influence over the future. (Note from Vael: I think we might have to pair this with something like \"and in loss of control, the environment then becomes [[uninhabitable to humans]{.ul}](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic) through pollution or consumption of important resources for humans to survive)

- Part 1 of "[[What failure looks like]{.ul}](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-failure-looks-like)"

 - This involves AIs pursuing easy-to-measure goals, rather than the goals humans actually care about, causing us to permanently lose some influence over the future. (Note from Vael: I think we might have to pair this with something like \"and in loss of control, the environment then becomes [[uninhabitable to humans]{.ul}](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic) through pollution or consumption of important resources for humans to survive)

- War

 - Some kind of war between humans, exacerbated by developments in AI, causes an existential catastrophe. AI is a significant risk factor in the catastrophe, such that no catastrophe would be occurred without the developments in AI. The proximate cause of the catastrophe is the deliberate actions of humans, such as the use of AI-enabled, nuclear or other weapons. See Dafoe ([[2018]{.ul}](https://www.fhi.ox.ac.uk/wp-content/uploads/GovAIAgenda.pdf)) for more detail. (Note from Vael: Though there\'s a recent argument that it may be [[unlikely for nuclear weapons to cause an extinction event]{.ul}](https://www.lesswrong.com/posts/sT6NxFxso6Z9xjS7o/nuclear-war-is-unlikely-to-cause-human-extinction), and instead it would just be catastrophically bad. One could still do it with synthetic biology though, probably, to get all of the remote people.)

- Misuse

 - Intentional misuse of AI by one or more actors causes an existential catastrophe (excluding cases where the catastrophe was caused by misuse in a war that would not have occurred without developments in AI). See Karnofsky ([[2016]{.ul}](https://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity)) for more detail.

- Other

\*\*Off-switch game and corrigibility\*\*

- [[Off-switch game]{.ul}](https://arxiv.org/abs/1611.08219) and [[corrigibility]{.ul}](https://intelligence.org/files/Corrigibility.pdf) paper, about incentives for AI to be shut down. This article from DeepMind about \"[[specification gaming]{.ul}](https://deepmindsafetyresearch.medium.com/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4)\" isn\'t about off-switches, but also makes me feel like there\'s currently maybe a tradeoff in task specification, where more building more generalizability into a system will result in novel solutions but less control. Their [[follow-up paper]{.ul}](https://deepmindsafetyresearch.medium.com/scalable-agent-alignment-via-reward-modeling-bf4ab06dfd84) where they outline a possible research to this problem makes me feel like encoding human preferences is going to be quite hard, and all of the other discussion in AI alignment, though we don\'t know how hard the alignment problem will be.

\*\*There\'s also a growing community working on AI alignment\*\*

- The strongest academic center is probably UC Berkeley\'s [[Center for Human-Compatible AI]{.ul}](https://humancompatible.ai/about/). Mostly there are researchers distributed at different institutions e.g. [[Dylan Hadfield-Menell]{.ul}](https://scholar.google.com/citations?hl=en&user=4mVPFQ8AAAAJ&view\_op=list\_works&sortby=pubdate) at MIT, [[Jaime Fisac]{.ul}](https://scholar.google.com/citations?hl=en&user=HvjirogAAAAJ&view\_op=list\_works&sortby=pubdate) at Princeton, [[David Krueger]{.ul}](https://twitter.com/davidskrueger) in Oxford, Sam Bowman at NYU, Alex Turner at Oregon, etc. Also, a good portion of the work is done by industry / nonprofits: [[Anthropic]{.ul}](https://www.anthropic.com/), [[Redwood Research]{.ul}](https://www.redwoodresearch.org/), OpenAI\'s safety team, DeepMind\'s Safety team, [[ARC]{.ul}](https://alignmentresearchcenter.org/), independent researchers in various places.

- \*\*There is money in the space! If you want to do AI alignment research, you can be funded by either Open Philanthropy ([[students]{.ul}](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/the-open-phil-ai-fellowship),\*\* \*\*[[faculty]{.ul}](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/request-for-proposals-for-projects-in-ai-alignment-that-work-with-deep-learning-systems)\-- one can also just email them directly instead of going through their grant programs) or\*\* \*\*[[LTFF]{.ul}](https://funds.effectivealtruism.org/funds/far-future) or\*\* \*\*[[FTX-]{.ul}](https://ftxfuturefund.org/area-of-interest/)- this is somewhat competitive and you do have to show good work, but it\'s less competitive than a lot of sources of funding in academia.\*\*

- If you wanted to rapidly learn more about the theoretical technical AI alignment space, walking through this [[curriculum]{.ul}](https://www.eacambridge.org/technical-alignment-curriculum) is one of the best resources. A lot of the interesting theoretical stuff is happening online, at [[LessWrong]{.ul}](https://www.lesswrong.com/posts/P3Yt66Wh5g7SbkKuT/how-to-get-into-independent-research-on-alignment-agency) / [[Alignment Forum]{.ul}](https://www.alignmentforum.org/posts/Yp2vYb4zHXEeoTkJc/welcome-and-faq) (Introductory [[Content]{.ul}](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ)), since this field is still pretty pre-paradigmatic and people are still working through a lot of the ideas.

\*\*There\'s also two related communities who care about these issues, who you might find interesting:\*\*

- [[Effective Altruism](https://www.effectivealtruism.org/articles/introduction-to-effective-altruism)]{.ul} community, whose strong internet presence is on the [[EA Forum]{.ul}](https://forum.effectivealtruism.org/posts/fd3iQRkmCKCCL289u/new-start-here-useful-links). [[Longtermism]{.ul}](https://80000hours.org/articles/future-generations/) is a concept they care a lot about, and you can schedule a one-on-one [[coaching]{.ul}](https://80000hours.org/speak-with-us/) call here.

- [[Rationalist]{.ul}](https://www.lesswrong.com/tag/rationalist-movement) community\-- the best blog from this community is from Scott Alexander ([[first blog]{.ul}](https://slatestarcodex.com/about/), [[second blog]{.ul}](https://astralcodexten.substack.com/?sort=top)), and they\'re present on [[LessWrong]{.ul}](https://www.lesswrong.com/bestoflesswrong). Amusingly, they also write fantastic fanfiction ([[e.g. Harry Potter and the Methods of Rationality]{.ul}](https://www.lesswrong.com/hpmor)) and I think some of their [[nonfiction]{.ul}](https://mindingourway.com/guilt/) is fantastic.

Happy to chat more about anything, and good to speak to you!

Best,

Vael

Informal Interview Notes
========================

### Thoughts from listening to myself doing these interviews

- There's obviously strong differences in what content gets covered, based on the interviewees' opinions and where they're at. I didn't realize that another important factor in what content gets covered is the interviewee's general attitude towards people / me. Are they generally agreeable? Do they take time to think over my statement, find examples that match what I'm saying? Do they try to speak over me? Are they curious about my opinions? How much time do they have? Separate from rapport (and my rapport differs with different interviewees), there's a strong sense of spaciousness in some interviews, while many feel like they're more rapid-fire exchanges of ideas. I often end up talking more in the agreeable / spacious interviews.

- Participants differ in how much they want to talk in response to one question. I tended to not interrupt my interviewees, though I think that's a good thing to do. ("I'm sorry, this is very interesting, but we really need to get to the next question.") That meant that for participants who tended to deliver long answers, I had fewer chances to ask questions, which meant I often engaged less with their previous responses and tried to move them on to new topics more abruptly.

- I make a lot of agreeable sounds, and try to rephrase what people say. People differ with how many agreeable sounds they make during my speech as well, and how much they're looking at the camera and looking for cues.

- I tended to adjust my talking speed to the interviewee somewhat, but usually ended up talking substantially more quickly. This made my speech harder to parse because of all the "like"s that get inserted while I'm thinking and talking at the same time. (I don't think I realized this at the time; it's more obvious when listening back through the interviews. I've removed a fair amount of the "like"s in the transcripts because it's harder to read than hear.) Generally, I found it useful to try to insert technical vocabulary and understanding as early as possible, so researchers would explain more complicated concepts and be calibrated on my level of understanding. I did somewhat reduce speaking speed and vocabulary when speaking with interviewees whose grasp of English was obviously weaker, though in those cases I think it's maybe not worth having the interview, since I found it quite hard to communicate across a concept gap under time and communication constraints. (These concepts are complicated, and hard enough to cover in 40m-60m even without being substantially limited by language.)

- When I'm listening to these interviews, I'm often like: Vael, how did you completely fail to remember something that the interviewee said one paragraph up, what's up with your working memory? And I think that's mostly because there's a lot to track during interviews, so my working memory gets occupied. I often found my attention on several things:

 - Trying to take on the framework of their answer, and fit it into the framework of how I think about these issues. Some people had substantially different frames, so this took a lot of mental energy.

 - Trying to figure out what counterpoint to respond with when I disagreed with something, so -- fitting their answer into a mental category, flitting through my usual replies to that category, and then holding my usual replies in mind for when they were done, if there were multiple replies lined up.

 - Trying to figure out whether I should reply to their answer, or move on. One factor here was whether they tended to take up a lot of talking space, so I needed to be very careful with what I used my conversational turn for. Another factor was how much agreement I had with my previous question, so that I could move on to the second. A third factor was tracking time-- I spent a lot of time tracking time in the interview, and holding in mind where we were in the argument tree, and where I thought we could get to.

 - If they'd said something that was actually surprising to me, and seemed true, rather than something I'd heard before and needed to reformulate my answer to, this often substantially derailed a lot of the above processing. I then needed to do original thinking while on a call, trying to evaluate whether something said in a different frame was true in my frames. In those cases I usually just got the interviewee to elaborate on their point, while throwing out unsophisticated, gut-level "but what if..." replies and seeing how they responded, which shifted the conversation towards more equality. I think thinking about these points afterwards (and many more things were new to me in the beginning of the interviews, compared to the end) was what made my later interviews better than my earlier interviews.

 - Trying to build rapport / be responsive / engage with their points well / make eye contact with the camera / watch my body language / remember what was previously said and integrate it. This was mostly more of a background process.

- Conversations are quite different if you're both fighting for talking time than if you're not. Be ready for both, I think? I felt the need to think and talk substantially faster the more interruptions there were in a conversation. I expected my interviewees to find the faster-paced conversations aversive, but many seemed not to and seemed to enjoy it. In conversations where the interviewee and I substantially disagreed, I actually often found faster-pace conversations more enjoyable than slower-paced conversations. This was because it felt more like an energetic dialogue in the faster conversations, and I often had kind of slow, sinking feeling that "we both know we disagree with each other but we're being restrained on purpose" feel in the slower conversations.

- My skill as an interviewer at this point seems quite related to how well I know the arguments, which like... I could definitely be better on that front. I do think this process is helpful for my own thinking, especially when I get stuck and ask people about points post-interview. But I do read these interviews and think: okay, but wouldn't this have been better if I had had a different or fuller understanding? How good is my thinking? It feels hard to tell.

### Content analysis

I have a lot to say about typical content in these types of interviews, but I think the above set of interviews is somewhat indicative of the spread. Hoping to have more information on these eventually once I finish sorting through more of my data.


Danijar Hafner - Gaming our way to┬áAGI-by Towards Data Science-video_id Bgz9eMcE5Do-date 20220112

# Danijar Hafner on Gaming Our Way to AGI. Danijar Hafner on procedural game generation for reinforcement learning agents by Jeremie Harris on the Towards Data Science Podcast

Until recently, AI systems have been narrow — they’ve only been able to perform the specific tasks that they were explicitly trained for. And while narrow systems are clearly useful, the holy grain of AI is to build more flexible, general systems.

But that can’t be done without good performance metrics that we can optimize for — or that we can at least use to measure generalization ability. Somehow, we need to figure out what number needs to go up in order to bring us closer to generally-capable agents. That’s the question we’ll be exploring on this episode of the podcast, with Danijar Hafner. Danijar is a PhD student in artificial intelligence at the [University of Toronto](http://learning.cs.toronto.edu/) with [Jimmy Ba](https://scholar.google.com/citations?user=ymzxRhAAAAAJ&hl=en&oi=ao) and [Geoffrey Hinton](https://scholar.google.com/citations?user=JicYPdAAAAAJ&hl=en&oi=ao) and researcher at [Google Brain](https://research.google/teams/brain/) and the [Vector Institute](https://vectorinstitute.ai/).

Danijar has been studying the problem of performance measurement and benchmarking for RL agents with generalization abilities. As part of that work, he recently released Crafter, a tool that can procedurally generate complex environments that are a lot like Minecraft, featuring resources that need to be collected, tools that can be developed, and enemies who need to be avoided or defeated. In order to succeed in a Crafter environment, agents need to robustly plan, explore and test different strategies, which allow them to unlock certain in-game achievements.

Crafter is part of a growing set of strategies that researchers are exploring to figure out how we can benchmark and measure the performance of general-purpose AIs, and it also tells us something interesting about the state of AI: increasingly, our ability to define tasks that require the right kind of generalization abilities is becoming just as important as innovating on AI model architectures. Danijar joined me to talk about Crafter, reinforcement learning, and the big challenges facing AI researchers as they work towards general intelligence on this episode of the TDS podcast.

Here were some of my favourite take-homes from the conversation:

- The Crafter environment includes a wide range of achievable goals, each of which involves performing a specific in-game task for the first time. Some of these tasks are simple, and can be achieved fairly consistently by current state-of-the-art RL agents (for example, finding sources of food and harvesting them). But some are more challenging, because they involve dependencies on other tasks: a “collect iron” task can only be achieved once a “make stone pickaxe” task has already been completed, for example. The result is a fairly deep tech tree that can only be completed by agents that have learned to plan.
- Because different tasks require different abilities, Crafter environments can give developers a way to profile their RL agents. By measuring an agent’s average performance across the full distribution of achievable tasks, they obtain a fingerprint of its capabilities that indicates the extent to which it’s picked up skills like planning and exploration, which are closely linked to generalization.
- One of the challenging aspects of developing benchmarks for machine learning is to ensure that they’re tuned to the right level of difficulty. Good benchmarks are challenging to master (so that they can motivate and direct progress in the field), yet tractable (so that developers have enough signal to iterate and improve their models).
- One interesting debate about the future of AI has to do with the extent to which further progress will come from dramatic improvements to algorithms, or from the increased availability of compute resources. Increasingly, we’ve seen cutting-edge results in reinforcement learning come from model-based systems that make heavier use of compute than their model-free counterparts, and progress in RL has closely tracked increases in compute budgets (e.g. MuZero and EfficientZero). To some, that’s an indication that current AI techniques might be sufficient to reach human-level performance, with relatively minor tweaks, if they’re only scaled up with more compute. While Danijar sees some merit to this argument, he does think that there remain fundamental advances in algorithm design that we’ll have to make before reaching human-level AI that go beyond leveraging raw compute horsepower.


How can we see the impact of AI strategy research _ Jade Leung _ EA Global - San Francisco 2019-by Centre for Effective Altruism-video_id 8M3nIu7GIsA-date 20190829

# Jade Leung How can we see the impact of AI strategy research - EA Forum

\_For now, the field of AI strategy is mostly focused on asking good questions and trying to answer them. But what comes next? In this talk, Jade Leung, Head of Research at the\_ [\_Center for the Governance of AI\_](https://www.fhi.ox.ac.uk/govai/)\_, discusses how we should think about practical elements of AI strategy, including policy work, advocacy, and branding.\_

\_Below is a transcript of Jade’s talk, which we’ve lightly edited for clarity. You can also watch the talk on\_ [\_YouTube\_](https://www.youtube.com/watch?v=8M3nIu7GIsA) \_or read it on\_ [\_effectivealtruism.org\_](https://effectivealtruism.org/articles/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research)\_.\_

## The Talk

Technology shapes civilizations. Technology has enabled us to hunt, gather, settle, and communicate. Technology today powers our cities, extends our lifespans, connects our ideas, and pushes the frontier of what it means to be human.

Technology has also fueled wars over power, ideology, prestige, history, and memories. Indeed, technology has pushed us to the precipice of risk in less than a decade — a fleeting moment in the timespan of human civilization. \[With just a few years of research,\] we equipped ourselves with the ability to wipe out the vast majority of the human population with the atomic bomb.

If we rewind to the emergent stage of these transformative technologies, we have to remember that we are far from being clear-eyed and prescient. Instead, we're some combination of greedy, clueless, confused, and reckless. But ultimately, how we choose to navigate between the opportunities and risks of transformative technologies will define what we gain from these transformative technologies — and also what risks we expose ourselves to in the process.

This is the canonical challenge of governance of these transformative technologies. Today, we're in the early stages of navigating a particular technology: artificial intelligence (AI). It may be one of the most consequential technologies of our time and the most important one for us to get right. But getting it right requires us to do something that we've never done before: Formulate a navigation strategy with deliberate caution and explicit altruistic intention. It requires us to have foresight and to orient \[ourselves\] toward the long-term future. This is the challenge of AI governance.

If we think about our history and track record, our baseline is pretty far from optimal. That’s a very kind way of saying that it sucks. We're not very good at governing transformative technologies. Sometimes we go down this path and \[the journey\] is somewhat safe and prosperous. Sometimes, we falter. We pursue the benefits of synthetic biology without thinking about how that affects biological weapons. Sometimes we stop ourselves at the starting line because of fear, failing to pursue opportunities like atomically precise manufacturing or STEM cell research. And sometimes we just fall into valleys.

![](https://images.ctfassets.net/ohf186sfn6di/5DCD4HJzhKUiuLdvynmunh/c89954c09a5a4cfdbeef94b0ece880cd/Slide02.png)

During the Cuban Missile Crisis, President John F. Kennedy estimated that the chance of nuclear war was one in three. \_One in three.\_

The reality is we've been pretty damn lucky. We deserve no credit for \[avoiding any of these catastrophes\]. But as my swimming coach once said, “If you're really, really, really bad at something, you only need to try a little bit to become slightly better at it.” So here's to being slightly better at navigating these transformative technologies.

I think there are three goals in the AI strategy and governance space \[that can help us rise\] slightly \[above\] our currently awful baseline.

![](https://images.ctfassets.net/ohf186sfn6di/2D0pEJYB95BPTUn2JfPz4T/ca9b35b14b98956277464087358593a4/Slide05.png)

Goal number one: Gain a better understanding of what this landscape looks like. Where are the mountains? Where are the valleys, the slippery slopes, the utopias? This is super-hard to do. It's very speculative and uncertain, so we need to be humble. But we should try anyway.

The second thing we can try to do is equip ourselves with good heuristics for navigation. If uncertainty is an occupational hazard of working in this space, then we can try to figure out, in general terms, what might be good and bad \[to pursue\]. How should we orient ourselves? Which directions do we want to go in?

The last goal is to translate these heuristics into actual navigation strategies. How do we ensure that our heuristics make it into the hands of the people who are turning this boat in certain directions?

If you'll stick with my navigation metaphor for a bit longer, we can think of the first goal as a mapping exercise to determine where the mountains, valleys, water sources, and cafes with good wifi are. The second goal is about equipping ourselves with a compass. If we know that there are aggressive rhinos to the south and good vegan restaurants in the north, we’ll go north instead of south.

This metaphor is kind of falling apart, but the third goal is the steering wheel. You can’t use it if you’re in the back of the car. That's ultimately what I want to focus on today: How do we make sure that \[our map and compass will be used to steer — i.e., to make real-world decisions about AI\]?

\[I have two reasons for focusing\] on this. First, AI strategy and governance research is effective when it happens upstream of actionable, real-world tactics and strategies. They can be relatively far upstream. I think we would lose a lot of good research questions if \[we were always motivated by\] whether something could inform a decision today. But I think it would be a mistake for anyone who does AI strategy and governance research to \[avoid\] thinking about how they expect their research to \[play out\] in relevant, real-world decisions.

That leads me to the second reason for focusing: I don't think we know how to do this \[make our research actionable\] well. I think we invest far more effort into understanding how to do good research than we do into understanding how to \[come up with\] good tactics. Don't get me wrong: I don't think we know how to do good research yet. We're still trying to figure that out. And I find it hilarious that people think that I know how to do good research; if only you knew how little I know! But I think we need to invest far more proportional effort into \[asking ourselves\]: Once we’ve done our research and have some insights in place, what do we do to \[apply\] them and \[influence\] the direction in which we're going?

![](https://images.ctfassets.net/ohf186sfn6di/idyfmsAFAWL7iSMXvGvlw/d3e0dbbcbf2693c342158b6837950756/Slide06.png)

With that in mind, let's start at the end. What are the decisions that we want to influence in the real world? Another way to ask this question is: Who is making the decisions that we want to change?

They fall into two broad categories: (1) those developing and deploying AI and (2) those shaping the environment in which AI is developed and deployed.

![](https://images.ctfassets.net/ohf186sfn6di/7abtDrND6UQYIvZs55YHrn/0afb7092fddf02f791de60b03957fb28/Slide07.png)

Those developing and deploying AI include researchers, research labs, companies, and governments. those shaping the environment.

In terms of \[the second group\], there are a number of different environments to shape:

\\* \*\*The research environment\*\* can be shaped by lab leaders, funders, universities, and CEOs. They shape the kind of research that is being invested in — i.e., the research considered within the [Overton window](https://en.wikipedia.org/wiki/Overton\_window).

\\* \*\*The legislative environment\*\*, which constrains what can be deployed and how, can be shaped by legislators, regulators, states, and the people \[being governed\].

\\* \*\*The market environment\*\*, which can be shaped by investors, funders, consumers, and employees. They create incentives that drive certain forms of development and deployment, because of supply and demand.

Now, you can either become one of these decision-makers or you can become a person who influences them. This is in no way a commentary on your brilliance as human beings. But none of you will become important. I'm unlikely to become important. The reality is that’s how the world works. If you do end up becoming an important person, the recording of this talk is your voucher for a free drink on me. But if you assume that I'm right, most of you are going to fall into the category of people who influence decisions as opposed to making them.

Therefore, I’m going to \[spend the rest of this talk\] focusing on this question: How do we increase our ability to influence the decisions being made \[about AI\]?

![](https://images.ctfassets.net/ohf186sfn6di/w8TRFh3DPKsEoHpvMUSWB/a245963bccdae8366fa85babd6eaecdd/Slide08.png)

There are many steps, but I see them falling into two broad areas. The first step is having good things to say. The second step is making sure that the people who matter \[are made aware of\] these good things.

![](https://images.ctfassets.net/ohf186sfn6di/35KkwXeV291FVSe2D59ZKx/73049db077dedaa23f5e3b53e1654480/Slide09.png)

A quick note on what I mean by “good”: I'm broadly conceiving of all of us as good in the normative sense of steering our world in a direction that we want, and good in the pragmatic sense, in that a decision-maker will be likely to actually go in that direction because it's reasonable and falls within their timeframe.

Oftentimes these two definitions of good conflict. For example, things you think will be good for the long-term future won't \[necessarily\] be things that are tractable or reasonable from a decision-maker’s point of view. I acknowledge that these two things are in tension. It's hard to figure out how to compromise between them sometimes.

![](https://images.ctfassets.net/ohf186sfn6di/4hjmoCrfCkKmFHgrovH4Kx/e8f30f0f4ac753cba914fbf666a19e0d/Slide10.png)

That being said, I think AI strategy and governance research can aim to have good things to say about a given decision-maker’s (1) priorities, (2) strategies, and (3) tactics. Those are three broad buckets to dig into a bit more.

\*\*Priorities:\*\* I think priorities are basically people’s goals. What benefits are they incentivized to pursue, and what costs are they willing to bear in the process of pursuing those goals? For example, if you manage to convince a lab that safety leads to product excellence, that can make safety a goal for the lab. If you manage to convince a government that cooperation is necessary for technology leadership in an international world, that can make cooperation a goal.

\*\*Strategies:\*\* You may aim to have useful things to say about certain strategies that \[decision-makers adopt\]. For example, resource allocation is a pretty common strategy that one could aim to influence. How are they distributing their budgets? How are they investing in research and development efforts across various streams? You also may have things to say about what a given actor chooses to promote or advocate for versus \[ignore\]. For example, in the case of influencing a government, you might want them to pursue certain pieces of legislation that can help you achieve certain goals. In the case of labs, you might want them to invest in certain types of new programs or different workstreams.

\*\*Tactics:\*\* The third area is tactics. These include public relations tactics. What do they signal to the external world, and how does that affect their ability to achieve their goals? And what about relationship tactics — with whom do they coordinate and cooperate? Whom do they trust (and distrust)? Whom do they decide to invest in?

To make this a little bit more concrete, I'm going to pick on an actor who needs a lot of good \[advice\]: the U.S. government.

![](https://images.ctfassets.net/ohf186sfn6di/2JXNTSUGAIpxX5uKjvXQp1/c65a101927ec5d225dcaee983993594a/Slide11.png)

One of the biggest risks is that nation states will slide into techno-nationalist economic blocs. The framing of strategic competition that we have around AI now could exacerbate a number of AI risks. I won't go into detail now, but [we've written a fair amount about it at The Governance of AI](https://www.fhi.ox.ac.uk/govai/#publications). We want to prevent nations from sliding into various economic blocs and the nationalization of bits of AI research and development.

What would a caricature of the U.S. government's position look like? (I say “caricature” because it's not at all clear that they actually have a coherent strategy.) It looks something like sliding into these economic blocs. And that's a bad thing. Their \[overarching\] priorities are technology leadership, in both an economic and military sense, with a corollary of preserving and maintaining national security. Costs that they may be willing to bear in extreme circumstances include anything that is required to gain control of an R&D pipeline and secure it within national borders.

Now they are making moves in the strategy and tactics space — for example, announcements of export controls that the U.S. government made in November 2018 indicate that they want to preserve domestic capacity for R&D at the cost of investing in international efforts and transnational communities. They also indicate an explicit intention of shutting out foreign competitors and adversaries. \[Overall\], their AI strategies and tactics point in the direction of “America first.” And the footnotes there suggest that when America is first, over the long term the world suffers. That's too bad. So, those are the kinds of stances that the U.S. posture points toward.

If one has \[the chance to try persuading\] the U.S. government, one could aim to convince them that their priorities, strategies, and tactics should move in a different direction. For example, a desirable priority could be technology leadership, but leadership could mean leading with a global, cosmopolitan viewpoint. You \[could focus on influencing them to\] bear the cost of investing in things like safety research in order to pursue this priority in a responsible way. The strategies and tactics you could inform them of when they conduct this research could \[involve international outreach\]. With whom should they ally themselves and cooperate? What kinds of signals should they send externally to ensure that others with a similar view of technology leadership will \[take steps in\] the same direction?

This is the type of decision set that you want to influence when conducting upstream AI strategy and governance research. \[Once you\] have a broad sense of what you think is good, you have the mega-task of trying to make those good things happen in the real world.

I have a few suggestions for how to approach that.

![](https://images.ctfassets.net/ohf186sfn6di/6z2ZpuOh6qSqXPcuuDFUA2/fb430acad3de772a61ac199a49561042/Slide12.png)

The first is to \[focus on\]\] a few \_tractable\_ good things. I say “tractable” here to mean things that will make sense to, or sit well with, decision-makers, such that they are likely to do something about it.

One way to do that is to find hooks between things that you care about and things that a decision-maker cares about. Find that intersection or middle part of the Venn diagram. One canonical bifurcation — which I don't actually like all that much — is the bifurcation between near-term and long-term concerns. Near-term concerns are things that are politically salient. \[They allow you to\] have a discussion in Congress and not look nuts. Long-term concerns are often things that make you look a little bit wacky. But there are some things at the intersection that could lead you to talk about near-term concerns in a way that lays the foundation for long-term concerns that you actually care about and want to seed discussions around.

For example, the automation of manufacturing jobs is a huge discussion in the U.S. at the moment. It’s a microcosm of a much larger-scale problem \[involving\] massive labor displacement, economic disruptions, and the distribution of economic power in ways that could be undesirable. That’s a set of long-term concerns. But talking about it in the context of truck drivers in the U.S. could be an inroad into making those long-term concerns relevant.

A similar thing can be said about the U.S. and China. People in Washington, D.C. care about the U.S.’s posture toward China, and what the U.S. does and signals now will be relevant to how this particular bilateral relationship pans out in the future. And that's incredibly relevant for how certain race dynamics pan out.

Once you've filtered for these things that are tractable, then you need to do the work of translating them in a digestible way for decision-makers.

![](https://images.ctfassets.net/ohf186sfn6di/28dJJMsYjDwnPD60VSgHYP/45fd30db87e2be201b8fda4cf79d8cbd/Slide13.png)

The assumption here is that decision-makers are often very time-constrained and attention-constrained. They will \[be more likely to respond to messages that are\] easy to remember and \[relayed\] in the form of memes. And unfortunately, long, well-argued, epistemically robust pieces end up \[having less impact\] than we would hope.

Superintelligence is perhaps one of the best examples. This is an incredibly epistemically robust \[topic\]. But ultimately, the meme it was boiled down to for the vast majority of people was: “Smart Oxford academics think AI is going to kill us.” So don't try to beat them with nuance. Try to just play this meme game and come up with better memes.

Here are three examples of memes that are currently in danger of taking off:

![](https://images.ctfassets.net/ohf186sfn6di/733mHkA6KjER3FaIsl4Mi/b3c0460e02a13dfa39dbe39577b28302/Slide14.png)

1\. The U.S. and China are in an arms race.
2\. Whoever wins will have a decisive strategic advantage.
3\. AI safety is always orthogonal to performance.

It's not clear to me that all of these things are true. And for some of them I'm quite sure that I don't want them to be true. But they are being propagated in ways that are informing decisions that are currently being made. I think that's a bad thing.

One thing to focus on, in terms of trying to have good things to say and making those good things heard, it to translate them into messages that are similarly digestible.

![](https://images.ctfassets.net/ohf186sfn6di/EujxjpH5wbaTAZExHdTrK/08da0769129a91e57a6de0ae0106e312/Slide15.png)

Candidates for memes we might propagate are things like: “the equivalent of leading in the AI space is to care about safety and governance”; “the windfall distributions from a transformative AI should be distributed according to some common principles of good”; and “governance doesn't equal government regulation, so multiple actors carry the responsibility to govern well.” Unless we propagate our messages in easy ways, it's going to be very hard to compete with the bad narratives out there.

![](https://images.ctfassets.net/ohf186sfn6di/4ss4g3lGYNvcg0iGC48t8t/48865b05b5a7b1bbeafe610f799eb560/Slide16.png)

The last step is to ensure that \[our messages\] reach some circles of influence. To do that, model your actor well. For example, if you want to target a specific lab, try to figure out who the decision-makers are, what they care about, and whom they listen to. Then, target your specific messages and work with those particular circles of influence in order to get heard. That's my hot take on how \[research\] can be made slightly more relevant in a real-world sense.

Some final points that I want you to take away:

![](https://images.ctfassets.net/ohf186sfn6di/4r31D2owuDzQMIzfMW08Uc/ea15b45a3d6a36bd1ba545417b91cc25/Slide17.png)

Ultimately, the impact of this work is contingent on how good our tactics are. The claim that I've made today is that we need to put far more work into this. I’m uncertain how well we can do that — and how much effort we should put into it. But broadly speaking, as soon as we have relevant insights, we should be intentional about investing in propagating them.

![](https://images.ctfassets.net/ohf186sfn6di/5oPLXGvYTnXCp0vT22Ue2A/ddff5d0a887324bc75444b70bcba7a46/Slide18.png)

Second, exercising this influence is going to be a messy political game. The world’s \[approach to\] decision-making is muddled, parochial, and suboptimal. We can have a bit of a cry about how suboptimal it is. But ultimately, we need to work within that system. \[Using\] effective navigation strategies is going to require us to work within a set of politics that we may disagree with to some extent in terms of values. But we need to be tactical and do it.

![](https://images.ctfassets.net/ohf186sfn6di/6syDzhMJywbv59mL1IQi2D/f6969f592a6e8d567b96a5479dc97976/Slide19.png)

Finally, governance is a very hard navigation challenge. We have no track record of doing it well, so we should be humble about our ability to do it. At the moment we don't know that we can succeed, but we can try our best.

\*\*Moderator:\*\* Thank you for that talk. I’d like to start with something that you ended with. You said that we're dealing with systems that are difficult to operate in. To what extent do you even think it's possible to get people to think more clearly? Should we instead just be focusing on institutional change?

\*\*Jade:\*\* I think there are things that we need to try out. Institutional change is valuable. Attempting to communicate through existing decision-makers in existing institutions is valuable. But I don't think we know enough about what's necessary and how tractable certain things are in order to put all of our eggs in one basket.

So maybe one meta-point is that as a field, we need to diversify our strategies. For example, I think some people should be focusing on modeling existing decision-makers — particularly decision-makers that we think \[have enough credibility\] to be relevant. And I think others could take the view that existing institutions are insufficient, and that institutional change is ultimately what is required. And then that becomes a particular strategy that is pursued.

The field is shrouded in enough uncertainty about what's going to be relevant and tractable that I would encourage folks to diversify.

\*\*Moderator:\*\* You focused on the U.S. government as one of the actors that people might pay particular attention to. Are there others that you would recommend people pay attention to?

\*\*Jade:\*\* Yeah. I generally advocate for focusing on modeling governments \[based in places that are likely to be relevant\] more than particular private actors. For example, the Chinese government would be worth focusing on. I think we have a better shot at modeling them based on history. There are more variants and anomalies in private spaces.

Second, focus on organizations that are important developers of this technology \[AI\]. The canonical ones are [DeepMind](https://deepmind.com/) and [OpenAI](https://openai.com/). There are others worth focusing on too.

\*\*Moderator:\*\* Someone could construe your advice as trying to understand what's happening currently in the policy landscape and in a variety of academic disciplines that people spend their lives in, and then melding all of those together into a recommendation for policymakers. That can feel a little overwhelming as a piece of advice. If someone has to start somewhere and hasn't worked in this field before, what would you say is the minimum that they should be paying attention to?

\*\*Jade:\*\* Good question. If you're not going to try to do everything (which is good advice), I think one can narrow down the space of things to focus on based on competitive advantage. So think through which arenas of policy decisions you're likely to be able to influence the most. Then, focus specifically on the subset of actors in that space.

\*\*Moderator:\*\* And assuming a person doesn't have expertise in one area and is just trying to fill a vacuum of understanding somewhere in this AI strategy realm, what would you \[recommend\] somebody get some expertise in?

\*\*Jade:\*\* That’s a hard question. There are a lot of resources that are out there that can help orient you to the research space. Good places to start would be our [website](https://www.fhi.ox.ac.uk/govai/). There's a research agenda, which has a lot of footnotes and references that are very useful. And then there's also a [blog post](https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1) by the safety research team at DeepMind — they’ve compiled a set of resources to help folks get started in this space.

If you're particularly interested in going deeper, you’re always welcome to [email me](https://www.fhi.ox.ac.uk/team/jade-leung/).