CHAI Newsletter #2 2022

07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 1/9
CHAI Newsletter
This newsletter includes CHAI activities from April 2022 to mid-September
2022
Events
CHAI 2022 annual W orkshop
On the ﬁrst weekend in June, CHAI held its 6th annual workshop in person at
Asilomar Conference Grounds in Paciﬁc Grove, CA. The nearly 145 attendees
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 2/9participated in panel discussions, keynote talks, and informal group discussions
during the 3 day event. There were also more casual activities including beach
walks and bonﬁres to bring the AI safety community together .

ICLR 2022 W orkshop on Agent Learning in Open-Endedness
This workshop was co-organized by Michael Dennis from CHAI and Minqi
Jiang, Jack Parker-Holder , Mikayel Samvelyan, Roberta Raileanu, Jakob N.
Foerster , Edward Grefenstette, Tim Rocktäschel.
Rapid progress in AI has produced agents that learn to succeed in many
challenging domains. However , these domains are simple relative to the
complexity of the real world, and once a domain has been mastered, the
learning process typically ends. In contrast, the real world is an open-ended
ecosystem of problems and solutions, presenting endless, novel challenges,
which in turn shape the evolution of humans and other living organisms that
must continually solve them for survival. While so far no artiﬁcial learning
algorithm has produced an intelligence as general as humans, we know that we
ourselves resulted from this open-ended process of coevolution among living
organisms and the environment. This workshop aims to unite researchers
across many relevant ﬁelds including reinforcement learning, continual learning,
evolutionary computation, and artiﬁcial life in pursuit of creating, understanding,
and controlling open-ended learning systems.
Find the link here.

Living in the Age of AI
A lecture on science education in the age of AI at the Japanese corporation
running the Lawrence Hall of Science program,  with Noyuri Mima from CHAI
as a speaker .
Find the link here.

World Economic Forum
Stuart Russell participated in three panels on AI at the 2022 W orld Economic
Forum in Davos, Switzerland: "Emerging Tech Futures", "WEF AI Toolkit", and
"Responsible AI for Societal Gains". He was the only AI researcher and the only
academic computer scientist invited to speak at the Forum.
Find the link here.
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 3/9
Normalization and V alue
Brian Christian gave a talk titled "Normalization and V alue" at the RL  as a
Model of Agency W orkshop as part of the 2022 RLDM (Reinforcement Learning
and Decision Making) conference, held at Brown University in Providence,
Rhode Island.
Find the link here.

Creating a Gender-Equal Society with AI - Education for the Future in
Berkeley , USA
An online Seminar "Creating a Gender-Equal Society with AI", with Noyuri
Mima from CHAI as a speaker .
Find the link here.

Weekly CHAI Seminars
Every W ednesday we host our Beneﬁcial AI Seminar from 10 am to 12 pm PST .
Seminars currently take place remotely through Zoom.  Here  is the schedule for
the Beneﬁcial AI seminars.
If you would like to attend the virtual seminars, please email chai-
admin@berkeley .edu.

Researcher News

Stuart Russell has been awarded the  IJCAI Research Excellence Award
for 2022.
From the International Joint Conference on Artiﬁcial Intelligence (IJCAI)
website: "The Research Excellence award is given to a scientist who has
carried out a program of research of consistently high quality throughout an
entire career yielding several substantial results. Past recipients of this honor
are the most illustrious group of scientists from the ﬁeld of Artiﬁcial Intelligence.
Professor Russell is recognized for his fundamental contributions to the
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 4/9development of Bayesian logic to unify logic and probability , the theory of
bounded rationality and optimization, and learning and inference strategies for
operations in uncertain environments."
On July 28, Prof. Russell delivered an invited lecture at the IJCAI meeting in
Vienna on the topic, "Artiﬁcial Intelligence: Some Thoughts?".

Pulkit V erma and Siddharth Srivastava from CHAI and Naman P . Shah
were awarded AAMAS 2022 Best Demo Award for " JEDAI: A System for
Skill-Aligned Explainable Robot Planning ". JEDAI (JEDAI explains decision-
making AI) is an interactive learning tool for AI+robotics that provides
explanations autonomously .

CHAI PhD student Micah Carroll was awarded the NSF Graduate Research
Fellowship Program, which totals at $138,000 of funding over 3 years. He
was also accepted to the ﬁrst iteration of the AI Policy Hub, aiming to explore
how the latest recommender systems research could be translated into policy-
relevant information.
Micah Carroll's research on incentives for manipulation in recommender
systems was featured on the Berkeley Engineering website .

Michael Dennis’ s Youtube coverage:
Evolving Curricula with Regret-Based Environment Design
Unsupervised Environment Design(UED) is a ﬁeld which aims to automatically
generate environments which are appropriate to train agents in.  UED's ability
to promote transfer to unknown environments is an invaluable asset to safety ,
as well its ability to empower the AI designer to have more control over the
resulting policy through controlling the training distribution.
Two important insights in Unsupervised Environment Design (UED) are:
* High Regret Levels promote efﬁcient learning and transfer (See P AIRED,
Robust PLR)
* Evolution is more efﬁcient at optimizing environments than RL  (See POET)
In this work we combine these two threads, curating level edits to maximize
regret, and allowing evolution to compound this ef fect over time.
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 5/9ACCELL  achieves state of the art performance in every domain tested
including:
* Bipedal W alker (used in POET)
* Minigrid and Car Racing environments (used in P AIRED and Robust PLR)
You can stress test our method yourself with our interactive demo right in your
browser! (https://accelagent.github.io/)

CHAI visitor Professor Noyuri Mima of Future University (Hakodate,
Japan) gave a talk on " Let's develop empathy ". There was a talk on Jan.
29th on Noyuri's recent book "Living in the Age of AI '' at the biggest
bookstore in Hakodate.
Professor Mima wrote about Can Artiﬁcial Intelligence Make Us Happy?  ,
explaining the existence and function of AI in daily life and how to deal
with it properly .
Professor Mima also got coverage on organizations and Society V aluing
Diversity and on Community values that appear in the early childhood
section of bookstores in Asahi Newspaper .
Professor Mima wrote about  the decreasing number of girls' high schools in
Hokkaido: 26 private schools at their peak to 6  in the Hokkaido Newspaper .
Papers
Equity and Access in Algorithms, Mechanisms, and Optimization
(EAAMO)
Thomas Krendl Gilbert, Sarah Dean, Nathan Lambert and T om Zick,
Reward Reports for Reinforcement Learning . EAAMO 2022.

International Conference on Learning Representations
Cassidy Laidlaw and Anca Dragan, The Boltzmann Policy Distribution:
Accounting for Systematic Suboptimality in Human Models . ICLR 2022.

Micah Carroll, Jessy Lin, Orr Paradise, Raluca Georgescu, Mingfei Sun,
David Bignell, Stephanie Milani, Katja Hofmann, Matthew Hausknecht,
Anca Dragan, and Sam Devlin. Towards Flexible Inference in Sequential
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 6/9Decision Problems via Bidirectional Transformers . Workshop on
Generalizable Policy Learning in the Physical W orld, ICLR 2022
(expanded version accepted to NeurIPS 2022).

International Conference on Machine Learning
Micah Carroll, Dylan Hadﬁeld-Menell, Stuart Russell and Anca
Dragan,  Estimating and Penalizing Induced Preference Shifts in Recommender
Systems . ICML  2022.
Scott Emmons, Caspar Oesterheld, Andrew Critch, V ince Conitzer , and
Stuart Russell, For Learning in Symmetric Teams, Local Optima are Global
Nash Equilibria . ICML  2022.
Timon W illi*, Alistair Letcher*, Johannes T reutlein*, and Jakob Foerster ,
COLA: Consistent Learning with Opponent-Learning Awareness. ICML
2022.

Scientiﬁc Reports
Tom Lenaerts,  Inferring strategies from observations in long iterated Prisoner ’s
dilemma experiments , Scientiﬁc Reports 12, 7589 (2022).
Tom Lenaerts,  Delegation to artiﬁcial agents fosters prosocial behaviors in the
collective risk dilemma , Scientiﬁc Reports 12, 8492 (2022).

The Atlantic
Brian Christian, How a Google Employee Fell for the Eliza Ef fect. The
Atlantic, 21/6/22

arXiv
Dylan Hadﬁeld-Menell, Alon Halevy , Parisa Assar , Craig Boutilier , Amar
Ashar , Lex Beattie, Michael Ekstrand, Claire Leibowicz, Connie Moon
Sehat, Sara Johansen, Lianne Kerlin, David V ickrey , Spandana Singh,
Sanne V rijenhoek, Amy Zhang, McKane Andrus, Natali Helberger , Polina
Proutskova, T anushree Mitra and Nina V asan,  Building Human V alues into
Recommender Systems: An Interdisciplinary Synthesis , arXiv:2207.10192.
Erik Jenner , Herke van Hoof, Adam Gleave, Calculus on MDPs: Potential
Shaping as a Gradient , arXiv:2208.09570.
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 7/9
International Conference on Knowledge Representation and Reasoning
(KR)
Pulkit V erma, Siddharth Srivastava, and Shashank Marpally , Discovering
User-Interpretable Capabilities of Black-Box Planning Agents . KR 2022.

Other Papers
Pri Bengani, Luke Thorburn, What’ s Right and What’ s Wrong with Optimizing
for Engagement . Understanding Recommenders blog, 27/4/22
Alex T urner , Reward is not the optimization target . Alignment Forum, 25/7/22
Papers
Siddharth Srivastava Publishes “Why Can't You Do That, HAL? Explaining
Unsolvability of Planning T asks”
CHAI PI Siddharth Srivastava, along with his co-authors Sarath Sreedharan,
Rao Kambhampati, David Smith, published “Why Can’t You Do That, HAL?
Explaining Unsolvability of Planning Tasks”  in the 2019 International Joint
Conference on Artiﬁcial Intelligence (IJCAI) proceedings. The paper discusses
how, as anyone who has talked to a 3-year-old knows, explaining why
something can’t be done can be harder than explaining a solution to a
problem.
Thomas Gilbert Submits “The Passions and the Reward Functions: Rival
Views of AI Safety?” to F AT* 2020
Thomas Krendl Gilbert submitted “The Passions and the Reward Functions:
Rival V iews of AI Safety?” to the upcoming Fairness, Accountability , and
Transparency (F AT*) 2020 Conference.
The paper compares dif ferent approaches to AI Safety (e.g. CHAI, Partnership
on AI, V alue-Sensitive Design) with dif ferent schools of classical political
economy . It organizes these approaches into four distinct camps, summarizes
their key claims and disagreements, and outlines what each can learn from
each other with reference to foundational disputes about the status of free
markets with respect to human preferences and political sovereignty .
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 8/9CHAI Researchers’  Paper “On the Utility of Learning about Humans for
Human-AI Coordination” Accepted to NeurIPS 2019.
The paper authored by Micah Carroll, Rohin Shah, Tom Grifﬁths, Pieter Abbeel,
and  Anca Dragan, along with two other researchers not afﬁliated with CHAI,
was accepted to NeurIPS 2019. An ArXiv link for the paper will be available
shortly .
The paper is about how deep reinforcement learning has been used to play
DotA and Starcraft, using methods like self-play and population-based training,
which create agents that are very good at coordinating with themselves. The
agents "expect" their partners to be similar to them and are unable to predict
what human partners would do. In competitive games, this is ﬁne: if the human
deviates from optimal play , even if you don't predict it you will still beat them.
However , in cooperative settings, this can be a big problem. This paper
demonstrates this with a simple environment that requires strong coordination
based on the popular game Overcooked. It shows that agents speciﬁcally
trained to play alongside humans perform much better than self-play or
population-based training when paired with humans, particularly by adapting to
suboptimal human gameplay .
Vacancies
Internship
Application
The application are due by November 1 1th, 2022.
If you have any questions, please contact chai-admin@berkeley .edu
We seek to ﬁll the following positions:
The CHAI Research Fellowship
Research Collaborators
If none of these positions are the right ﬁt for you but you would still like
to express an interest in working with us, please ﬁll out this form .
General questions about jobs at CHAI can be sent to chai-
admin@berkeley .edu
Donations
07/04/2023, 07:54 CHAI Newsletter + Internship Application
https://mailchi.mp/humancompatible.ai/chai-newsletter -internship-application?e=[UNIQID] 9/9If you are interested in supporting CHAI, you can ﬁnd our online donation
page  here .
For any inquiries regarding donations or grants, please email  chai-
admin@berkeley .edu

View this email in your browser
To see more, visit us at humancompatible.ai
Copyright © 2022 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
2121 Berkeley W ay. Ofﬁce #8029. Berkeley , CA. 94704.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .


CHAI Newsletter #1 2022

5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 1/9CHAI Newsletter
This newsletter includes CHAI activities from the end of December 2021 to March
2022.
Research Highlight
5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 2/9
Thomas Krendl Gilbert has co-authored  a whitepaper  on a new form of AI
documentation, Reward Reports, which track what a given AI system is
optimizing for over time and how the terms of that optimization interact with the
surrounding dynamics of deployment.
Vacancies
CHAI has created a new  Research Fellowship  and is seeking applications for
5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 3/9the inaugural Fellows. CHAI Research Fellows will work with CHAI faculty
members Stuart Russell, Pieter Abbeel, and Anca Dragan and will also have
the opportunity to collaborate with  CHAI afﬁliate faculty  at Berkeley , other
faculty members in  Berkeley AI Research  (where CHAI is located), and many
other institutions. The Fellowship is aimed at training highly qualiﬁed
postdoctoral researchers to carry out research to advance beneﬁcial AI.
Researcher News
Researcher Updates
Stuart Russell has joined the inaugural cohort of  AI2050  fellows, which has
been tasked with solving the hard problems of AI. Eric and W endy Schmidt
have committed $125 Million in funding to the initiative over the next ﬁve years.
Professor Russell’ s work is focused on probabilistic programming with a goal of
improving AI’ s interpretability , provable safety , and performance.
Stuart Russell has been appointed to the UNESCO Expert Group on the
implementation of the  Recommendation on the Ethics of AI  adopted November
2021.
On March 21, Stuart Russell testiﬁed at the European Parliament on the
proposed AI Act.
Thomas Krendl Gilbert's team with Andrew Brozek, Megan W elle at the Notre
Dame-IBM T ech Ethics Lab  has been awarded $25,000  to produce a white
paper outlining a service for monitoring how training data translates to AI model
outputs and enabling recognition and correction of errors.
Noyuri Mima has gotten media coverage in a column of Hakodate Newspaper
on AI and empathy . She also delivered keynotes on the importance of Diversity ,
Equity , and Inclusion in the age of AI: " Why Can't Japan Close the Gender
Gap? " delivered at the University of Electro-Communications Symposium, and
what we should learn about the age of AI: “ Living in the Age of AI : Creativity
and Empathy to Design the Future ” delivered at  the Japan Environmental
Education Forum Kiyosato Meeting 2021. Noyuri also gave an author talk at the
"Living in the Age of AI" event at T sutaya Books.  The talk  was about what we
need to know in the age of AI.
Brian Christian spoke alongside Former Prime Minister of Norway Gro
Bruntland and mRNA vaccine pioneer Katalin Karikó about AI as one of the
essential trends af fecting the world, and speciﬁcally public health, in the 21st
5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 4/9century . The talk was a part of The Prince Mahidol A ward Conference
2022.  Understanding the Megatrends of the 21st Century — A Critical Step
Towards Getting the W orld W e Want was the ﬁrst main session of the
conference.
Brian Christian discussed his book  The Alignment Problem , and questions of AI
alignment more generally , with a group of UK MPs and Lords at the All-Party
Parliamentary Group for AI and the All-Party Parliamentary Group for Future
Generations. He aso gave a lecture about The Alignment Problem and
participated in a discussion about AI alignment with the Nebraska Governance
and T echnology Center Faculty Fellows. Brian Christian also spoke with UC
Berkeley students as part of the AI V alue Alignment Speaker Series, organized
by Mahendra Prasad.
Podcast Interviews
Brian Christian was interviewed by the Nebraska Governance and T echnology
Center's Gus Hurwitz for their  Tech Refactored podcast .
Stuart Russell Appearances
11/1/21, Reith Lecture 1, "The Biggest Event in Human History ," British Library ,
London
11/3/21, Reith Lecture 2, "AI in W arfare," University of Manchester
11/5/21, Reith Lecture 3, "AI in the Economy ," University of Edinburgh
11/10/21, Reith Lecture 4, "AI: A Future for Humans?" University of Newcastle
11/16/21, Keynote Lecture, "Provably Beneﬁcial Artiﬁcial Intelligence", 2021
Open Data Science Conference, San Francisco
11/24/21, Keynote Lecture, "Human-Compatible: How to Coexist With AI",
Munhwa Global AI Summit, Seoul, South Korea
11/30/21, Keynote Lecture, "Provably Beneﬁcial Artiﬁcial Intelligence", Council
of Europe 6th Plenary Conference on AI, Strasbourg
12/2/21, Keynote lecture and discussion, "Beneﬁcial AI", T ortoise AI Summit,
London
12/9/21, Keynote Lecture, "Provably Beneﬁcial Artiﬁcial Intelligence", AI
International Symposium, Korea Advanced Institute of Science and T echnology
(KAIST), Seoul, South Korea
12/10/21, Keynote Lecture, "Provably Beneﬁcial Artiﬁcial Intelligence",
Conference for the 50th Anniversary of the Founding of Bangladesh, Dhaka,
Bangladesh
5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 5/912/15/21, Keynote Lecture, "Human Compatible: the Future of Artiﬁcial
Intelligence", Epoch Foundation 30th Anniversary , Taipei, T aiwan
12/15/21, Plenary Invited Lecture, "Human Compatible Artiﬁcial
Intelligence"W orkshop on Interpretability , Safety and Security in AI, Alan T uring
Institute, London
2/27/22, Plenary Lecture, "W ell-Founded, Human-Compatible AI: Some
Thoughts", Stanford Existential Risk Institute Annual Meeting, Stanford
2/28/22, Invited lecture, "AI and Education: Some Thoughts", OECD Education
Policy Meeting  on High Performing Systems for T omorrow , Paris
3/4/22, Plenary Lecture, "AI: Beyond Deep Learning", CT O Forum on Future
Directions in AI, San Francisco
3/11/22, Invited lecture, "Human-Compatible AI", Department of Philosophy and
College of Engineering, CalPoly San Luis Obispo
3/25/22, Keynote Lecture, "Human-Compatible AI", Intelligent User Interfaces
(IUI) conference, Helsinki

Papers
Technology in Society
The Anh Han, T om Lenaerts, Francisco C. Santos, Luís Moniz
Pereira,  Voluntary safety commitments provide an escape from
over-regulation in AI development , Technology in Society , Volume
68
International Conference on Learning Representations
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov , Sergey
Levine,  RvS: What is Essential for Ofﬂine RL via Supervised
Learning? , ICLR2022
Association for the Advancement of Artiﬁcial Intelligence
Rashmeet Kaur Nayyar , Pulkit V erma, and Siddharth
Srivastava,  Differential Assessment of Black-Box AI Agents ,
AAAI2022
5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 6/9International Conference on Autonomous Agents and Multiagent
Systems
Naman Shah, Pulkit V erma, T revor Angle, Siddharth
Srivastava,  JEDAI: A System for Skill-Aligned Explainable Robot
Planning , AAAMS 2022
UC Berkeley Center for Long-T erm Cybersecurity
Thomas Krendl Gilbert, Sarah Dean, T om Zick, Nathan
Lambert,  Choices, Risks, and Reward Reports: Charting Public
Policy for Reinforcement Learning Systems , (UC Berkeley CL TC
White Paper Series)
Conference on Neural Information Processing Systems
Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Y uandong T ian,
Joseph E. Gonzalez, and Stuart Russell,  MADE: Exploration via
Maximizing Deviation from Explored Regions . In Advances in Neural
Information Processing Systems 34, 2022. (Proceedings of
NeurIPS-21.)
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and
Stuart Russell,  Bridging Ofﬂine Reinforcement Learning and
Imitation Learning: A T ale of Pessimism.  In Advances in Neural
Information Processing Systems 34, 2022. (Proceedings of
NeurIPS-21.)
Cassidy Laidlaw and Stuart Russell,  Uncertain Decisions Facilitate
Better Preference Learning.  In Advances in Neural Information
Processing Systems 34, 2022. (Proceedings of NeurIPS-21.)
Cynthia Chen, Sam T oyer, Cody Wild, Scott Emmons, Ian Fischer ,
Kuang-Huei Lee, Neel Alex, Steven W ang, Ping Luo, Stuart Russell,
Pieter Abbeel, and Rohin Shah, An Empirical Investigation of
Representation Learning for Imitation. In Advances in Neural
Information Processing Systems 34, 2022. (Proceedings of
NeurIPS-21, Datasets and Benchmarks.)
International Conference on Robotics and Automation
Justin Svegliato, Connor Basich, Sandhya Saisubramanian, Shlomo
Zilberstein,  Metareasoning for Safe Decision Making in Autonomous
Systems , ICRA 2022
International Conference on Automated Planning and Scheduling
5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 7/9Abhinav Bhatia, Justin Svegliato, Samer Nashed, Shlomo
Zilberstein,  Tuning the Hyperparameters of Anytime Planning: A
Metareasoning Approach with Deep RL , ICAPS 2022
Other Papers
Arnaud Fickinger , Samuel Cohen, Stuart Russell, Brandon
Amos,  CROSS-DOMAIN IMIT ATION LEARNING VIA OPTIMAL
TRANSPOR T
Mehdi Dadvar , Keyvan Majd, Elena Oikonomou, Georgios Fainekos,
Siddharth Srivastava,  Joint Communication and Motion Planning for
Cobots
Pulkit V erma, Shashank Rao Marpally , Siddharth
Srivastava,  Discovering User-Interpretable Capabilities of Black-Box
Planning Agents
Arnaud Fickinger , Hengyuan Hu, Brandon Amos, Stuart Russell,
Noam Brown,  Scalable Online Planning via Reinforcement Learning
Fine-T uning
Political Economy of Reinforcement Learning Systems (PERLS)
workshop
Sponsored by the Center for Human-Compatible AI at UC Berkeley , and with
support from the Simons Institute and the Center for Long-T erm Cybersecurity ,
we convened a cross-disciplinary group of researchers to examine the near-
term policy concerns of Reinforcement Learning (RL). RL is a rapidly growing
branch of AI research, with the capacity to learn to exploit our dynamic behavior
in real time. From Y ouTube’s recommendation algorithm to post-surgery opioid
prescriptions, RL algorithms are poised to permeate our daily lives. The ability
of the RL system to tease out behavioral responses, and the human
experimentation inherent to its learning, motivate a range of crucial policy
questions about RL ’s societal implications that are distinct from those
addressed in the literature on other branches of Machine Learning (ML).
We began addressing these issues as part of the  2020 Simons Institute
program on the Theory of Reinforcement Learning , and throughout 2020/21 we
broadened the discussion through an  ongoing reading group , includingEvents
5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 8/9perspectives from Law and Policy . The aim of this workshop was to establish a
common language around the state of the art of RL across key societal
domains. From this examination, we identiﬁed speciﬁc interpretive gaps that
can be elaborated or ﬁlled by members of our community . Our ultimate goal
was to map near-term societal concerns and indicate possible cross-
disciplinary avenues towards addressing them.
Link:  https://neurips.cc/virtual/2021/workshop/21864
Visit from T oby Ord
Toby Ord , Senior Research Fellow in Philosophy at Oxford University , Effective
Altruism, visited CHAI on March 14, 2022. His work focuses on the big picture
questions facing humanity , the long term future of humanity , and the risks which
threaten to destroy our entire potential. While visiting CHAI, he spoke with
students, met with Stuart Russell, and presented at the weekly CHAI seminar
in-person.
Weekly CHAI Seminars
Every W ednesday we host our Beneﬁcial AI Seminar from 10 am to 12 pm
PST. Seminars currently take place remotely through Zoom.  Here  is the
schedule for the Beneﬁcial AI seminars.
If you would like to attend the virtual seminars, please email  chai-
admin@berkeley .edu.

We are extremely grateful for the generosity of each of our donors, and for their
support of our work. Thank you to Open Philanthropy for their recent grant of
$1,126,260 to BERI, earmarked for compute resources to be used by the
Center for Human-Compatible AI!
If you are interested in supporting CHAI, you can ﬁnd our online donation
page  here .
For any inquiries regarding donations or grants, please email  chai-
admin@berkeley .edu.Donations

5/4/22, 3:17 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-1 1452829 9/9To see more, visit us at humancompatible.ai
Copyright © 2020 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
2121 Berkeley W ay. Ofﬁce #8029. Berkeley , CA. 94704.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .


CHAI Newsletter #3 2021

5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/8faimbkd80 1/6CHAI Newsletter
This newsletter includes CHAI activities from October 2021 to December 2021.
Research Highlight
CHAI co-published a report  with the W orld Economic Forum. The “Positive AI
Economic Futures” report was written by CHAI’ s Stuart Russell and Caroline
Jeanmaire along with Daniel Susskind of Oxford University . It attempts to
articulate positive and realistic visions of a future where Artiﬁcial Intelligence
can do most of what we currently call work. Y ou can read a summary of the
report here.
Two Kavli Centers for Ethics, Science, and the Public  – at the University of
California, Berkeley , and the University of Cambridge – are launching to
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/8faimbkd80 2/6engage the public in identifying and exploring ethical considerations and
impacts born from scientiﬁc discovery . Stuart Russell will be the inaugural
Director of the new center at Berkeley , and AI will be one of the three initial foci
along with neuroscience and gene editing.
Vacancies
CHAI Research Fellowships  are available for recent and soon-to-be PhDs for
postdoctoral research with CHAI faculty .
The Kavli Center for Ethics, Science, and the Public at Berkeley  is seeking an
Executive Director .
The Berkeley Institute for Data Science  has an opening for a Postdoctoral
Scholar in Responsible AI
Organizational News
New postdoc Justin Svegliato  joined CHAI, will be mentored by Stuart Russell.
Justin received his PhD in Computer Science at UMass Amherst where he was
advised by Shlomo Zilberstein. The goal of his research is to build autonomous
systems that operate in the open world for long periods of time in an efﬁcient,
reliable, and ethical way . Recently , his work has focused on ensuring that
autonomous systems comply with dif ferent ethical theories, such as
utilitarianism, Kantianism, and prima facie duties. His work has also focused on
using metareasoning to enable autonomous systems to recover from
exceptions, maintain and restore safe operation, and optimize the performance
of anytime planning algorithms for bounded rationality .
Researcher News
Researcher Updates
Brian Christian gives keynote at the Linux Foundation Member Summit
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/8faimbkd80 3/6Brian Christian gave a keynote address  at the annual Linux Foundation
Member Summit  about the alignment problem and the role the open-
source community can play in addressing issues of AI safety .
Brian Christian’ s book was named by Microsoft CEO Satya Nadella as one
of the 5 books that inspired him in 2021.
"Clear and compelling," he writes, the book "moves us from the
theoretical to the practical while attempting to answer one of our
industry’ s most pressing questions: How do we teach machines, and what
should we teach them?"
Noyuri Mima  published her book, Living in the Age of AI : Creativity and
Empathy to Design the Future
The author , a current visitor to CHAI from Future University Hakodate in
Japan, has academic backgrounds in computer science, cognitive
psychology , and education. In the book she explains the history of AI and
how it works, and talks about how to create a better society , as well as
what each of us can do, using "creativity and empathy" as keywords.
Stuart Russell Appearances
Stuart Russell gives the 2021 Reith Lectures , receives OBE (Ofﬁce of the
Most Excellent Order of the British Empire) (1 1.1. 2021 - 1.19.2021)
Stuart Russell delivered the 2021 Reith Lectures. The Reith Lecture
series  began in 1948 and is considered the most prestigious lecture
series in the English-speaking world. Previous lecturers include Bertrand
Russell, Arnold T oynbee, J. Robert Oppenheimer , George Kennan, J. K.
Galbraith, Edward Said, W ole Soyinka, and Stephen Hawking. Russell is
the ﬁrst computer scientist selected for this honor . The lectures took place
in London, Manchester , Edinburgh, and Newcastle in early November and
were broadcast weekly in December on BBC Radio 4 and the BBC W orld
Service.
On November 9, Stuart Russell visited Windsor Castle and received the
OBE from Prince William, Duke of Cambridge.
Papers
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/8faimbkd80 4/6NeurIPS 2021
Michael Dennis, Minqi Jiang, Jack Parker-Holder , Jakob Foerster , Edward
Grefenstette, T im Rocktäschel Replay-Guided Adversarial Environment
Design
Arnaud Fickinger , Stuart Russell, Brandon Amos, Noam Brown Scalable
Online Planning via Reinforcement Learning Fine-T uning
Cassidy Laidlaw , Stuart Russell Uncertain Decisions Facilitate Better
Preference Learning
Xin Chen, Sam T oyer, Cody Wild, Scott Emmons, Ian Fischer , Kuang-
Huei Lee, Neel Alex, Steven H W ang, Ping Luo, Stuart Russell, Pieter
Abbeel, Rohin Shah An Empirical Investigation of Representation
Learning for Imitation
Alexander Matt T urner , Logan Smith, Rohin Shah, Andrew Critch, Prasad
Tadepalli Optimal Policies T end T o Seek Power , NeurIPS 2021 spotlight
Kimin Lee, Laura Smith, Anca Dragan, Pieter Abbeel B-Pref:
Benchmarking Preference-Based Reinforcement Learning
Tianjun Zhang*, Paria Rashidinejad*, Jiantao Jiao, Y uandong T ian,
Joseph E. Gonzalez, Stuart Russell, MADE: Exploration via Maximizing
Deviation from Explored Regions
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, Stuart Russell,
Bridging Ofﬂine Reinforcement Learning and Imitation Learning: A T ale of
Pessimism
Events
Weekly CHAI Seminars
Every W ednesday we host our Beneﬁcial AI Seminar from 10 am to 12 pm
PST. Seminars currently take place remotely through Zoom. Here  is the
schedule for the Beneﬁcial AI seminars.
If you would like to attend the virtual seminars, please email chai-
admin@berkeley .edu.

5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/8faimbkd80 5/6Political Economy of Reinforcement Learning (PERLS) W orkshop  - December
14
Sponsored by the Center for Human-Compatible AI at UC Berkeley , and with
support from the Simons Institute and the Center for Long-T erm Cybersecurity ,
this cross-disciplinary group of researchers was convened to examine the near-
term policy concerns of Reinforcement Learning (RL). The workshop was
organized by CHAI’ s Michael Dennis, Thomas Krendl Gilbert, and Stuart
Russell along with T om Zick and Aaron Snoswell. It featured talks from Mark
Nitzberg and Stuart Russell.
Donations
We are extremely grateful for the generosity of each of our donors, and for their
support of our work. Thank you to each of the following:
1. $50,000 from Whipple-Scrumptious Inc. via Ef fective Giving.
2. $611,500 from DALHAP Investments via Ef fective Giving.
3. $250,000 from Kwan W ai Meng and Ngo Hwee Ngah via Ef fective
Giving.
4. €65,000 from Stichting Powoki via Ef fective Giving.
5. $10,000 from Andrea La Mesa via Founders Pledge.
6. $248,000 from Jaan T allinn via Survival and Flourishing Fund for the
BERI-CHAI Collaboration Fund.
7. $250,000 from Jed McCaleb via Survival and Flourishing Fund for the
BERI-CHAI Collaboration Fund.
We appreciate the $898,000 to Andrew Critch’ s fund from Survival and
Flourishing Fund via Founders Pledge.
Thank you again to all of our donors for your support!
If you are interested in supporting CHAI, you can ﬁnd our online donation page
here.
For any inquiries regarding donations or grants, please email chai-
admin@berkeley .edu.
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/8faimbkd80 6/6To see more, visit us at humancompatible.ai
Copyright © 2020 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
2121 Berkeley W ay. Ofﬁce #8029. Berkeley , CA. 94704.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .


CHAI Newsletter #2 2021

5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 1/8CHAI Newsletter
This newsletter includes CHAI activities from June 2021 to August 2021.
Research Highlight
On 7-8 June, CHAI held its ﬁfth annual workshop. Over 150 attendees,
including professors, students, industry researchers, and policymakers met
virtually to discuss AI safety and beneﬁcial systems.
Participants came from over 100 institutions and attended 26 talks, six
dialogues, one panel, and plenaries given by Stuart Russell, Peter Eckersley ,
Glen W eyl, Adam Kalai, and Anca Dragan. Attendees participated in organized
meet-ups and conversation in the virtual space Gather .town. V iew the
workshop program here .
Vacancies
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 2/8CHAI Operations Assistant
BERI is seeking a part-time operations assistant as part of their collaboration
with the Center for Human-Compatible Artiﬁcial Intelligence. View the listing
and application here .
CHAI Research Fellowship
If you are interested in postdoctoral positions at CHAI, please ﬁll out this
application . If you are interested in visiting CHAI to collaborate on research in
other roles, please ﬁll out this application .
Please email chai-admin@berkeley .edu if you have any questions.
Organizational News
J.P. Gonzales Named Executive Assistant to
Stuart Russell
In August, James Paul “J.P ." Gonzales began
the role of Executive Assistant to Professor
Stuart Russell, employed by BERI. He provides
administrative, event-planning, and PR
assistance to Prof. Russell and CHAI with the
primary goal of supporting Prof. Russell's AI
safety research and advocacy goals. J.P . holds a
bachelor's in Cognitive Science from UC Merced
and a master's in Experimental Psychology from
Arizona State University .
Donations
CHAI Received Donation from Whipple-Scrumptious
Whipple-Scrumptious donated $50,000 to CHAI through Ef fective Giving.
Thank you to the generous donors who continue to make the work at CHAI
possible.
Researcher News
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 3/8Researcher Updates
Seven CHAI members are helping launch The MineRL BASAL T Competition on
Learning from Human Feedback  at NeurIPS 2021. CHAI members involved
with BASAL T include Rohin Shah, Cody Wild, Steven W ang, Neel Alex, Pieter
Abbeel, Stuart Russell, and Anca Dragan. Read more on the BAIR blog  and the
MineRL website .
Stuart Russell was named Ofﬁcer of the Most Excellent Order of the British
Empire by HM Queen Elizabeth “for services to artiﬁcial intelligence research.”
CHAI afﬁliate Alison Gopnik, Professor of Psychology at UC Berkeley , won the
Carl Sagan prize  for promoting science.
CHAI Members Continuing to New Roles
Congratulations to Karthika Mohan, CHAI postdoctoral scholar , who has
accepted the position of Assistant Professor at Oregon State University .
Dylan Hadﬁeld-Menell, Assistant Professor at MIT , published his thesis The
Principal-Agent Alignment Problem in Artiﬁcial Intelligence.
Thanard Kurutach graduated with the thesis Learning, Planning, and Acting
with Models and will work at Cruise as an AI/ML Research Scientist.
Congratulations to our Research Engineers Steven W ang, who is starting a
Master ’s of Computer Science at ETH Zurich, and Cody Wild, who is joining
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 4/8Google Research, as well as Operations Assistant Noor Brody , who is now a
Data Engineer at SimpleLab Inc.
Podcast Interviews
The New Y ork T imes podcast The Ezra Klein
Show features CHAI afﬁliate Brian Christian in an
episode  titled “If ‘All Models Are W rong,’ Why Do
We Give Them So Much Power?”
Daniel Filan’ s podcast  AXRP  released new episodes.
The Sunday Show podcast interviewed Jonathan Stray in the episode  “Can
social media help depolarize society?” They discuss his latest paper , Designing
Recommender Systems to Depolarize , which will appear in First Monday .
Stuart Russell Appearances
Stuart Russell participated  in Debates on the Future of Robotics Research, a
virtual workshop at the International Conference on Robotics and Animation
(ICRA) 2021. CHAI alum Jaime Fernández Fisac served as General Co-Chair
on the organizing team. W atch recordings of the debates at
roboticsdebates.org .
Stuart Russell gave a featured interview in the BBC Panorama documentary ,
“Are you scared yet, human? ” He gave the keynote "How Not to Destroy the
World With AI" to the IJCAI W orkshop on Adverse Impacts of Artiﬁcial
Intelligence  and the French National AI conference Plate-Forme Intelligence
Artiﬁcielle, Bordeaux. He also presented it as an invited lecture to the Ecole des
Hautes Etudes Commerciales de Paris (HEC). He gave the keynote "Provably
Beneﬁcial AI and the Problem of Control," at the National Academies
Conference on "How Artiﬁcial Intelligence and Machine Learning T ransform the
Human Condition." He also presented it as the inaugural lecture in the Prato
Dialog series . He gave the keynote “Poorly Designed AI” to the EU Humane AI
Network and the keynote “AI for the Comprehensive Nuclear-T est-Ban T reaty"
to the CTBT O Science and T echnology Summit, V ienna.
Papers
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 5/8ICML 2021
Kimin Lee, Laura Smith, and Pieter Abbeel, PEBBLE: Feedback-Efﬁcient
Interactive Reinforcement Learning via Relabeling Experience and
Unsupervised Pre-training . ICML 2021
Johannes T reutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster , A
New Formalism, Method and Open Issues for Zero-Shot Coordination . ICML
2021
Smitha Milli, Luca Belli, Moritz Hardt, Causal Inference Struggles with Agency
on Online Platforms . ICML 2021
Kimin Lee, Michael Laskin, Aravind Srinivas, Pieter Abbeel,  SUNRISE: A
Simple Uniﬁed Framework for Ensemble Learning in Deep Reinforcement
Learning . ICML 2021 Spotlight paper
NeurIPS 2021
Kimin Lee, Laura Smith, Anca Dragan, Pieter Abbeel, B-Pref: Benchmarking
Preference-Based Reinforcement Learning . Accepted to NeurIPS 2021
Benchmark track
ICLR 2021
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell and Jan Leike,
Quantifying Dif ferences in Reward Functions . ICLR 2021
Cassidy Laidlaw , Sahil Singla, Soheil Feizi, Perceptual Adversarial Robustness:
Defense Against Unseen Threat Models . ICLR 2021
Other V enues
Roel Dobbe, Thomas Krendl Gilbert, Y onatan Mintz, Hard Choices in Artiﬁcial
Intelligence . Artiﬁcial Intelligence V olume 300, November 2021
Smitha Milli, Luca Belli, Moritz Hardt, From Optimizing Engagement to
Measuring V alue. FAccT 2021
Charlotte Roman, Michael Dennis, Andrew Critch, and Stuart
Russell,  Accumulating Risk Capital Through Investing in Cooperation . In Proc.
AAMAS-21.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, Dawn Song,
Natural Adversarial Examples . CVPR 2021
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank W ang,
Evan Dorundo, Rahul Desai, T yler Zhu, Samyak Parajuli, Mike Guo, Dawn
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 6/8Song, Jacob Steinhardt, Justin Gilmer , The Many Faces of Robustness: A
Critical Analysis of Out-of-Distribution Generalization . ICCV 2021
Avik Jain, Lawrence Chan, Daniel S. Brown, Anca D. Dragan, Optimal Cost
Design for Model Predictive Control . Proceedings of the 3rd Conference on
Learning for Dynamics and Control, PMLR 144:1205-1217, 2021
George Matheos, Alexander K. Lew , Matin Ghavamizadeh, Stuart Russell,
Marco Cusumano-T owner , Vikash K. Mansinghka, Transforming W orlds:
Automated Involutive MCMC for Open-Universe Probabilistic Models . In Proc.
3rd Symposium on Advances in Approximate Bayesian Inference (AABI), 2021
Stuart Russell, Human-Compatible Artiﬁcial Intelligence . In Stephen Muggleton
and Nick Chater (eds.), Human-Like Machine Intelligence, Oxford University
Press, 2021
Stuart Russell Updates
New translations of Human Compatible:
Stuart Russell, İnsanlık için Y apay Zekâ: Y apay Zekâ ve Kontrol Problemi .
Translated to T urkish by Barış Satılmış. Ankara: Buzdağı Y ayınevi, 2021.
Stuart Russell, 어떻게  인간과  공존하는  인공지능을  만들  것인가. Translated to
Korean by Lee Han-eum. Seoul: Gimm-Y oung, 2021.
Stuart Russell, AI新⽣. Translated to Japanese by Nobuhiko Matsui.
Tokyo: Misuzu Shobo, 2021.
Stuart Russell,  Сумісний з людиною. Штучний інтелект і проблема
контролю , Translated to Ukrainian by V iktoria Zenhyea. Kiev: Bookchef,
2020.
Stuart Russell published two articles on the topic of lethal autonomous
weapons:
Stuart Russell, It's time to ban autonomous killer robots before they
become a threat , Financial T imes. August 5, 2021.
Stuart Russell, Anthony Aguirre, Emilia Javorsky and Max T egmark,
Lethal Autonomous W eapons Exist; They Must Be Banned . IEEE
Spectrum, June 16, 2021.
Stuart Russell published two chapters in Reﬂections on Artiﬁcial Intelligence for
Humanity , based on the Global Forum on AI for Humanity held in Paris in 2019:
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 7/8Raja Chatila, V irginia Dignum, Michael Fisher , Fosca Gianotti, Katharina
Morik, Stuart Russell, David Sadek, and Karen Y eung, “T rustworthy AI.” In
Bertrand Braunschweig and Malik Ghallab (eds.), Reﬂections on Artiﬁcial
Intelligence for Humanity , Springer , 2021.
Jocelyn Maclure and Stuart Russell, “AI for Humanity: The Global Challenges.”
In Bertrand Braunschweig and Malik Ghallab (eds.), Reﬂections on Artiﬁcial
Intelligence for Humanity , Springer , 2021.
Events
Weekly CHAI Seminars
Every W ednesday we host our Beneﬁcial AI Seminar from 10 am to 12 pm
PST. Seminars currently take place remotely through Zoom. Here  is the
schedule for the Beneﬁcial AI seminars.
If you would like to attend the virtual seminars, please email chai-
admin@berkeley .edu.
Donations
If you are interested in supporting CHAI, you can ﬁnd our online donation page
here.
For any inquiries regarding donations or grants, please email chai-
admin@berkeley .edu.
To see more, visit us at humancompatible.ai
Copyright © 2020 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
2121 Berkeley W ay. Ofﬁce #8029. Berkeley , CA. 94704.
Want to change how you receive these emails?
5/4/22, 3:16 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-9352805 8/8You can update your preferences  or unsubscribe from this list .


CHAI Newsletter #1 2021

5/4/22, 3:15 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-7967265 1/6CHAI Newsletter
This newsletter includes CHAI activities from January 2021 to May 2021.
Research Highlight
The Science article Who needs a teacher? Artiﬁcial intelligence designs lesson
plans for itself  features three recent papers co-authored by members of CHAI
(in parentheses below).
The papers, which explore the learning paradigm known as autocurricula, are
Emergent Complexity and Zero-shot T ransfer via Unsupervised Environment
Design  (Michael Dennis, Stuart Russell, Andrew Critch), A Novel Automated
Curriculum Strategy to Solve Hard Sokoban Planning Instances (Bart Selman),
and Automatic Curriculum Learning through V alue Disagreement  (Pieter
Abbeel).
5/4/22, 3:15 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-7967265 2/6Vacancies
CHAI Research Fellowship
CHAI has created a new Research Fellowship and is seeking applications for
the inaugural Fellows. The Fellowship is aimed at training highly qualiﬁed
postdoctoral researchers to carry out research to advance beneﬁcial AI. Find
more information and apply here . If you are interested in visiting CHAI to
collaborate on research in other roles, please ﬁll out this application .
Executive Assistant to Professor Stuart Russell
We are seeking an experienced and dedicated executive assistant to help
Professor Stuart Russell manage tasks related to his roles as a professor and
public ﬁgure. Please see the application on the website  of our partner
organization BERI.
Please email chai-admin@berkeley .edu if you have any questions.
Organizational News
New  Collaborators
Jonathan Stray joins CHAI as a V isiting Scholar
in his role as Research Scientist at BERI.
Jonathan conducts research into the ways in
which social media content selection algorithms
affect human well-being. At CHAI, he works
on the design of recommender systems for better
personalized news and information.
Professor T om Lenaerts joins CHAI as a V isiting
Scholar from the Computer Science Department
of the Université Libre de Bruxelles. Prof.
Lenaerts and Michael Dennis are
investigating the relationship between translucent
players and commitment devices, considering an
evolutionary game theory perspective.
New Collaborators Joar Skalse (Oxford) and Oliver Richardson (Cornell) will
work alongside Adam Gleave. Antoni Lorente (King’ s College London,
5/4/22, 3:15 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-7967265 3/6Universitat Autònoma de Barcelona) will work through the fall with Thomas
Krendl Gilbert.
2021 Intern Cohort
CHAI welcomes thirteen interns in 2021, our largest internship cohort. The
following interns will virtually work with CHAI researchers this year: Y awen
Duan (University of Hong Kong), Rafael Albert (R WTH Aachen University),
Nathan Miller (UC Berkeley), Pavel Czempin (T echnical University of Munich),
Lauro Langosco di Langosco (ETH Zürich), Matthew Farrugia-Roberts
(University of Melbourne), Thomas W oodside (Y ale), Y ulong Lin (Cambridge),
Neel Nanda (Cambridge), Erik Jenner (Universiteit van Amsterdam), Joseph
Benton (Cambridge), Carlo Attubato (Oxford), and Michelle Li (MIT).
Donations
This spring CHAI received a donation of $2,334.68 from Founders Pledge as
well as a total of $17,746.40 from other donors via our online donation page .
We are grateful for the ongoing support of CHAI's work.
Researcher News
Researcher Updates
Congratulations to Thomas Krendl Gilbert , who is
graduating with his PhD in Machine Ethics and
Epistemology . This fall he will join the Digital Life
Initiative at Cornell T ech as a postdoc and work
on expanding the PERLS (Political Economy of
Reinforcement Learning Systems) community .
Congratulations to  Vael Gates , who is graduating
with their PhD in Neuroscience with a focus on
computational cognitive science. This fall V ael will
start as a postdoc at Stanford HAI / CISAC,
where they intend to work on an ethnography of
AI researchers.
The LA T imes Book Prizes  named The Alignment Problem  by Brian Christian a
ﬁnalist for best Science & T echnology book of 2020.
5/4/22, 3:15 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-7967265 4/6Rachel Freedman spoke  to the Berkeley High School STEMinist Club on the
topic of human-compatible AI research.
Pieter Abbeel launched a new podcast, The Robot
Brains , where he hosts leading experts in AI robotics.
The T alkRL podcast recently featured  Michael
Dennis  and Tom Gilbert .
Daniel Filan’ s podcast AXRP  released three new episodes featuring
conversations with Beth Barnes, V anessa Kosoy , and Evan Hubinger .
Stuart Russell accepted invitations to join the Scientiﬁc Advisory Board of the
Sorbonne Center for AI in Paris and the Academic Committee of the T singhua
University AI International Governance Institute.
In the last quarter Stuart Russell gave a number of talks on the subject of
human-compatible AI. He presented a plenary  in the WEF 2021 Global
Technology Governance Summit, a lecture at Applied Physics Lab Colloquium
at Johns Hopkins University , a lecture at the MIT China Innovation and
Entrepreneurship Forum, a panel at the Oxford China Forum, a keynote at
the Center for the Future Mind at Florida Atlantic University , a lecture at the
Microsoft Research New England Colloquium, the 18th Kim Okgill Memorial
Lecture at Ewha W omen's University , and a keynote at the inaugural
Conference of the Stanford Existential Risk Institute.
Papers
Scott Emmons, Andrew Critch, and Stuart Russell published Symmetry ,
Equilibria, and Robustness in Common-Payof f Games  in the Games and
Incentives W orkshop 2021 .
CHAI researchers and former interns Daniel Filan, Stephen Casper , Shlomi
Hod, Cody Wild, Andrew Critch, and Stuart Russell posted Clusterability in
Neural Networks  on arXiv .
CHAI researchers and collaborators Paria Rashidinejad, Banghua Zhu, Cong
Ma, Jiantao Jiao, and Stuart Russell posted Bridging Ofﬂine Reinforcement
Learning and Imitation Learning: A T ale of Pessimism  on arXiv .
Current and former CHAI members including Paul Knott, Micah Carroll, Anca
Dragan, and Rohin Shah co-authored the paper Evaluating the Robustness of
5/4/22, 3:15 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-7967265 5/6Collaborative Agents  alongside Sam Devlin, Kamil Ciosek, and Katja Hofmann.
Thomas Krendl Gilbert co-authored AI Development for the Public Interest:
From Abstraction T raps to Sociotechnical Risks , published in IEEE IST AS20 .
He also published the white paper Mapping the Political Economy of
Reinforcement Learning Systems: The Case of Autonomous V ehicles  on the
Simons Institute website.
Brian Christian, CHAI afﬁliate and Simons Institute Science Communicator in
Residence, published the article  Watch and Learn: Ofﬂine Reinforcement
Learning  on the Simons Institute website.
CHAI faculty and afﬁliates presented the following papers at AAAI 2021 :
Probabilistic Dependency Graphs  by Oliver Richardson and Joseph Y .
Halpern at Cornell University
Asking the Right Questions: Learning Interpretable Action Models
Through Query Answering  by Pulkit V erma, Shashank Rao Marpally , and
Siddharth Srivastava at Arizona State University
Unifying Principles and Metrics for Safe and Assistive AI  by Siddharth
Srivastava
Events
Weekly CHAI Seminars
Every W ednesday we host our Beneﬁcial AI Seminar from 10 am to 12 pm
PST. Seminars currently take place remotely through Zoom. Here  is the
schedule for the Beneﬁcial AI seminars.
If you would like to attend the virtual seminars, please email chai-
admin@berkeley .edu.
Donations
If you are interested in supporting CHAI, you can ﬁnd our online donation page
here.
For any inquiries regarding donations or grants, please email chai-
admin@berkeley .edu.
5/4/22, 3:15 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu-7967265 6/6To see more, visit us at humancompatible.ai
Copyright © 2020 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
2121 Berkeley W ay. Ofﬁce #8029. Berkeley , CA. 94704.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .


CHAI Newsletter #3 2020

5/4/22, 3:40 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu 1/7CHAI Newsletter
This newsletter includes CHAI activities from October 2020 through January 2021.
Through distance and transition, CHAI had much to celebrate in 2020. Our
researchers published numerous papers, remotely mentored the 2020 intern
cohort, and welcomed six new PhD students. W e brought together 150
participants in CHAI’ s ﬁrst virtual workshop and congratulated CHAI members
continuing on to new research positions and professorships.
We are thankful for your support of CHAI’ s mission to reorient AI research
towards provably beneﬁcial systems, and we look forward to moving into 2021
together .
Vacancies
Postdoctoral Researcher
5/4/22, 3:40 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu 2/7If you are interested in postdoctoral positions at CHAI, please ﬁll out this
application . If you are interested in visiting CHAI to collaborate on research in
other roles, please ﬁll out this application .
Personal Assistant to Professor Stuart Russell
We are seeking an experienced and dedicated personal assistant to help
Professor Stuart Russell manage tasks related to his roles as a professor and
public ﬁgure. Please see the application on the website  of our partner
organization BERI.
Please email chai-admin@berkeley .edu if you have any questions.
Organizational News
Annual Progress Report
CHAI published its 2020 Progress Report  highlighting some of the most
important research that CHAI has produced since its inception.
The report also documents our contributions beyond the AI community
including contributions to public awareness of the risks from AI systems,
contributions to world leaders’ awareness, and connecting with China. Lastly ,
the report lays out some of our future plans for the next few years.
Donations
Open Philanthropy awarded a gift of $1 1,355,246.00 to support CHAI’ s work
into 2026.
The Survival and Flourishing Fund  donated $799,000 to CHAI, as well as
$247,000 to the Berkeley Existential Risk Initiative to be used for the BERI-
CHAI Collaboration. On Giving T uesday , BERI also received over $46,000.
Ben Hoskin made a personal donation of $20,000. His support enables our
mission to reorient AI research towards provably beneﬁcial systems.
CHAI gives many thanks to all the generous donors who continue to support
their work.
Third Positive AI Economic Futures W orkshop
On December 17, CHAI hosted the third Positive AI Economic Futures
Workshop in collaboration with the W orld Economic Forum. More than 100
participants met virtually to plan scenarios of a future economy and society
5/4/22, 3:40 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu 3/7transformed by AI. The workshop brought together AI experts, science ﬁction
authors, economists, policymakers, and social scientists.
Researcher News
Stuart Russell Appearances
Stuart Russell debated Melanie
Mitchell, Professor at the Santa Fe
Institute, in the latest episode of The
Munk Debates, a debate series on
major policy issues. In the episode,
titled The Rise of Thinking Machines,
Prof. Russell argued for the
question, “Be it resolved, the quest
for true AI is one of the great
existential risks of our time.” Listen to
the debate here or on any podcast
service.
In the last quarter , Prof. Russell presented a keynote  at Forum Humanum , a
panel  at the Paris Peace Forum , a keynote  and roundtable  at the Governance
Of and By Digital T echnology conference , a keynote at The Hague Conference
on Responsible AI for Peace, Justice and Security , the annual lecture  at the UC
San Diego Institute for Practical Ethics, a keynote at the Fujitsu Labs Annual
Technical Symposium , and a lecture at the Oxford Review of Economic Policy
editorial seminar .
Researcher Updates
Rohin Shah published his PhD dissertation, “ Extracting and Using Preference
Information from the State of the W orld.” He is now working as a Research
Scientist on the technical AGI safety team at DeepMind.
In December 2020 CHAI PhD candidate Daniel Filan
launched the AI X-risk Research Podcast (AXRP) . In
each episode, Daniel interviews the author of a paper
and discusses how it might reduce the existential risk of
AI. In the ﬁrst three episodes, Daniel features CHAI
5/4/22, 3:40 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu 4/7researchers Andrew Critch, Rohin Shah, and Adam
Gleave.
Caroline Jeanmaire, Director of Strategic
Research and Partnerships at CHAI, has been
recognized as one of 100 Brilliant W omen in AI
Ethics in 2021 . The list is published annually by
Women in AI Ethics, which has the mission to
make AI more diverse and accessible, in part by
recognizing rising stars in the ﬁeld.
On October 21 2020, CHAI celebrated the launch of The Alignment Problem  by
UC Berkeley V isiting Scholar and CHAI Afﬁliate Brian Christian. Journalist Nora
Young hosted an interview with Christian and an audience Q&A. The Alignment
Problem tells the story of the ethics and safety movement in AI and reports
directly from those working on the ﬁeld’ s frontier , including current CHAI
researchers. Read more  and watch a recording of the interview and Q&A here.
New Afﬁliates
CHAI added two new afﬁliates, Jakob Foerster and Moritz Hardt.
Jakob Foerster is an Assistant Professor at the
University of T oronto, where he has been awarded a
CIFAR AI Chair , and is a faculty member at the V ector
Institute. Most recently he has worked as a Research
Scientist at Facebook AI Research. He was the lead
organizer of the ﬁrst Emergent Communication
workshop at NeurIPS in 2017, which he has helped
organize ever since. During his PhD studies at Oxford
University , he helped bring deep multi-agent
reinforcement learning to the forefront of AI research.
Moritz Hardt is an Assistant Professor in the
Department of Electrical Engineering and Computer
Sciences at UC Berkeley . He investigates algorithms
and machine learning with a focus on reliability , validity ,
and societal impact. After obtaining a PhD in Computer
Science from Princeton University , he held positions at
5/4/22, 3:40 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu 5/7
IBM Research Almaden, Google Research and Google
Brain. Prof. Hardt is a co-founder of the W orkshop on
Fairness, Accountability , and T ransparency in Machine
Learning (F AT/ML) and a co-author of the forthcoming
textbook “Fairness and Machine Learning”. He has
received an NSF CAREER award, a Sloan fellowship,
and best paper awards at ICML 2018 and ICLR 2017.
Papers
NeurIPS 2020 Papers
NeurIPS 2020 acceptances co-authored by CHAI researchers include these
main conference papers (asterisk denotes equal contribution):
Scott Emmons*, Ajay Jain*, Michael Laskin*, Thanard Kurutach, Pieter
Abbeel, Deepak Pathak. “ Sparse Graphical Memory for Robust Planning ”
Sam T oyer, Rohin Shah, Andrew Critch, Stuart Russell. “ The MAGICAL
Benchmark for Robust Imitation ”
Michael Dennis, Natasha Jaques, Eugene V initsky , Alexandre Bayen,
Stuart Russell, Andrew Critch, Sergey Levine. “ Emergent Complexity and
Zero-shot T ransfer via Unsupervised Environment Design ” (Oral
presentation, Reinforcement Learning)
Paria Rashidinejad, Jiantao Jiao, Stuart Russell. “ SLIP: Learning to
Predict in Unknown Dynamical Systems with Long-T erm Memory ” (Oral
presentation, Dynamical Sys/Density/Sparsity)
Alexander Matt T urner , Neale Ratzlaf f, Prasad T adepalli. “ Avoiding Side
Effects in Complex Environments ” (Spotlight presentation, Reinforcement
Learning)
Yuqing Du, Stas T iomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel,
Anca Dragan. “ AvE: Assistance via Empowerment ”
Hong Jun Jeon, Smitha Milli, Anca D. Dragan. “ Reward-Rational (Implicit)
Choice: A Unifying Formalism for Reward Learning ”
Kush Bhatia, Ashwin Pananjady , Peter Bartlett, Anca Dragan, and Martin
Wainwright. “ Preference Learning Along Multiple Criteria: A Game-
theoretic Perspective ”
5/4/22, 3:40 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu 6/7In addition, the following papers appeared in workshops:
Pedro Freire, Adam Gleave, Sam T oyer, Stuart Russell. “ DERAIL:
Diagnostic Environments for Reward and Imitation Learning ” (Deep RL
Workshop)
Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitrii
Krasheninnikov , Lawrence Chan, Michael Dennis, Pieter Abbeel, Anca
Dragan, Stuart Russell. “ Beneﬁts of Assistance over Reward Learning ”
(Cooperative AI W orkshop; won a Best Paper A ward)
Eric J. Michaud, Adam Gleave, Stuart Russell. “ Understanding Learned
Reward Functions ” (Deep RL W orkshop)
ICLR 2021 Papers
Adam Gleave, Michael D. Dennis, Shane Legg, Stuart Russell, Jan Leike.
“Quantifying Dif ferences in Reward Functions ” (Spotlight paper)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika,
Dawn Song, Jacob Steinhardt. “ Measuring Massive Multitask Language
Understanding ”
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn
Song, Jacob Steinhardt. “ Aligning AI With Shared Human V alues ”
David Lindner , Rohin Shah, Pieter Abbeel, Anca Dragan. “ Learning What T o Do
by Simulating the Past ”
IEEE IST AS20 Paper
Thomas Krendl Gilbert co-authored “ AI Development for the Public Interest:
From Abstraction T raps to Sociotechnical Risks ,” published in IEEE IST AS20 .
IJCAI 2020 AI Safety W orkshop
“Choice Set Misspeciﬁcation in Reward Inference ” by Rachel Freedman, Rohin
Shah, and Anca Dragan received the Best Paper A ward at the IJCAI 2020
AISafety workshop.
Events
Weekly CHAI Seminars
Every W ednesday we host our Beneﬁcial AI Seminar from 10 am to 12 pm
PST. Seminars currently take place remotely through Zoom. Here  is the
schedule for the Beneﬁcial AI seminars.
5/4/22, 3:40 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/humancompatible.ai/aqhgtk12mu 7/7If you would like to attend the virtual seminars, please email chai-
admin@berkeley .edu.
Donations
If you are interested in supporting CHAI, you can ﬁnd our online donation page
here.
For any inquiries regarding donations or grants, please email chai-
admin@berkeley .edu.
To see more, visit us at humancompatible.ai
Copyright © 2020 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
2121 Berkeley W ay. Ofﬁce #8029. Berkeley , CA. 94704.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .


CHAI Newsletter #2 2020

5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 1/114th Annual CHAI W orkshop
In June, CHAI held its fourth annual workshop! Due to COVID-19, the workshop
was held online.
The two-day workshop brought together professors, students, and researchers
from 65 institutions sharing a common interest in reducing existential and
societal-scale risks from advanced AI. There were 150 attendees, by far the
largest number since the series began in 2017. Sessions included plenaries,
individual and dialogue talks, and a panel discussion. Attendees were also able
to continue their conversations from sessions and casually chat in the virtual
environment CHAI T own, hosted by  Gather . A little over 50% of participants
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 2/11reported making signiﬁcant changes to their work as a result of the workshop.
The event itself was private, but  the schedule is listed here  for reference.
The Alignment Problem by Brian Christian
Longstanding CHAI participant Brian Christian has published The Alignment
Problem: Machine Learning and Human V alues, which chronicles the growth
and progress of the ﬁeld of technical AI safety , highlighting its recent milestones
and open problems. The book showcases the work of many CHAI researchers,
including Pieter Abbeel, Andrew Critch, Anca Dragan, Daniel Filan, Jaime
Fernandez Fisac, V ael Gates, Alison Gopnik, T om Grifﬁths, Dylan Hadﬁeld-
Menell, Dan Hendryks, Michael Littman, Smitha Milli, Stuart Russell, Satinder
Singh, Jacob Steinhardt, and Alex T urner .
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 3/11 Christian is a V isiting Scholar at UC Berkeley , where he also works with
CITRIS and the Simons Institute. His previous writing includes the acclaimed
bestsellers Algorithms to Live By (with T om Grifﬁths) and The Most Human
Human, which have been translated into nineteen languages.
 You can currently order the book in its US edition , and preorder it in its UK
edition .
Organizational News
New Afﬁliate Faculty
CHAI added ﬁve new afﬁliate members: Rediet Abebe, Niko Kolodny , Nika
Haghtalab, Brian Christian, and V incent Corruble. W elcome!
Rediet Abebe is a Junior Fellow at the Harvard Society of Fellows and an
incoming Assistant Professor of Computer Science at UC Berkeley . Her
research is in the ﬁelds of algorithms and AI, with a focus on optimization,
network science, and their applications to social justice. Abebe co-founded the
Mechanism Design for Social Good and Black in AI initiatives to improve
access to opportunity  for underserved communities.
Niko Kolodny is Professor of Philosophy at UC Berkeley . He works in moral and
political philosophy and has written on the nature of rationality and the future of
humanity . Recently he designed a new undergraduate course, Moral Questions
of Data Science, which explores the ethics of collecting, drawing inferences
from, and acting on data, especially on a large scale.
Nika Haghtalab is an Assistant Professor of Computer Science at Cornell
University . She researches the theoretical aspects of machine learning and
algorithmic economics. Her work focuses on developing a theory for machine
learning that accounts for its interactions with people and organizations and
their diversity of social and economic behavior .
Brian Christian is the author of The Most Human Human (201 1) and Algorithms
to Live By (2016). His new book published in October 2020, The Alignment
Problem , explores what goes wrong when we build AI systems and the
movement to ﬁx them. He is currently Journalist in Residence at the Simons
Institute. Christian holds degrees in computer science, philosophy , and poetry
from Brown University and the University of W ashington.
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 4/11Vincent Corruble is an Associate Professor at Sorbonne Université. He has
contributed to research on multi-agent reinforcement learning with applications
to affective computing in games, medicine, and urban simulation. Lately he has
developed an interest in AI safety , speciﬁcally on how agents and humans can
develop cooperation and trust over time.

Summer 2020 Interns
This summer CHAI members virtually mentored eight interns. CHAI interns
develop essential research skills in the ﬁelds of machine learning and AI safety ,
preparing them for graduate school and industry opportunities.
1. Cynthia Chen, an undergrad from University of Hong Kong, was mentored
by Sam and Scott, and they were working on using tools in causality to
assist ML with a focus on imitation learning.
2. Eric Michaud, an undergrad at UC Berkeley , was mentored by Adam, and
they were working on reward model interpretability .
3. Stephen Casper , an undergrad at Harvard University , was mentored by
Daniel, and they were working on clustering/modularity .
4. Charlotte Roman, PhD student at University of W arwick (UK),
was mentored by Michael, and they were working on detecting and
defending against policy exploitation in MARL.
5. Harry Giles, a maths graduate who spent a few years working in ﬁnance,
was mentored by Lawrence. They were looking at theoretical justiﬁcations
of modeling human behavior on Boltzmann rationality in discrete and
continuous cases.
6. Johannes T reutlein, an incoming MSc in CS student at University of
Toronto (formerly at T echnical University of Berlin), was mentored by
Michael and Jakob Foerster . They were working on multi-agent RL.
7. Michael McDonald, incoming M.S. student at UC Berkeley and former
undergrad at UC Berkeley , was mentored by Dylan, and they
were working on imitation learning for multi-goal tasks.
8. Sergei V olodin, a Master student at the Swiss Federal Institute of
Technology in Lausanne (EPFL) studying Computer Science and
Neuroscience was mentored by Adam. They were working on defenses
for adversarial policies.
Researcher News

5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 5/11This summer , CHAI alumni,  Jaime Fernández Fisac , began working as an
Assistant Professor of Electrical Engineering at Princeton University . Jaime’ s
work combines safety analysis, machine learning techniques, and insights from
cognitive science to enable robots to strategically plan their interaction with
humans.
Rohin Shah  and Dylan Hadﬁeld-Menell  have accepted jobs after the completion
of their PhD studies. Rohin will work at DeepMind as a Research Scientist and
Dylan will begin work as an Assistant Professor at MIT in July 2021.
Thomas Krendl Gilbert , PhD candidate in Machine Ethics and Epistemology ,
has been awarded the Simons Institute Law and Society Fellowship for its
upcoming research program on Theory of Reinforcement Learning.
Alyssa Dayan, an incoming PhD student advised by Stuart Russell, was
awarded a fellowship  from The Hertz Foundation.
CHAI was featured in three recent episodes of the Future of Life Podcast.
Professor Stuart Russell  joined Harvard Psychology Professor Steven Pinker to
discuss the foundations, beneﬁts, and existential threat of AI. PhD student
Rohin Shah  and MIRI researcher Buck Shlegeris shared their thoughts on the
current state of research ef forts for beneﬁcial AI. Research scientist Andrew
Critch  discussed the paper “AI Research Considerations for Human Existential
Safety ,” coauthored with David Krueger .
Professor Stuart Russell’ s book Human Compatible was published in Chinese .
Professor Stuart Russell gave several noteworthy presentations over the last
few months.
On May 26, he gave the annual T uring Lecture, sponsored by the Alan
Turing Institute in London, on the topic of "Provably Beneﬁcial Artiﬁcial
Intelligence". Over 700 people attended, making it the most highly
attended T uring Lecture so far . The lecture itself is here. Prof. Russell
also provided written answers to the many questions from audience
members here. His talk was also written about here.
On June 16, Professor Russell spoke at the W orld Peace Forum, held in
Beijing, on the subject of lethal autonomous weapons.
On August 30, he gave the keynote lecture for the annual European
Conference on AI, held at Santiago de Compostela, Spain, on the subject
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 6/11of "How Not to Destroy the W orld With AI."
On September 10, his book Human Compatible was the subject of a
meet-the-author session at the American Political Science Association's
annual conference in San Francisco.
Finally , on September 22, he gave the keynote lecture for the annual
United Nations Global Summit on AI for Good, held in Geneva. His talk
was titled "On Beneﬁcial AI" and asked how we are to deﬁne "Good" in
the phrase "AI for Good".
Papers
Andrew Critch Coauthors ARCHES
CHAI Research Scientist Andrew Critch and David Krueger coauthored “AI
Research Considerations for Human Existential Safety (ARCHES).”  The report
attempts to represent the ways in which a wide variety of AI research directions
relate to existential safety . It also introduces the concept of prepotence, a
property of hypothetical AI systems that is weaker than human-level AI, but
which is adequate to pose a substantial existential risk.
NeurIPS 2020 Accepts Seven Papers by CHAI Researchers
Seven papers co-authored by CHAI researchers were accepted at NeurIPS
2020:
1. “The MAGICAL Benchmark for Robust Imitation”  by Sam T oyer, Rohin
Shah, Andrew Critch, and Stuart Russell. Existing benchmarks for
imitation learning and inverse reinforcement learning only evaluate how
well agents can satisfy the preferences of a human demonstrator in the
speciﬁc setting where demonstrations were provided. This paper
proposes the MAGICAL benchmark, which makes it possible to evaluate
how well agents satisfy demonstrator preferences in a range of situations
that dif fer systematically from the one in which demonstrations were
provided.
2. “Emergent Complexity and Zero-shot T ransfer via Unsupervised
Environment Design” by Michael Dennis, Natasha Jaques, Eugene
Vinitsky , Alexandre Bayen, Stuart Russell, Andrew Critch and Sergey
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 7/11Levine (PDF will be available later). This paper was accepted for oral
presentation. Many approaches for robustness and transfer learning
require specifying a distribution of tasks in which a policy will be trained.
However , existing approaches to generate environments suf fer from
common failure modes: domain randomization rarely generates structure,
and minimax adversarial training leads to unsolvable environments. W e
propose generating environments which minimize worst case regret, to
generate difﬁcult but solvable environments. T o approximate this, we
introduce a second, antagonist agent that is allied with an adversary
which chooses the parameters of the environment. W e call our technique
Protagonist Antagonist Induced Regret Environment Design (P AIRED).
Our experiments demonstrate that P AIRED produces a natural curriculum
of increasingly complex environments, and higher zero-shot transfer
performance.
3. "SLIP: Learning to predict in unknown dynamical systems with long-term
memory"  by Paria Rashidinejad, Jiantao Jiao, and Stuart Russell. This
paper was accepted for an oral presentation. The authors consider the
problem of prediction in unknown and partially observed linear dynamical
systems (LDS). When the system parameters are known, the optimal
linear predictor is the Kalman ﬁlter . When the system parameters are
unknown, the performance of existing predictive models is poor in
important classes of LDS that are only marginally stable and exhibit long-
term forecast memory . The authors investigate the possibility of a uniform
approximation by analyzing a generalized version of the Kolmogorov
width of the Kalman ﬁlter coefﬁcient set. This motivates the design of an
algorithm, which the authors call spectral LDS improper predictor (SLIP),
based on conducting a tight convex relaxation of the Kalman predictive
model via spectral methods. The authors provide a ﬁnite-sample analysis,
showing that our algorithm competes with the Kalman ﬁlter in hindsight
with only logarithmic regret. The authors’ regret analysis relies on
Mendelson's small-ball method and circumvents concentration,
boundedness, or exponential forgetting requirements, providing a sharp
regret bound independent of mixing coefﬁcient and forecast memory .
Empirical evaluations demonstrate that the authors’ algorithm outperforms
state-of-the-art methods in LDS prediction.
4. “Avoiding Side Ef fects in Complex Environments”  by Alex T urner , CHAI
visiting researcher . This paper was accepted for a spotlight talk. Reward
function speciﬁcation can be difﬁcult, even in simple environments.
Realistic environments contain millions of states. Rewarding the agent for
making a widget may be easy , but penalizing the multitude of possible
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 8/11negative side ef fects is hard. In toy environments, Attainable Utility
Preservation (AUP) avoids side ef fects by penalizing shifts in the ability to
achieve randomly generated goals. W e scale this approach to large,
randomly generated environments based on Conway's Game of Life. By
preserving optimal value for a single randomly generated reward function,
AUP incurs modest overhead, completes the speciﬁed task, and avoids
side ef fects.
5. “AvE: Assistance via Empowerment”  by Yuqing Du , Stas T iomkin , Emre
Kiciman , Daniel Polani , Pieter Abbeel , and Anca Dragan . The paper
addresses the problem of goal inference in assistive artiﬁcial agents with
a new paradigm that increases the human’ s ability to control their
environment. They formalize this approach by augmenting reinforcement
learning with human empowerment and successfully demonstrate their
method in a shared autonomy user study for a challenging simulated
teleoperation task with human-in-the-loop training.
6. “Reward-rational (implicit) choice: A unifying formalism for reward
learning”  by Hong Jun Jeon, Smitha Milli, and Anca Dragan. It is often
difﬁcult to hand-specify what the correct reward function is for a task, so
researchers have instead aimed to learn reward functions from human
behavior or feedback. The types of behavior interpreted as evidence of
the reward function have expanded greatly in recent years. The authors’
key insight is that dif ferent types of behavior can be interpreted in a single
unifying formalism - as a reward-rational choice that the human is making,
often implicitly . The formalism of fers both a unifying lens with which to
view past work, as well as a recipe for interpreting new sources of
information that are yet to be uncovered. The authors provide two
examples to showcase this: interpreting a new feedback type, and
reading into how the choice of feedback itself leaks information about the
reward.
7. “Preference learning along multiple criteria: A game-theoretic perspective”
by Kush Bhatia, Ashwin Pananjady , Peter Bartlett, Anca Dragan, and
Martin W ainwright. In this paper , the authors generalize the notion of a
von Neumann winner to the multi-criteria setting by taking inspiration from
Blackwell’ s approachability . Their framework allows for nonlinear
aggregation of preferences across criteria, and generalizes the
linearization-based approach from multi-objective optimization
IJCAI-20 Accepts T wo Papers by Rachel Freedman
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 9/11CHAI PhD student Rachel Freedman will present two papers accepted to IJCAI
2020. The ﬁrst paper , “Choice Set Misspeciﬁcation in Reward Inference ,” is
coauthored with CHAI Professor Anca Dragan and PhD student Rohin Shah.
The paper analyzes what happens when a robot inferring reward functions from
human feedback makes incorrect assumptions about the human’ s choice set.
The second paper , “Aligning with Heterogeneous Preferences for Kidney
Exchange ,” addresses the problem of preference aggregation by AI algorithms
in a real-world public health context: kidney exchange. The paper suggests a
roadmap for future automated moral decision making on behalf of
heterogeneous groups.
Thomas Krendl Gilbert Publishes “Subjectifying Objectivity”
CHAI PhD student Thomas Krendl Gilbert and collaborator Andrew Loveridge
published “Subjectifying objectivity: Delineating tastes in theoretical quantum
gravity research”  in Social Studies of Science.
CHAI Researchers Published in Cognitive Science
PhD student V ael Gates and Professors Anca Dragan and T om Grifﬁths
published “ How to Be Helpful to Multiple People at Once ” in the journal
Cognitive Science. The authors consider the problem of assisting multiple
recipients with very dif ferent preferences, with one aim of constraining the
space of desirable behavior in assistive artiﬁcial intelligence systems.
Joseph Halpern Presents at the 2020 Conference on Uncertainty in
Artiﬁcial Intelligence
Cornell Professor Joseph Halpern and Xinming Liu presented “ Bounded
Rationality in Las V egas: Probabilistic Finite Automata Play Multi-Armed
Bandits ” at the 2020 Conference on Uncertainty in Artiﬁcial Intelligence. In the
paper , they leverage the multi-armed bandit problem to test the hypothesis that
humans’ limited computational power helps explain their predictably irrational
behavior . Using a probabilistic ﬁnite automaton to play a multi-armed bandit,
they ﬁnd that as the number of states of the automaton decreases, it exhibits a
number of standard human biases.
Michael W ellman Paper Accepted by IJCAI-20
"Market Manipulation: An Adversarial Learning Framework for Detection and
Evasion ," a new paper by University of Michigan Professor Michael W ellman
and Xintong W ang, has been accepted by IJCAI-20. In the paper , they propose
an adversarial learning framework to capture the evolving game between a
regulator who develops tools to detect market manipulation and a manipulator
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 10/11who obfuscates actions to evade detection. Their experimental results
demonstrate the possibility of automatically generating a diverse set of unseen
manipulation strategies that can facilitate the training of more robust detection
algorithms.
Events
Weekly CHAI Seminars
Every W ednesday we host our Beneﬁcial AI Seminar from 10 am to 12 pm PST
and our CHAI Discussion Seminar from 12 to 2 pm PST . Seminars currently
take place remotely through Zoom.
Here  is the schedule for the Beneﬁcial AI seminars, and  here is the schedule
for the Discussion Seminars.
If you would like to attend the virtual seminars, please email  chai-
admin@berkeley .edu.
Vacancies
2021 Summer Internship Applications will be available soon. The
application will be posted on the CHAI website.
Visiting Scholars and Researchers
If you are interested in visiting CHAI to collaborate on research, please ﬁll out
this application .
Please email  chai-admin@berkeley .edu if you have any questions.
Donations

If you are interested in supporting CHAI, then you can ﬁnd our online donation
page here.
For any inquiries regarding donations or grants, please email  chai-
admin@berkeley .edu.
5/4/22, 3:38 PM CHAI Newsletter
https://mailchi.mp/humancompatible.ai/chai-q4-newsletter -intern-applications-due-soon-5068337 11/11To see more, visit us at humancompatible.ai
Copyright © 2019 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
621D Stuardja Dai Hall. UC Berkeley . Berkeley , CA. 94720.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .


CHAI Newsletter #1 2020

5/4/22, 3:37 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/lists/chaiq12020 1/4New CHAI Building!
Berkeley Way West
Our new ofﬁces are in Berkeley W ay W est along with the Berkeley Artiﬁcial
Intelligence Research (BAIR) Lab.
Photo credit: Capital Strategies
Organizational News
Toby Ord and the Centre for Effective Altruism Donated to CHAI
CHAI would like to thank T oby Ord and the Centre for Ef fective Altruism for their
generous donation. W e greatly appreciate their commitment to CHAI and the
continuation of AI safety research.
AI Economic Futures W orkshop Grant
CHAI members Stuart Russell and Caroline Jeanmaire received a $50,000
5/4/22, 3:37 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/lists/chaiq12020 2/4grant from the Good V entures Foundation to support a workshop with the W orld
Economic Forum. The workshop centers around designing positive AI policies
related to employment.
Congratulations to Micah Carroll for choosing UC Berkeley for his PhD!
Micah graduated with a B.A. in Statistics from UC Berkeley in 2019 and will be
continuing his work at CHAI in the fall.
Researcher News
The MIT T echnology Review  highlighted Adam Gleave  in a feature on his paper
“Adversarial Policies: Attacking Deep Reinforcement Learning,” coauthored
with fellow CHAI members Michael Dennis, Cody Wild, Neel Kant, Sergey
Levine, and Stuart Russell. Their research was accepted to be presented at the
International Conference on Learning Representations (ICLR).
Caroline Jeanmarie, Director of Strategic Research and Partnerships at
CHAI, gave a presentation at the 2020 Foresight AGI Strategy Meeting on the
U.S. Guidance for Regulation of AI Applications.
Professor Anca Dragan was featured  on the popular podcast Artiﬁcial
Intelligence with Lex Fridman. They discussed human-robot interaction during
semi-autonomous driving and reward engineering.  Professor Dragan also
spoke  at the WIRED25 summit, explaining some of the challenges robots face
when interacting with people.
CHAI PhD Student Rohin Shah  spoke  on the Machine Ethics podcast about
alignment problems in AI, constraining AI behavior , current AI vs future AI,
recommendation algorithms and extremism, and other topics.
Blog Posts
Adam Gleave wrote a blog post, "Physically Realistic Attacks on Deep
Reinforcement Learning"
CHAI PhD student Adam Gleave wrote a new  blog post  on the BAIR Blog. It
provides a rundown of the vulnerabilities of deep reinforcement learning
policies to adversarial attacks. The blog post accompanies a recently
5/4/22, 3:37 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/lists/chaiq12020 3/4revised  paper , "Adversarial Policies: Attacking Deep Reinforcement
Learning,"  coauthored with additional members of CHAI.
Papers
CHAI student Smitha Milli and Professor Anca Dragan coauthored a paper ,
“Reward-rational (implicit) choice: A unifying formalism for reward learning.”

The Conference on AI, Ethics, and Society (AIES) 2020 accepted a paper ,
“Hard Choices in Artiﬁcial Intelligence: Addressing Normative Uncertainty
through Sociotechnical Commitments,” coauthored by CHAI machine ethics
researcher Thomas Gilbert.

Events
Weekly CHAI Seminars
Every W ednesday we host our interdisciplinary seminar from 12 to 2 pm and
our technical seminar from 3 to 5 pm. While they are typically held on the eighth
ﬂoor of Berkeley W ay W est, seminars currently take place remotely through
Zoom.
If you would like to attend the virtual seminars, then please e-mail chai-
admin@berkeley .edu.
Here  is the schedule for the interdisciplinary seminars, and here is the schedule
for the technical seminars.
Vacancies
Visiting Scholars and Researchers
If you are interested in visiting CHAI to collaborate on research, please ﬁll
out this application .
Please contact  chai-admin@berkeley .edu for any questions.
5/4/22, 3:37 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/lists/chaiq12020 4/4To see more, visit us at humancompatible.ai
Copyright © 2020 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
621D Sutardja Dai Hall. UC Berkeley . Berkeley , CA. 94720.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .