CHAI Newsletter #4 2019

5/4/22, 3:36 PM CHAI Q4 Newsletter + Intern Applications Due Soon
https://mailchi.mp/13c99f6887ba/chai-q4-newsletter -intern-applications-due-soon 1/7Human Compatible: Artiﬁcial Intelligence
and the Problem of Control

Stuart Russell's new book, Human Compatible: Artiﬁcial Intelligence and the
Problem of Control, is available on Amazon .
Professor Russell explains why he has come to consider his own discipline an
existential threat to our species, and lays out how we can change course before
it's too late. The book explores the idea of intelligence in humans and in
5/4/22, 3:36 PM CHAI Q4 Newsletter + Intern Applications Due Soon
https://mailchi.mp/13c99f6887ba/chai-q4-newsletter -intern-applications-due-soon 2/7machines, describes the beneﬁts we can expect (from intelligent personal
assistants to vastly accelerated scientiﬁc research), and outlines the AI
breakthroughs that still have to happen before we reach superhuman AI.
Professor Russell spells out the ways humans are already ﬁnding to misuse AI,
from lethal autonomous weapons to the manipulation of opinions on a global
scale. Finally , he explains why we must ensure that we never lose control of
machines more powerful than we are. He suggests that we can rebuild AI on a
new foundation, according to which machines are designed to be inherently
uncertain about the human preferences they are required to satisfy .
Also, Elon Musk endorsed the book on Twitter !
Professor Russell has been doing a lot of interviews for this book. Please check
out some of his recent interviews on TechCrunch , Vox, The W all Street Journal ,
The Guardian , Financial T imes , BBC, and The New Y ork T imes .
For media inquiries, please contact Caroline Jeanmaire
(caroline.j@berkeley .edu).
Organizational News
Internship Application's Deadline is Coming Up! Application is Due by
12/15 (Sunday)
If you are interested in being an intern at CHAI in 2020, then please ﬁll out this
application ! The application closes 12/15 which is this upcoming Sunday .
This internship is an excellent opportunity for undergraduate and graduate
students to contribute to technical AI safety research and develop research
skills. W e also rely on word-of-mouth so if you know anyone who is interested
in applying, please share the application with them! If you have any questions
regarding the internship, please contact chai-admin@berkeley .edu.
Founders Pledge's Members Donated to CHAI
CHAI would like to thank Founders Pledge's members for their generous
donation. W e greatly appreciate their commitment to ensure that CHAI
continues to produce world class technical AI safety research.
Open Philanthropy Project Donated to CHAI
CHAI would like to thank Open Philanthropy Project for their generous
5/4/22, 3:36 PM CHAI Q4 Newsletter + Intern Applications Due Soon
https://mailchi.mp/13c99f6887ba/chai-q4-newsletter -intern-applications-due-soon 3/7supplement of their grant to CHAI. Open Philanthropy Project has been an
extraordinary supporter of CHAI since 2016 and we thank them for their
continued commitment to CHAI and the AI safety academic ﬁeld.
David Lindner Joins CHAI as an Intern
David is a master's student at ETH Zurich working on reinforcement learning.
Broadly , his research is concerned with building reliable intelligent systems that
interact with the world. Recently he has been interested in learning robust
reward functions from human feedback and ways to reason about the
uncertainty an RL agent has about its reward function. At CHAI, David is
working with Rohin Shah on inferring human preferences from the current state
of the world in complex environments.
Researcher News

Stuart Russell's Reddit AMA on 12/16
Professor Russell will have an 'Ask Me Anything' session on Monday ,
December 16th from 9-1 1am PST . The AMA will be live on the front page
of https://www .reddit.com/r/books .
Joe Halpern Gave Presentations at V arious Conferences and Universities
Joe Halpern, CHAI PI and Professor of Computer Science and Engineering at
Cornell University , gave presentations at multiple dif ferent conferences and
universities around the world.
5/4/22, 3:36 PM CHAI Q4 Newsletter + Intern Applications Due Soon
https://mailchi.mp/13c99f6887ba/chai-q4-newsletter -intern-applications-due-soon 4/7He was a Keynote speaker and gave a survey talk on causality at
the International Conference on Data Mining.
He also gave talks about blameworthiness, intention, distributed computing,
common knowledge, and causality at University of Wisconsin-Madison, North
Carolina State University , Summer School on Cybersecurity and Privacy in
Stockholm, and the International W orkshop on Causality , Probability , and
Logic.
Blog Posts
Rohin Shah and Micah Carroll W rote a Blog Post T itled, "Collaborating
with Humans Requires Understanding Them"
Rohin Shah, CHAI PhD student, and Micah Carroll, CHAI Intern, wrote this blog
post on the BAIR Blog.
Here is a short summary of the blog post: "Recently , deep reinforcement
learning has been used to play Dota and Starcraft, using methods like self-play
and population-based training, which create agents that are very good at
coordinating with themselves. They 'expect' their partners to be similar to them;
they are unable to predict what human partners would do. In competitive
games, this is ﬁne: if the human deviates from optimal play , even if you don't
predict it you will still beat them. However , in cooperative settings, this can be a
big problem. W e demonstrate this with a simple environment that requires
strong coordination based on the popular game Overcooked. W e show that
agents speciﬁcally trained to play alongside humans perform much better than
self-play or population-based training when paired with humans, particularly by
adapting to suboptimal human gameplay ."
Papers
Rachel Freedman Published Paper About Adapting a Kidney Exchange
Algorithm
Rachel Freedman, CHAI PhD student, wrote this paper  with Professor V incent
Conitzer and other researchers at Duke University and the University of
Maryland. The paper was published by The AI Journal and was funded by a FLI
AI Safety grant.
5/4/22, 3:36 PM CHAI Q4 Newsletter + Intern Applications Due Soon
https://mailchi.mp/13c99f6887ba/chai-q4-newsletter -intern-applications-due-soon 5/7The paper presents an end-to-end methodology for identifying and quantifying
human value frameworks that af fect kidney allocation, incorporating these into
the donor-matching algorithm used for the U.S. national kidney exchange, and
running simulations to evaluate the impact of this modiﬁed algorithm on
matching.
NeurIPS Conference Accepted Paper by Rohin Shah and Dmitrii
Krasheninnikov
Rohin Shah, CHAI PhD student, and Dmitrii Krasheninnikov , former CHAI
intern, had their paper accepted by the 2019 NeurIPS Conference. The paper  is
titled, "Combining Reward Information from Multiple Sources."
Here is the abstract of the paper: "Reward functions can be learned from many
sources of information, such as instructions, demonstrations, and comparisons
between behaviors. Ideally , we would be able to combine rewards learned from
multiple sources of information. This paper takes a generic view of the problem:
we assume that we have two rewards from dif ferent sources, that may conﬂict
due to misspeciﬁcation, and try to combine information from both to get a
distribution over reward functions. W e identify several desiderata for such a
method, and analyze several algorithms along these dimensions both
theoretically and empirically , including a novel algorithm, Multitask Inverse
Reward Design (MIRD)."
Joe Halpern Recently Published a Few Papers
Joe Halpern, CHAI PI and Professor of Computer Science and Engineering at
Cornell University , published a few papers within the past few months.
Professor Halpern published:
1. "Approximate Causal Abstraction "
2. "A Conceptually W ell-Founded Characterization of Iterated Admissibility
Using an 'All I Know' Operator "
3. "On the Existence of Nash Equilibrium in Games with Resource-Bounded
Players "
AI, Ethics, and Society Conference Accepted Paper by Alex T urner and
Dylan Hadﬁeld-Menell
Alex T urner , former CHAI V isiting Researcher , and Dylan Hadﬁeld-Menell,
CHAI PhD student, and Prasad T adepalli had their paper , "Conservative
5/4/22, 3:36 PM CHAI Q4 Newsletter + Intern Applications Due Soon
https://mailchi.mp/13c99f6887ba/chai-q4-newsletter -intern-applications-due-soon 6/7Agency via Attainable Utility Preservation"  accepted by the 2020 AI, Ethics, and
Society Conference.
Here is the abstract of the paper: "An agent pursuing a misspeciﬁed goal can
become unable to achieve the goal the designers had in mind. For example, a
robotic factory assistant could break expensive equipment. T o mitigate this risk,
we balance the optimization of the primary goal with the maintanence of the
ability to achieve auxiliary goals. W e show that this eliminates negative side
effects in a suite of toy environments, even when the auxiliary goals are
randomly generated."
Events
Weekly CHAI Seminars
Every W ednesday from 10am to 12pm and 4pm to 6pm we have our technical
and interdisciplinary seminars respectively in the Newton Room of Sutardja Dai
Hall (Room 730) at UC Berkeley . If you are interested in attending, please
contact chai-admin@berkeley .edu.
Here  is the schedule for the seminars.
Vacancies
Internship
Application
The application will close on December 15th  (this upcoming Sunday).
The internship is mainly for undergraduate and graduate students who
are interested in contributing to technical AI safety research. W e also
accept international applicants.
If you have any questions, please contact chai-admin@berkeley .edu
Postdoctoral Researcher
We are currently hiring for a postdoctoral researcher who has a strong
track record of high quality research in relevant academic areas.
For more information, please go here.
Please contact chai-admin@berkeley .edu for any questions.
5/4/22, 3:36 PM CHAI Q4 Newsletter + Intern Applications Due Soon
https://mailchi.mp/13c99f6887ba/chai-q4-newsletter -intern-applications-due-soon 7/7Visiting Scholars and Researchers
If you are interested in visiting CHAI to collaborate on research, please ﬁll
out this application .
Please contact chai-admin@berkeley .edu for any questions.
To see more, visit us at humancompatible.ai
Copyright © 2019 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
621D Stuardja Dai Hall. UC Berkeley . Berkeley , CA. 94720.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .

This email was sent to <<Email Address>>
why did I get this?     unsubscribe from this list     update subscription preferences
Center for Human-Compatible AI · 2121 Berkeley W ay · Ofﬁce #8029 · Berkeley , CA 94704 · USA


CHAI Newsletter #3 2019

5/4/22, 3:34 PM CHAI Newsletter + Internship/Operations Assistant Openings
https://mailchi.mp/f50d06cf0756/chai-newsletter -internshipoperations-assistant-openings 1/7Human Compatible is available now!
Stuart Russell's new book, Human Compatible: Artiﬁcial Intelligence and
the Problem of Control, is available on Amazon .
In the book, he explains why he has come to consider his own discipline
an existential threat to our species, and lays out how we can change
course before it's too late. The book explores the idea of intelligence in
humans and in machines, describes the beneﬁts we can expect (from
5/4/22, 3:34 PM CHAI Newsletter + Internship/Operations Assistant Openings
https://mailchi.mp/f50d06cf0756/chai-newsletter -internshipoperations-assistant-openings 2/7intelligent personal assistants to vastly accelerated scientiﬁc research),
and outlines the AI breakthroughs that still have to happen before we
reach superhuman AI. Professor Russell spells out the ways humans are
already ﬁnding to misuse AI, from lethal autonomous weapons to the
manipulation of opinions on a global scale. Finally , he explains why we
must ensure that we never lose control of machines more powerful than
we are. He suggests that we can rebuild AI on a new foundation,
according to which machines are designed to be inherently uncertain
about the human preferences they are required to satisfy .
For media inquiries, please contact Caroline Jeanmaire
(caroline.j@berkeley .edu).
“Human Compatible made me a convert to Russell's concerns with our
ability to control our upcoming creation—super-intelligent machines.
Unlike outside alarmists and futurists, Russell is a leading authority on
AI. His new book will educate the public about AI more than any book I
can think of, and is a delightful and uplifting read.” —Judea Pearl, T uring
Award-winner and author of The Book of Why
Organizational News
We Would Like to W elcome Martin Fukui and Caroline Jeanmaire to CHAI,
and Bid Farewell to Brandon Perry
We would like to congratulate Martin Fukui  and Caroline Jeanmaire  for joining
CHAI as our Assistant Director and Director of Strategic Research and
Partnerships respectively . Martin was previously brought on as an Operations
and Events Manager and was the interim Assistant Director between June and
September 2019. Caroline was previously an AI Policy Researcher and Project
Manager at The Future Society , a think-tank incubated at Harvard’ s Kennedy
School of Government.
We would also like to wish Brandon Perry , who was CHAI's Administrative
Assistant, the best of luck as he starts a new job in W ashington DC with the
Atlantic Council's new GeoT ech program.
Researcher News
5/4/22, 3:34 PM CHAI Newsletter + Internship/Operations Assistant Openings
https://mailchi.mp/f50d06cf0756/chai-newsletter -internshipoperations-assistant-openings 3/7
Congrats to our 2019 Class of Summer Interns
Congratulations to Dmitrii Krasheninnikov , Alex T urner, Shlomi Hod , and  Micah
Carrol l for completing their summer internship at CHAI! Each student had the
opportunity to work on a project with one of our PhD student researchers.
We’re also looking forward to David Lindler joining us for an internship in
October .
Vincent Corruble Added to CHAI Afﬁliates
Vincent Corruble  is an Associate Professor of Computer Science at Sorbonne
Université has joined CHAI as a faculty afﬁliate.
CHAI Researchers Publish Imitation Learning Library
Steven W ang, Adam Gleave, and Sam T oyer put together an extensible and
benchmarked implementation of imitation learning algorithms commonly used
at CHAI (Notably GAIL and AIRL) for public use. Y ou can visit the Github  here.
Siddharth Srivastava A warded NSF Grant on AI and the Future of W ork
Siddharth Srivastava, along with other faculty from Arizona State University ,
was awarded a grant as a part of their Convergence Accelerator  program. The
project focuses on safe, adaptive AI systems/robots that enable workers to
learn how to use them on the ﬂy . The central question behind their research is:
how can we train people to use adaptive AI systems, whose behavior and
functionality is expected to change from day to day? Their approach uses self-
explaining AI to enable on-the-ﬂy training. Y ou can read more about the project
here.
Blog Posts
Rohin Shah Professionalizes the Alignment Newsletter
CHAI PhD student Rohin Shah's  Alignment Newsletter  has grown from a
handful of volunteers to a team of people paid to summarize content.
The Alignment Newsletter , which was incubated at CHAI in 2018, has gone
from a small team with a few volunteers working for free to its current iteration.
New changes include:
Everyone helping out with the newsletter has the option of being paid
5/4/22, 3:34 PM CHAI Newsletter + Internship/Operations Assistant Openings
https://mailchi.mp/f50d06cf0756/chai-newsletter -internshipoperations-assistant-openings 4/7There are now eight active content summarizers and a formal application
process for new members.
Georg Arndt has taken over the publisher role from Rohin Shah.
The processes, norms and standards of the newsletter have been
standardized
Mark Nitzberg Publishes Article on W ired Magazine
CHAI’ s Executive Director Mark Nitzberg, along with Olaf Groth, published an
article in WIRED Magazine  that advocates for the creation of an “FDA for
algorithms.”
Rohin Shah posts Learning Biases and Rewards Simultaneously on
Alignment Forum
Rohin Shah published a short summary  of the CHAI paper “On the Feasibility of
Learning, Rather than Assuming, Human Biases for Reward Inference," along
with some discussion of its implications on the Alignment Forum.
Rohin Shah posts Clarifying Some Key Hypotheses on AI Alignment
Forum
Rohin Shah, along with Ben Cottier , published the blog post “Clarifying Some
Key Hypotheses in AI Alignment”  on the AI Alignment Forum. The post maps
out dif ferent key and controversial hypotheses of the AI Alignment problem and
how they relate to each other .

Papers
Siddharth Srivastava Publishes “Why Can't Y ou Do That, HAL? Explaining
Unsolvability of Planning T asks”
CHAI PI Siddharth Srivastava, along with his co-authors Sarath Sreedharan,
Rao Kambhampati, David Smith, published “Why Can’t Y ou Do That, HAL?
Explaining Unsolvability of Planning T asks”  in the 2019 International Joint
Conference on Artiﬁcial Intelligence (IJCAI) proceedings. The paper discusses
how, as anyone who has talked to a 3-year-old knows, explaining why
something can’t be done can be harder than explaining a solution to a
problem.
Thomas Gilbert Submits “The Passions and the Reward Functions: Rival
Views of AI Safety?” to F AT* 2020
5/4/22, 3:34 PM CHAI Newsletter + Internship/Operations Assistant Openings
https://mailchi.mp/f50d06cf0756/chai-newsletter -internshipoperations-assistant-openings 5/7Thomas Krendl Gilbert submitted “The Passions and the Reward Functions:
Rival V iews of AI Safety?” to the upcoming Fairness, Accountability , and
Transparency (F AT*) 2020 Conference.
The paper compares dif ferent approaches to AI Safety (e.g. CHAI, Partnership
on AI, V alue-Sensitive Design) with dif ferent schools of classical political
economy . It organizes these approaches into four distinct camps, summarizes
their key claims and disagreements, and outlines what each can learn from
each other with reference to foundational disputes about the status of free
markets with respect to human preferences and political sovereignty .
CHAI researchers’ paper “On the Utility of Learning about Humans for
Human-AI Coordination” accepted to NeurIPS 2019.
The paper authored by Micah Carroll, Rohin Shah, T om Grifﬁths, Pieter Abbeel,
and  Anca Dragan, along with two other researchers not afﬁliated with CHAI,
was accepted to NeurIPS 2019. An ArXiv link for the paper will be available
shortly .
The paper is about how deep reinforcement learning has been used to play
DotA and Starcraft, using methods like self-play and population-based training,
which create agents that are very good at coordinating with themselves. The
agents "expect" their partners to be similar to them and are unable to predict
what human partners would do. In competitive games, this is ﬁne: if the human
deviates from optimal play , even if you don't predict it you will still beat them.
However , in cooperative settings, this can be a big problem. This paper
demonstrates this with a simple environment that requires strong coordination
based on the popular game Overcooked. It shows that agents speciﬁcally
trained to play alongside humans perform much better than self-play or
population-based training when paired with humans, particularly by adapting to
suboptimal human gameplay .
Events
Weekly CHAI Seminars
Every W ednesday from 10am to 12pm and 4pm to 6pm we have our technical
and interdisciplinary seminars respectively in the Newton Room of Sutardja Dai
Hall (Room 730). If you are interested in attending, contact chai-
admin@berkeley .edu.
5/4/22, 3:34 PM CHAI Newsletter + Internship/Operations Assistant Openings
https://mailchi.mp/f50d06cf0756/chai-newsletter -internshipoperations-assistant-openings 6/7Vacancies
Internship
Application
The application will close on December 15th.
If you have any questions, please contact chai-admin@berkeley .edu
Operations Assistant
For the next 2-3 months, CHAI is seeking an assistant who can help the
CHAI staf f with administrative tasks.
You must be available during the following times:
Tuesdays - 1 1:15am to 12:30pm
Wednesdays - 9:15am to 10:30am
Fridays - 1 1:15am to 12:30pm
Compensation is $25/hour . You will be paid by BERI which is one of
CHAI’ s partner organizations.
To apply , please ﬁll out this form
Please contact chai-admin@berkeley .edu if you have any questions.
To see more, visit us at humancompatible.ai
Copyright © 2018 Center for Human-Compatible AI, All rights reserved.
Our mailing address is:
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .

5/4/22, 3:34 PM CHAI Newsletter + Internship/Operations Assistant Openings
https://mailchi.mp/f50d06cf0756/chai-newsletter -internshipoperations-assistant-openings 7/7This email was sent to <<Email Address>>
why did I get this?     unsubscribe from this list     update subscription preferences
Center for Human-Compatible AI · 2121 Berkeley W ay · Ofﬁce #8029 · Berkeley , CA 94704 · USA


CHAI Newsletter #2 2019

5/4/22, 2:43 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/1e421759290d/jq4ushvzjd 1/5H u ma n - C o mp a t i b l e Now Available for Pre-Order on
Amazon
Stuart Russell's new book,  Human Compatible: Artiﬁcial Intelligence and
the Problem of Control , is available for Pre-Order on Amazon . Stuart’ s new
book covers major topics in AI safety , value alignment, and autonomous
weapons. The book will be released October 8th.
Read More
5/4/22, 2:43 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/1e421759290d/jq4ushvzjd 2/5News
Professor Stuart Russell A warded AAAI Feigenbaum Prize and Carnegie
Fellowship
CHAI PI Stuart Russell was awarded the Association for the Advancement of
Artiﬁcial Intelligence’ s Feigenbaum Prize , a prize that is awarded “biennially to
recognize and encourage outstanding Artiﬁcial Intelligence research advances
that are made by using experimental methods of computer science.” Stuart was
also awarded the Carnegie Fellowship , which includes the $200,000 "brainy"
award for his work connecting AI and the social sciences and developing a
basic theory of AI safety .
Professor Joe Halpern Elected to the National Academy of Engineering
and Moore Scholar at Cal T ech
CHAI Professor Joe Halpern has received one of the highest honors in the ﬁeld
of engineering, joining the National Academy of Engineering  for his outstanding
contribution to “reasoning about knowledge, belief, and uncertainty and their
applications to distributed computing and multiagent systems.” He was also
recently chosen as a Moore Scholar  at Cal T ech's Division of the Humanities
and Social Sciences.
Former Intern Alex T urner W ins AI Alignment Prize
Alex, a former CHAI intern, was announced as one of the two winners of the AI
alignment prize for his paper on Penalizing Impact via Attainable Utility
Preservation . The Alignment Prize is a submission based competition for the
best AI safety paper presented.
CHAI at Effective Altruism Global (EAG) Conference
CHAI staf f and researchers attended  EAG in San Francisco  this weekend (June
21st - 23rd).
Thanks to our Generous Donors!
We recently created a method for CHAI to receive online donations , and we'd
like to thank everyone who has used this so far . We'd also like to give particular
thanks to Ben Delo and another (currently anonymous) donor for their
signiﬁcant contributions. Gifts like these provide vital support in helping us
achieve our mission, and we are delighted to have their support.
5/4/22, 2:43 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/1e421759290d/jq4ushvzjd 3/5Tired of waiting until our next newsletter to stay up to date? Check out
our Twitter  and subscribe to our RSS Feed !
Events
Twitter Chat on AI Ethics with Rosie Campbell
On Friday , June 28, 8am PST (1 1am EST , 3pm GMT) AI Ethics advocate Mia
Dand will host a ‘T witter Chat’ with CHAI Assistant Director Rosie Campbell.
Follow and use the hashtags #AIEthics #T witterChat to join in.
Vacancies
Postdoc
If you are a graduating PhD student, taking a postdoc at CHAI before moving
into industry or a faculty position can help you to develop an expertise in the
ﬁeld of AI safety and control, positioning you to become a leader in this
developing ﬁeld. Check out the listing here.

Blog Posts
Learning Preferences by Looking at the W orld by Rohin Shah and Dmitrii
Krasheninnikov (see also  this post  for elaboration by Rohin)
Test Cases for Impact Measures  by Daniel Filan
Value Learning Sequence  by Rohin Shah
Papers
Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Grifﬁths, Sanjit Seshia,
Pieter Abbeel, and Anca Dragan On the Utility of Learning about Humans for
5/4/22, 2:43 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/1e421759290d/jq4ushvzjd 4/5Human-AI Coordination

Adam Gleave, Michael Dennis, Neel Kant, Cody Wild, Sergey Levine, and
Stuart Russell Adversarial Policies: Attacking Deep Reinforcement Learning

Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan On the
Feasibility of Learning, Rather than Assuming, Human Biases for Reward
Inference

 Xinlei Pan, W eiyao W ang, Xiaoshuai Zhang, Bo Li, Jinfeng Y i, and Dawn Song
How Y ou Act T ells a Lot: Privacy-Leaking Attack on Deep Reinforcement
Learning

Zhuolin Y ang, Bo Li, Pin-Y u Chen, and Dawn Song Characterizing Audio
Adversarial Examples Using T emporal Dependency

Smitha Milli and Anca D. Dragan Literal or Pedagogic Human? Analyzing
Human Model Misspeciﬁcation in Objective Learning

Tom Gilbert and McKane Andrus  Towards a Just Theory of Measurement: A
Principled Social Measurement Assurance Program

Smitha Milli, John Miller , Anca Dragan, and Moritz Hardt  The Social Cost of
Strategic Classiﬁcation. Conference on Fairness, Accountability , and
Transparency  (FAT*), 2019
Smitha Milli, Ludwig Schmidt, Anca Dragan, and Moritz Hardt  Model
Reconstruction from Model Explanations. Conference on Fairness,
Accountability , and T ransparency  (FAT*), 2019
Thanh H. Nguyen, Arunesh Sinha, Y ongzhao W ang, and Michael P .
Wellman  Deception in Finitely Repeated Security Games

Shihui Li, Y i Wu, Xinyue Cui, Honghua Dong, Fei Fang and Stuart
Russell  Robust Multi-Agent Reinforcement Learning via Minimax Deep
Deterministic Policy Gradient

Anagha Kulkarni, Siddharth Srivastava, and Rao Kambhampati  A Uniﬁed
Framework for Planning in Adversarial and Cooperative Environments , AAAI
5/4/22, 2:43 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/1e421759290d/jq4ushvzjd 5/52019

Joe Halpern and Evan Piermont  Partial Unawareness , to appear AAAI 2019

Sander Beckers and Joe Halpern  Abstracting Causal Models , to appear AAAI
2019

Joe Halpern and Meir Friedenberg  Blameworthiness in Multi-Agent Settings , to
appear AAAI 2019

To see more, visit us at humancompatible.ai
Copyright © 2018 Center for Human-Compatible AI, All rights reserved.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .

This email was sent to <<Email Address>>
why did I get this?     unsubscribe from this list     update subscription preferences
Center for Human-Compatible AI · 2121 Berkeley W ay · Ofﬁce #8029 · Berkeley , CA 94704 · USA


CHAI Newsletter #1 2019

5/4/22, 2:42 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c0e765c32ede/wkjotgxfx8 1/5CHAI at NeurIPS 2018
CHAI graduate students Rohin Shah,
Adam Gleave, Daniel Filan, and Dylan
Hadeld-Menell, along with interns
Matthew Rahtz and Aaron Tucker,
attende d the NeurIPS 2018 conference,
where they presented papers, met with
researchers, and learned about work in
the rest of the eld.Aaron and Adam presented Inverse
reinforcement learning for video
games at the Deep RL workshop, while
Rohin presented The Implicit Preference
Information in an Initial State at
the Deep RL and Learning by
Instruction workshops.
News
Stuart Russell to Receive AAAI Feignenbaum Prize
The Association for the Advancement of Artiﬁcial Intelligence (AAAI) has
announced that CHAI PI Stuart Russell is the winner of the 2019 Feigenbaum
Prize, to be awarded at the AAAI conference in Honolulu in January "in
recognition of [his] high-impact contributions to the ﬁeld of artiﬁcial intelligence
through innovation and achievement in probabilistic knowledge representation,
reasoning, and learning, including its application to global seismic monitoring
for the Comprehensive Nuclear-T est-Ban T reaty ." The Feigenbaum Prize is
awarded biennially to recognize and encourage outstanding Artiﬁcial
Intelligence research advances that are made by using experimental methods
of computer science.
Stuart Russell Featured in MIT Interview
Stuart Russell was interviewed by Lex Fridman, a research scientist at MIT who
focuses on human-centered AI. The long-form interview  goes into great
detail on general intelligence, progress in AI, current and future AI risks, and
5/4/22, 2:42 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c0e765c32ede/wkjotgxfx8 2/5much more.
Anca Dragan Hosts W orkshop on Machine Learning for Intelligent
Transportation at NeurIPS 2018
Over 150 experts from all areas attended CHAI PI  Anca Dragan's workshop  to
discuss the challenges involved with machine learning and transportation, such
as coordination between both between vehicles and between vehicles and
humans, object tracking, and more.
Rosie Campbell Speaks About AI Safety and Neural Networks at San
Francisco and East Bay AI meetups
CHAI Assistant Director Rosie Campbell gave a talk at the San Francisco AI
Meetup to explain in basic terms what a neuron is, how they combine to form a
network, why convolutional neural networks have had so much attention, and
the challenges and opportunities of AI progress. See the slides  here and a
write-up of an earlier version of the talk here. Rosie also sat down for a chat
with tech blogger Mia Dand on " Keeping AI Safe and Beneﬁcial for Humanity ".
Their discussion covered topics such as CHAI's mission and approach, the
problem of AI alignment, and why you might or might not believe AI safety
research is important.
CHAI Inﬂuences V ox Article  The Case for T aking AI Seriously as a Threat to
Humanity
Stuart Russell and Rosie Campbell were quoted in the piece. It was featured in
Vox's Future Perfect  section, which focuses on covering high impact causes
such as climate change, AI safety , and animal welfare. The article covers
everything from the basics of what AI is to the arguments for why AI could pose
a danger to humanity .
Paper Highlights

Summarized by CHAI PhD researcher , Rohin Shah
A Learning and Masking Approach to Secure Learning , 9th Conference on
Decision and Game Theory for Security
Linh Nguyen, Sky W ang, and Arunesh Sinha
One way to view the problem of adversarial examples is that adversarial
attacks map "good" clean data points that are classiﬁed correctly into a nearby
5/4/22, 2:42 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c0e765c32ede/wkjotgxfx8 3/5"bad" space. So, in order to attack a model, we can use a neural net to  learn  a
transformation from good data points to bad ones. The neural net is
encouraged to keep the new data point similar to the original data point, but to
have a dif ferent classiﬁcation. Similarly , in order to defend a model, we can
learn a neural net transformation that maps bad data points to good ones. W e
continue to encourage similarity between the data points, but now we
encourage that the new data point have the correct classiﬁcation. Note that we
need to use some attack algorithm in order to generate the bad data points that
are used for training.
Learning Plannable Representations with Causal InfoGAN , NeurIPS 2018
Thanard Kurutach, A viv Tamar , Ge Y ang, Stuart Russell, and Pieter Abbeel
This paper shows how to create high-level plans, and applies it in the domain of
a robot manipulating a rope. Given a high-level plan showing the main steps in
rope manipulation, we can ﬁll in the detailed low-level actions using
reinforcement learning (RL). It would be useful if we could map the initial image
and goal image into a useful  representation  of the state of the robot and the
rope. In a good representation, we could generate a high level plan by moving
in a straight line from the representation of the initial image to the
representation of the goal image. The key contribution of the paper is
the Causal InfoGAN which can learn such a representation. Speciﬁcally , we ﬁnd
a representation such that a  generator  network can take in the representations
of two states, and generate the corresponding images. The generator is trained
to fool a  discriminator  network, which tries to tell whether a given pair of images
comes from the generator or from the true environment.
Meta-Learning MCMC Proposals , NeurIPS 2018
Tongzhou W ang, Y i Wu, David A. Moore, and Stuart J. Russell
Probabilistic programming of fers the potential to encode our domain knowledge
about a particular area into structured probabilistic models. However ,
algorithms that answer queries about these models can be very slow , even if
we are only looking for an approximate answer . MCMC algorithms are
approximate algorithms that propose  values for variables, which are then either
accepted  or rejected . They only work well if the proposer is able to suggest high
probability values, which allows for quick convergence. So, in practice
researchers use hand-tuned proposers with these algorithms. This paper
suggests that we could instead use a neural net as a proposer . The net is
trained to give good proposals on small sections of models (called motifs ), with
5/4/22, 2:42 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c0e765c32ede/wkjotgxfx8 4/5a range of possible parameter values that af fect what should be proposed.
Then, faced with a new model, the net is able to look at the motifs in the new
model and propose good values for those motifs.
Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-
Making , NeurIPS 2018
Stuart Russell, Andrew Critch, and Nishant Desai
As AI systems are deployed in the real world, they will have to navigate value
differences between dif ferent users. Regardless of our beliefs on how they
should trade of f between users, we at least want our AI systems to be Pareto
optimal -- if there is some way to help one user without harming any other user ,
it should do it. A classic result from social choice theory , Harsanyi's theorem,
states that a Pareto optimal agent must be optimizing a weighted linear
combination of the users' utility functions. (The weights determine how to trade
off between users.) However , this assumes that the users all have the same
beliefs . When users have dif ferent beliefs, we can get even better outcomes if
the users bet on their beliefs. If I'm sure that our robot will make a chocolate
chip cookie, while you're sure its going to be a brownie, rather than splitting
whatever the robot brings, we can agree that if it's a cookie I get it, whereas if
it's a brownie you get it, and this makes both of us better of f by our own beliefs.
This paper shows that a Pareto optimal agent helping users with dif ferent
beliefs must be optimizing a weighted linear combination of the user's utility
functions, where the weights are proportional to how accurate the user's beliefs
turn out to be. It's as if the users bet with each other about their beliefs, with the
stakes being the assistance of the agent.
Events
There are no public events scheduled at this time.
Vacancies
Machine Learning Engineer Specializing in AI Safety and Control
We're looking for a ML engineer to help with research projects by CHAI faculty
everywhere from UC Berkeley , to MIT , to Cambridge. Successful candidates
will be given a visiting research scholar position and have the opportunity to
help shape the CHAI research agenda.
5/4/22, 2:42 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c0e765c32ede/wkjotgxfx8 5/5Postdoc Specializing in AI Safety and Control
We're looking for someone with a PhD to work with Stuart Russell or one of the
other co-principle investigators at CHAI. Successful candidates do not need
prior research in AI safety , and will have a considerable degree of ﬂexibility in
exploring novel research projects.
For more information or to apply , please visit the CHAI website

To see more, visit us at humancompatible.ai
Copyright © 2018 Center for Human-Compatible AI, All rights reserved.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .

This email was sent to <<Email Address>>
why did I get this?     unsubscribe from this list     update subscription preferences
Center for Human-Compatible AI · 2121 Berkeley W ay · Ofﬁce #8029 · Berkeley , CA 94704 · USA


CHAI Newsletter 2018

5/4/22, 2:39 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c8b632a14601/center -for-human-compatible-ai-chai-newsletter 1/4Congratulations to our 2017 & 2018 Intern
Cohorts
"Interning at CHAI has been one of those rare experiences that makes you
question beliefs you didn't realize were there to be questioned... I'm coming
away with a very different view on what kind of research is valuable and
where the field of safety research currently stands" - Matthew Rahtz
Congratulations to our two previous cohorts of CHAI interns who spent a
semester not only learning more about AI safety , but also set themselves up for
amazing future opportunities.
Each intern had the opportunity to pursue their own research project, with one
of our PhD students as a mentor to give them advice, seeing them through from
the beginning of their project all the way to publication. Our interns also
received experience in handling large projects, deep RL, attending seminars
and lectures by some of the world's leading AI experts, and getting a taste for
what a PhD program would look like.
We would like to wish our best luck to our interns who are now moving on to
new projects. For example,  Beth Barnes  has accepted a position at DeepMind,
and Chris Cundy  joined Stanford's AI lab as a PhD student. Our other previous
interns include:  Dmitrii Krasheninnikov , Aaron David T ucker , Alex T urner ,
James Drain , Jordan Alexander , Matthew Rahtz , and Steven W ang.
If this experience sounds like something you or someone you know would be
interested in, then consider joining our team as an intern  and send an
application before our deadline on December 21st.
5/4/22, 2:39 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c8b632a14601/center -for-human-compatible-ai-chai-newsletter 2/4News
CHAI Co-Sponsored the talk  Guardians, Aliens, or Robot Overlords?
Security in the Age of Artiﬁcial Intelligence
Professor Edward Felten, a specialist in computer science, public policy, and
Deputy CT O of the Unites States in the Obama White House , gave a talk on the
security implications of articial intelligence. To learn more, you can read the
Institute of International Studies' page on the event and watch the lecture here.
Rosie Campbell, CHAI Assistant Director , has been named one of the
hundred women in AI ethics to follow on social media
Rosie was named by inﬂuential technology blogger Mia Dand in one of her
recent posts. She was also invited to do a ﬁreside chat about AI ethics by the
author of the article; details in the events section below . You can read about it
more in the article here.
Peter Abbeel and Anca Dragan presented a paper at IROS 2018
Two CHAI co-PIs attended the 2018 International Conference on Intelligent
Robots and Systems in Madrid, Spain to present their paper  Establishing
Appropriate T rust via Critical States . The paper discusses how humans can
develop reliable mental models of how robotic systems in certain situations
where actions need to be taken by showing humans what they
understand. Their paper can be accessed on ArXiv here.
Co-PI Anca Dragan attended the Conference on Robot Learning with PhD
student Jaime Fisac
The two presented a paper at the 2018 Conference on Robot Learning,
which took place October 29th-31st, 2018, in Zürich, Switzerland. Their paper is
titled  Learning under Misspeciﬁed Objective Spaces , is on trying to make robots
learn to correct their objectives based on human input in a way that the robot
considers if the humans correction applies to their space of hypotheses. The
paper can be accessed on ArXiv here.
A paper by Professor Stuart Russell,  PhD student Adam Gleave, and
intern Aaron T ucker has been accepted at the NIPS DeepRL workshop
Head PI Professor Stuart Russell, Adam Gleave, and Aaron T ucker worked
together to submit a paper to the NIPS DeepRL W orkshop, which will take
place December 2nd-8th, 2018 in Palais des Congrès de Montréal, Canada.
Their paper is titled Inverse reinforcement learning for video games, and is on
how an adversarial inverse reinforcement learning algorithm can be used in a
5/4/22, 2:39 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c8b632a14601/center -for-human-compatible-ai-chai-newsletter 3/4video game to ﬁnd the objective function for a behavior , as opposed to a
programmer trying to create an objective function that describes a behavior .
The paper can be accessed on ArXiv here.
Interested in staying on top of the latest in AI safety research? Make sure
to sign up for Rohin Shah's Alignment Newsletter .
Events

Join Assistant-Director Rosie
Campbell for a reside chat on
keeping AI safe and benecial for
humanity. The talk will be hosted by
Mia Dand and will take place in
Berkeley, CA on November 28th.
Click here to RSVP for the event.
Vacancies

Machine Learning Engineer Specializing in AI Safety and Control
We're looking for a ML engineer to help with research projects by CHAI faculty
everywhere from UC Berkeley , to MIT , to Cambridge. Successful candidates
will be given a visiting research scholar position and have the opportunity to
help shape the CHAI research agenda.
Postdoc Specializing in AI Safety and Control
We're looking for someone with a PhD to work with Stuart Russell or one of the
other co-principle investigators at CHAI. Successful candidates do not need
prior research in AI safety , and will have a considerable degree of ﬂexibility in
exploring novel research projects.
Research Internship at CHAI
We're of fering research internships at CHAI for undergrads and people with
technical experience wanting to transition into AI safety research. Experience in
a technical ﬁeld such as machine learning, computer science, statistics, or
mathematics is required. These internships are held over the summer , and the
5/4/22, 2:39 PM Center for Human-Compatible AI (CHAI) Newsletter
https://mailchi.mp/c8b632a14601/center -for-human-compatible-ai-chai-newsletter 4/4ﬁrst round of applications will be due on December 21st.
For more information or to apply , please visit the CHAI website
To see more, visit us at humancompatible.ai
Copyright © 2018 Center for Human-Compatible AI, All rights reserved.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .

This email was sent to <<Email Address>>
why did I get this?     unsubscribe from this list     update subscription preferences
Center for Human-Compatible AI · 2121 Berkeley W ay · Ofﬁce #8029 · Berkeley , CA 94704 · USA


CHAI Newsletter 2017

5/4/22, 2:38 PM CHAI Newsletter - Q3 2017
https://mailchi.mp/03333358c1 12/chai-newsletter -q3-2017 1/5
This is the inaugural issue of the Center for Human-Compatible AI’s quarterly newsletter. CHAI’s mission
is to develop the conceptual and technical foundations required to build provably beneﬁcial AI systems.
CHAI was inaugurated in fall 2016 and is funded by the Open Philanthropy Project, the Future of Life
Institute, the Center for Long-Term Cybersecurity, and the Leverhulme Trust. CHAI investigators include
faculty members at Berkeley, Cornell, and Michigan.
In this issue we’ll highlight key research from our faculty and students, round up recent media spotlights,
and share insights from our 2017 Annual Workshop.
Research highlights

Obedience, off-switches, and cooperative inverse
reinforcement learning
Last year, PhD student Dylan Hadﬁeld-Menell and his advisors formalized the
cooperative inverse reinforcement learning (CIRL) problem, in which a human
and robot are both rewarded according to the human’s reward function, but
the robot starts oﬀ not knowing what this reward function is. Since CIRL
makes it in both players’ best interests for the robot to learn how to act in a
way that fulﬁlls the human’s reward function, this setup leads to desirable
cooperative learning behaviors such as active teaching, active learning, and
communicative actions.
This year, meaningful progress has been made within the CIRL setup.
Should Robots be Obedient? (Milli et al., 2017) explores the tradeoﬀs of robots
being perfectly obedient vs. exhibiting independent judgment. The research
ﬁnds that when a human is imperfectly rational and a robot attempts to infer
the human’s underlying preferences rather than being perfectly obedient, the
robot performs its function better than it would by following the human’s
orders precisely. In these situations, having a well-speciﬁed model of the
features the human cares about or the human’s level of rationality are essential
for the robot to exhibit good judgment in strategically disobeying orders.
5/4/22, 2:38 PM CHAI Newsletter - Q3 2017
https://mailchi.mp/03333358c1 12/chai-newsletter -q3-2017 2/5In the example below (extreme for the purpose of illustration): the human
wants a slice of cake, but in a slip of the tongue, asks for a vial of poison
instead. In this case, obedience becomes harmful.
Milli et al., 2017
The Oﬀ-Switch Game (Hadﬁeld-Menell et al., 2017) presents meaningful
progress on the corrigibility problem. An intuitive desideratum for a corrigible
AI system is the ability for a human to turn the system oﬀ without the system
interfering. The oﬀ-switch game extends the CIRL setup: we have a robot R
with an oﬀ switch that it can disable, and a human H that can press the robot’s
oﬀ switch. The idea of a human-usable oﬀ-switch is diﬃcult to preserve, since
reward-maximizing agents will typically have strong incentives for self-
preservation. But in a CIRL environment where both H and R are maximizing a
shared reward function and only H knows what that reward function looks like,
R can only update its prior beliefs about the reward function by observing H’s
actions. If H wants to shut down R, and R doesn’t know the reward
parameters, then R will want this to happen, as a way to prevent errors arising
from its own lack of knowledge!
5/4/22, 2:38 PM CHAI Newsletter - Q3 2017
https://mailchi.mp/03333358c1 12/chai-newsletter -q3-2017 3/5
a = take an action, s = hit the oﬀ switch, w(a) = wait for the human to decide.
(Hadﬁeld-Menell et al., 2017)
The key insight is that the more uncertain R is about the reward function (i.e.
the more variance there is in R’s prior), the more corrigible it is -- it is more
amenable to letting H switch it oﬀ. The intuition is simple: if R believes that H is
a rational actor, and R is uncertain about the reward function, then R is more
likely to defer to H in the decision to hit the oﬀ switch. On the other hand, if R
believes it has precise knowledge of the reward function, then it knows exactly
what it should do. This might include proceeding with action a and preventing
the human from hitting the oﬀ switch. If we can have this failsafe in our back
pocket, we should be very happy. See the talk here.
Of course, a perfect “CIRL solver” robot can still fail to be corrigible—that is to
say, resist being shut down in a detrimental way—if its engineers have
misspeciﬁed the CIRL problem that it’s supposed to solve, for example, with
an incorrect model of the human’s rationality, or an incorrect model of how the
robot should update its beliefs about human preferences over time.  Thus,
while we believe CIRL presents a valuable lense through which to view the
problem of corrigibility, a great deal of valuable follow-up work remains to be
done in order to more comprehensively address the potential for engineering
errors to yield incorrigible systems.
Causality , intention, and responsibility
One objective of CHAI’s current research is to formalize established concepts
in social sciences that lack the logical and mathematical precision required for
incorporating these concepts into the design of intelligent agents.
5/4/22, 2:38 PM CHAI Newsletter - Q3 2017
https://mailchi.mp/03333358c1 12/chai-newsletter -q3-2017 4/5Causality and blame are evaluated daily in courts of law and have long been
debated in the context of thought experiments like the Trolley Problem -- but
as we develop autonomous agents in the real world that must necessarily
make moral judgments in their operation, we will require a more speciﬁc way
of deﬁning the extent to which an agent’s actions (causes) are responsible, or
blameworthy, for certain outcomes (eﬀects).
Joe Halpern’s recent research at Cornell focuses on this topic. When
formalizing a heuristic for autonomous vehicles like “avoid harming humans,”
how do we incorporate probabilities into tradeoﬀs (e.g. .01% chance of
harming a person vs. 99.9% chance of creating massive property damage)?
How do we determine whether an agent’s actions are indeed responsible for
certain beneﬁt or harm done, and the extent to which they are responsible
when many actors are involved? A prerequisite to implementing any moral
decision-making framework is having a way to deﬁne causality and quantify
responsibility. Halpern’s book Actual Causality (2016) and his recent talk go
into more detail on these topics.
2017 Annual Workshop recap
The Center for Human-Compatible Artiﬁcial Intelligence (CHAI) hosted its ﬁrst
annual workshop on Friday, May 5 and Saturday, May 6, 2017. Topics covered
include preference aggregation, single-principal value alignment, interactive
control, and minimum-viable world-takeover capabilities. See more details
here.
CHAI assorted links: news, articles, videos
Stuart Russell, Three principles for creating safer AI, TED talk, Vancouver,
April 25, 2017.
The future is now, debate organized by Wadham College, Oxford, in San
Francisco, June 5, 2017.
Stuart Russell, AI for Good, interview at the UN Global Summit on AI for
Good, June 9, 2017.
Stuart Russell, Provably Beneﬁcial Artiﬁcial Intelligence (starts at 55.25),
keynote at IJCAI 2017 (slides here).
As Tech Titans Bicker, Should You Worry About the Rise of AI?, by Ryan
Nakashima and Matt O'Brien, Top Tech News, July 30, 2017.
Stuart Russell, Do we want machines to make decisions on killing in
war?, interview on BBC Tomorrow's World, June 1, 2017.
Top US general warns against rogue killer robots, by John Bowden, The
Hill, July 18, 2017.
For inquiries, feedback, or requests for future content, please contact chai-
newsletter-editor@humancompatible.ai.
5/4/22, 2:38 PM CHAI Newsletter - Q3 2017
https://mailchi.mp/03333358c1 12/chai-newsletter -q3-2017 5/5
Copyright © 2017 Center for Human-Compatible AI, All rights reserved.
Want to change how you receive these emails?
You can update your preferences  or unsubscribe from this list .