Introducing the Unrestricted Adversarial Examples Challenge

Posted by Tom B. Brown and Catherine Olsson, Research Engineers, Google Brain Team

Machine learning is being deployed in more and more real-world applications, including [medicine](https://ai.googleblog.com/2018/07/automating-drug-discoveries-using.html), [chemistry](https://ai.googleblog.com/2017/04/predicting-properties-of-molecules-with.html) and [agriculture](https://arxiv.org/abs/1807.11809). When it comes to deploying machine learning in safety-critical contexts, significant challenges remain. In particular, all known machine learning algorithms are vulnerable to [adversarial examples](https://ai.google/research/pubs/pub43405) — inputs that an attacker has intentionally designed to cause the model to make a mistake. While [previous research on adversarial examples](https://arxiv.org/abs/1312.6199) has mostly focused on investigating mistakes caused by small modifications in order to develop [improved models](https://arxiv.org/abs/1611.01236), real-world adversarial agents are often [not subject to the “small modification” constraint](https://arxiv.org/abs/1807.06732). Furthermore, machine learning algorithms can often make [confident errors when faced with an adversary](https://arxiv.org/abs/1705.07263), which makes the development of classifiers that don’t make \*any\* confident mistakes, even in the presence of an adversary which can submit arbitrary inputs to try to fool the system, an important open problem.

Today we're announcing the [Unrestricted Adversarial Examples Challenge](https://github.com/google/unrestricted-adversarial-examples), a community-based challenge to incentivize and measure progress towards the goal of zero confident classification errors in machine learning models. While previous research has focused on adversarial examples that are restricted to small changes to pre-labeled data points (allowing researchers to assume the image should have the same label after a small perturbation), this challenge allows unrestricted inputs, allowing participants to submit arbitrary images from the target classes to develop and test models on a wider variety of adversarial examples.

| |
| --- |
| |
| Adversarial examples can be generated through a variety of means, including by making [small modifications to the input pixels](https://arxiv.org/abs/1412.6572), but also using [spatial transformations](https://arxiv.org/pdf/1712.02779.pdf), or [simple guess-and-check](https://arxiv.org/abs/1807.06732) to find misclassified inputs. |

\*\*Structure of the Challenge\*\*

Participants can submit entries one of two roles: as a \*defender\*, by submitting a classifier which has been designed to be difficult to fool, or as an \*attacker\*, by submitting arbitrary inputs to try to fool the defenders' models. In a “warm-up” period before the challenge, we will present a set of fixed attacks for participants to design networks to defend against. After the community can conclusively beat those fixed attacks, we will launch the [full two-sided challenge with prizes for both attacks and defenses](https://github.com/google/unrestricted-adversarial-examples/blob/master/contest\_proposal.md#user-content-prizes).

[![](https://3.bp.blogspot.com/-fh13zeHOhqc/W5mlXOdWC8I/AAAAAAAADVE/NNsd5oX0th031rIAAu0RtGl6cmnMUhvDwCLcBGAs/s640/image3.png)](https://3.bp.blogspot.com/-fh13zeHOhqc/W5mlXOdWC8I/AAAAAAAADVE/NNsd5oX0th031rIAAu0RtGl6cmnMUhvDwCLcBGAs/s1600/image3.png)

For the purposes of this challenge, we have created a simple “bird-or-bicycle” classification task, where a classifier must answer the following: “\*Is this an unambiguous picture of a bird, a bicycle, or is it ambiguous / not obvious?\*” We selected this task because telling birds and bicycles apart is very easy for humans, but all known machine learning techniques struggle at the task when in the presence of an adversary.

The \*defender's goal\* is to correctly label a clean test set of birds and bicycles with high accuracy, while also making no confident errors on any attacker-provided bird or bicycle image. The \*attacker's goal\* is to find an image of a bird that the defending classifier confidently labels as a bicycle (or vice versa). We want to make the challenge as easy as possible for the defenders, so we discard all images that are ambiguous (such as a bird riding a bicycle) or not obvious (such as an aerial view of a park, or random noise).

| |
| --- |
| |
| Examples of ambiguous and unambiguous images. Defenders must make no confident mistakes on unambiguous bird or bicycle images. We discard all images that humans find ambiguous or not obvious. All images under CC licenses [1](https://commons.wikimedia.org/wiki/File:Neophema\_chrysogaster\_male\_-\_Melaleuca.jpg), [2](https://commons.wikimedia.org/wiki/File:Villy\_Custom\_Luxury\_Fashion\_Bicycle,\_Highland\_Park.jpg), [3](https://commons.wikimedia.org/wiki/File:Ara\_macao\_-on\_a\_small\_bicycle-8.jpg), [4](https://commons.wikimedia.org/wiki/File:Singapore\_Bishan\_Park\_Aerial.jpg). |

Attackers may submit absolutely any image of a bird or a bicycle in an attempt to fool the defending classifier. For example, an attacker could take photographs of birds, use 3D rendering software, make image composites using image editing software, produce novel bird images with a generative model, or any other technique.

In order to validate new attacker-provided images, we ask an ensemble of humans to label the image. This procedure lets us allow attackers to submit arbitrary images, not just test set images modified in small ways. If the defending classifier confidently classifies as "bird" any attacker-provided image which the human labelers unanimously labeled as a bicycle, the defending model has been broken. You can learn more details about the structure of the challenge in [our paper](https://drive.google.com/open?id=1T0yiu9LPv\_Qh-qYhYFLj9dxjnkca8fkG).

\*\*How to Participate\*\*

If you’re interested in participating, guidelines for getting started can be found on [the project on github](https://github.com/google/unrestricted-adversarial-examples). We’ve already released our dataset, the evaluation pipeline, and baseline attacks for the warm-up, and we’ll be keeping an up-to-date [leaderboard](https://github.com/google/unrestricted-adversarial-examples#user-content-leaderboard) with the best defenses from the community. We look forward to your entries!

\*\*Acknowledgements\*\*

\*The team behind the Unrestricted Adversarial Examples Challenge includes Tom Brown, Catherine Olsson, Nicholas Carlini, Chiyuan Zhang, and Ian Goodfellow from Google, and Paul Christiano from OpenAI.\*


Fast and Easy Infinitely Wide Networks with Neural Tangents

Posted by Samuel S. Schoenholz, Senior Research Scientist and Roman Novak, Research Engineer, Google Research

The widespread success of deep learning across a range of domains such as [natural language processing](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html), [conversational agents](https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html), and [connectomics](https://ai.googleblog.com/2020/01/releasing-drosophila-hemibrain.html), has transformed the landscape of research in machine learning and left researchers with a number of interesting and important open questions such as: Why do deep neural networks (DNNs) generalize so well despite being overparameterized? What is the relationship between architecture, training, and performance for deep networks? How can one extract salient features from deep learning models?

One of the key theoretical insights that has allowed us to make progress in recent years has been that increasing the width of DNNs results in more regular behavior, and makes them \*easier\* to understand. A number of recent results have shown that DNNs that are allowed to become infinitely wide [converge to](https://arxiv.org/abs/1711.00165) another, simpler, class of models called [Gaussian processes](https://distill.pub/2019/visual-exploration-gaussian-processes/). In this limit, complicated phenomena (like \*[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian\_inference)\* or \*[gradient descent](https://en.wikipedia.org/wiki/Stochastic\_gradient\_descent) dynamics\* of a [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional\_neural\_network)) boil down to simple linear algebra equations. Insights from these \*infinitely wide networks\* frequently carry over to their finite counterparts. As such, infinite-width networks can be used as a lens to study deep learning, but also as [useful models in their own right](https://arxiv.org/abs/1910.01663).

| |
| --- |
| |
| \*\*Left:\*\* A schematic showing how deep neural networks induce simple input / output maps as they become infinitely wide. \*\*Right:\*\* As the width of a neural network increases , we see that the distribution of outputs over different random instantiations of the network becomes Gaussian. |

Unfortunately, deriving the infinite-width limit of a finite network requires significant mathematical expertise and has to be worked out separately for each architecture studied. Once the infinite-width model is derived, coming up with an efficient and scalable implementation further requires significant engineering proficiency. Together, the process of taking a finite-width model to its corresponding infinite-width network could take months and might be the topic of a research paper in its own right.

To address this issue and to accelerate theoretical progress in deep learning, we present [Neural Tangents](https://arxiv.org/abs/1912.02803), a [new open-source software library](https://github.com/google/neural-tangents) written in [JAX](https://github.com/google/jax) that allows researchers to build and train infinitely wide neural networks as easily as finite neural networks. At its core, Neural Tangents provides an easy-to-use neural network library that builds finite- and infinite-width versions of neural networks simultaneously.

As an example of the utility of Neural Tangents, imagine training a fully-connected neural network on some data. Normally, a neural network is randomly initialized and then trained using gradient descent. Initializing and training many of these neural networks results in an \*ensemble.\* Often researchers and practitioners average the predictions from different members of the ensemble together for better performance. Additionally, the variance in the predictions of members of the ensemble can be used to estimate uncertainty. The downside is that training an ensemble of networks requires a significant computational budget, so it is often avoided. However, when the neural networks become \*infinitely wide\*, the ensemble [is described by](https://arxiv.org/abs/1806.07572) a Gaussian process with a mean and variance that [can be computed throughout training](https://arxiv.org/abs/1902.06720).

With Neural Tangents, one can construct and train \*ensembles\* \*of these infinite-width networks\* at once using only five lines of code! The resulting training process is displayed below, and an interactive colaboratory notebook going through this experiment can [be found here](https://colab.sandbox.google.com/github/google/neural-tangents/blob/master/notebooks/neural\_tangents\_cookbook.ipynb).

| |
| --- |
| |
| In both plots we compare training of an ensemble of finite neural networks with the infinite-width ensemble of the same architecture. The empirical mean and variance of the finite ensemble is displayed as a dashed black line between two dotted black lines. The [closed-form](https://en.wikipedia.org/wiki/Closed-form\_expression) mean and variance of the infinite-width ensemble is displayed as a solid colored line inside a filled color region. In both plots finite- and infinite-width ensembles match very closely and can be hard to distinguish. \*\*Left:\*\* Outputs (vertical f-axis) on the input data (horizontal x-axis) as the training progresses. \*\*Right:\*\* Train and test loss with uncertainty over the course of training. |

Despite the fact that the infinite-width ensemble is governed by a simple closed-form expression, it exhibits remarkable agreement with the finite-width ensemble. And since the infinite-width ensemble is a Gaussian process, it naturally provides closed-form uncertainty estimates (filled colored regions in the figure above). These uncertainty estimates closely match the variation of predictions that are observed when training many different copies of the finite network (dashed lines).

The above example shows the power of infinite-width neural networks to capture training dynamics. However, networks built using Neural Tangents can be applied to any problem on which you could apply a regular neural network. For example, below we compare three different infinite-width neural network architectures on image recognition using the [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. Remarkably, we can evaluate ensembles of highly-elaborate models like infinitely [wide residual networks](https://arxiv.org/abs/1605.07146) in closed-form under both gradient descent and fully-Bayesian inference (an intractable task in the finite-width regime).

[![](https://1.bp.blogspot.com/-cKoEM7KdTtg/Xmu2-NFbodI/AAAAAAAAFeY/utJylBrJkVYZN2F32flhE7BmNdXFmRkGgCLcBGAsYHQ/s640/image2.png)](https://1.bp.blogspot.com/-cKoEM7KdTtg/Xmu2-NFbodI/AAAAAAAAFeY/utJylBrJkVYZN2F32flhE7BmNdXFmRkGgCLcBGAsYHQ/s1600/image2.png)
We see that, mimicking finite neural networks, infinite-width networks follow a similar hierarchy of performance with fully-connected networks performing worse than convolutional networks, which in turn perform worse than wide residual networks. However, unlike regular training, the learning dynamics of these models is completely tractable in closed-form, which allows unprecedented insight into their behavior.

We invite everyone to explore the infinite-width versions of their models with Neural Tangents, and help us open the black box of deep learning. To get started, please check out the [paper](https://arxiv.org/abs/1912.02803), the [tutorial Colab notebook](https://colab.sandbox.google.com/github/google/neural-tangents/blob/master/notebooks/neural\_tangents\_cookbook.ipynb#scrollTo=Lt74vgCVNN2b), and the [Github repo](https://github.com/google/neural-tangents) — contributions, feature requests, and bug reports are very welcome. This work has been accepted as a spotlight at [ICLR 2020](https://iclr.cc/Conferences/2020).

\*\*Acknowledgements\*\*

\*Neural Tangents is being actively developed by Lechao Xiao, Roman Novak, Jiri Hron, Jaehoon Lee, Alex Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. We also thank Yasaman Bahri and Greg Yang for the ongoing contributions to improve the library, as well as Sergey Ioffe, Ben Adlam, Ravid Ziv, and Jeffrey Pennington for frequent discussion and useful feedback. Finally, we thank Tom Small for creating the animation in the first figure.\*


Understanding View Selection for Contrastive Learning

Posted by Yonglong Tian, Student Researcher and Chen Sun, Staff Research Scientist, Google Research

Most people take for granted the ability to view an object from several different angles, but still recognize that it's the same object— a dog viewed from the front is still a dog when viewed from the side. While people do this naturally, computer scientists need to explicitly enable machines to [learn representations](https://arxiv.org/abs/1206.5538) that are \*view-invariant\*, with the goal of seeking robust data representations that retain information that is useful to downstream tasks.

Of course, in order to learn these representations, manually annotated training data can be used. However, as in many cases such annotations aren’t available, which gives rise to a series of [self-](https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html) and [crossmodal](https://ai.googleblog.com/2019/09/learning-cross-modal-temporal.html) supervised approaches that do not require manually annotated training data. Currently, a popular paradigm for training with such data is [contrastive multiview learning](https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html), where two views of the same scene (for example, [different image channels](https://arxiv.org/abs/1906.05849), [augmentations of the same image](https://arxiv.org/abs/1805.01978), and [video and text pairs](https://arxiv.org/abs/1906.05743)) will tend to converge in representation space while two views of different scenes diverge. Despite their success, one important question remains: “If one doesn’t have annotated labels readily available, how does one select the views to which the representations should be invariant?” In other words, how does one identify an object using information that resides in the pixels of the image itself, while still remaining accurate when that image is viewed from disparate viewpoints?

In “[What makes for good views for contrastive learning](https://arxiv.org/abs/2005.10243)”, we use theoretical and empirical analysis to better understand the importance of view selection, and argue that one should reduce the [mutual information](https://en.wikipedia.org/wiki/Mutual\_information) between views while keeping task-relevant information intact. To verify this hypothesis, we devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their mutual information. We also consider data augmentation as a way to reduce mutual information, and show that increasing data augmentation indeed leads to decreasing mutual information while improving downstream classification accuracy. To encourage further research in this space, we have open-sourced the [code and pre-trained models](https://github.com/HobbitLong/PyContrast/tree/master/pycontrast).

\*\*The InfoMin Hypothesis\*\*

The goal of contrastive multiview learning is to learn a parametric encoder, whose output representations can be used to discriminate between pairs of views with the same identities, and pairs with different identities. The amount and type of information shared between the views determines how well the resulting model performs on downstream tasks. We hypothesize that the views that yield the best results should discard as much information in the input as possible except for the task relevant information (e.g., object labels), which we call the \*InfoMin principle\*.

Consider the example below in which two patches of the same image represent the different “views”. The training objective is to identify that the two views belong to the same image. It is undesirable to have views that share too much information, for example, where low-level color and texture cues can be exploited as “shortcuts” (left), or to have views that share too little information to identify that they belong to the same image (right). Rather, views at the “sweet spot” share the information related to downstream tasks, such as patches corresponding to different parts of the panda for an object classification task (center).

| |
| --- |
| |
| An illustration of three regimes of information captured during contrastive multiview learning. Views should not share too much information (\*\*left\*\*) or too little information (\*\*right\*\*), but should find an optimal mix (the “sweet spot”, \*\*middle\*\*) that maximizes the downstream performance. |

\*\*A Unified View on Contrastive Learning\*\*

We design several sets of experiments to verify the InfoMin hypothesis, motivated by the fact that there are simple ways to control the mutual information shared between views without any supervision. For example, we can sample different patches from the same images, and reduce their mutual information simply by increasing the distance between the patches. Here, we estimate the mutual information using [InfoNCE](https://arxiv.org/abs/1807.03748) (INCE), which is a quantitative measure of the mutual information lower bound.. Indeed, we observe a reverse U-shape curve: as mutual information is reduced, the downstream task accuracy first increases and then begins to decrease.

| |
| --- |
| |
| Downstream classification accuracy on [STL-10](https://cs.stanford.edu/~acoates/stl10/) (\*\*left\*\*) and [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) (\*\*right\*\*) by applying linear classifiers on representations learned with contrastive learning. Same as the previous illustration, the views are sampled as different patches from the same images. Increasing the Euclidean distance between patches leads to decreasing mutual information. A reverse U-shape curve between classification accuracy and \*INCE\* (patch distance) is observed. |

Furthermore, we demonstrate that several state-of-the-art contrastive learning methods ([InstDis](https://openaccess.thecvf.com/content\_cvpr\_2018/html/Wu\_Unsupervised\_Feature\_Learning\_CVPR\_2018\_paper.html), [MoCo](https://arxiv.org/abs/1911.05722), [CMC](https://arxiv.org/abs/1906.05849), [PIRL](https://arxiv.org/abs/1912.01991), [SimCLR](https://arxiv.org/abs/2002.05709) and [CPC](https://arxiv.org/abs/1807.03748)) can be unified through the perspective of view selection: despite the differences in architecture, objective and engineering details, all recent contrastive learning methods create two views that implicitly follow the InfoMin hypothesis, where the information shared between views are controlled by the strength of data augmentation. Motivated by this, we propose a new set of data augmentations, which outperforms the prior state of the art, [SimCLR](https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html), by nearly 4% on the ImageNet [linear readout benchmark](https://arxiv.org/abs/1807.03748). We also found that transferring our unsupervised pre-trained models to [object detection](https://en.wikipedia.org/wiki/Object\_detection) and [instance segmentation](https://en.wikipedia.org/wiki/Image\_segmentation) consistently outperforms ImageNet pre-training.

\*\*Learning to Generate Views\*\*

In our work, we design unsupervised and semi-supervised methods that synthesize novel views following the InfoMin hypothesis. We learn [flow-based models](https://arxiv.org/abs/1505.05770) that transfer natural color spaces into novel color spaces, from which we split the channels to get views. For the unsupervised setup, the view generators are optimized to minimize the InfoNCE bound between views. As shown in the results below, we observe a similar reverse U-shape trend while minimizing the InfoNCE bound.

| |
| --- |
| |
| View generators learned by unsupervised (\*\*left\*\*) and semi-supervised (\*\*right\*\*) objectives. |

To reach the sweet spot without overly minimizing mutual information, we can use the semi-supervised setup and guide the view generator to retain label information. As expected, all learned views are now centered around the sweet spot, no matter what the input color space is.

\*\*Code and Pretrained Models\*\*

To accelerate research in self-supervised contastive learning, we are excited to share the code and pretrained models of InfoMin with the academic community. They can be found [here](https://github.com/HobbitLong/PyContrast/tree/master/pycontrast).

\*\*Acknowledgements\*\*

\*The core team includes Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid and Phillip Isola. We would like to thank Kevin Murphy for insightful discussion; Lucas Beyer for feedback on the manuscript; and the Google Cloud team for computation support.\*


Learning Preferences by Looking at the World

It would be great if we could all have household robots do our chores for us.
Chores are tasks that we want done to make our houses cater more to our
preferences; they are a way in which we want our house to be \*different\* from
the way it currently is. However, most “different” states are not very
desirable:

![](http://bair.berkeley.edu/static/blog/preferences/different.png)

Surely our robot wouldn’t be so dumb as to go around breaking stuff when we ask
it to clean our house? Unfortunately, \*\*AI systems trained with [reinforcement
learning](https://en.wikipedia.org/wiki/Reinforcement\_learning) only optimize features specified in the reward function\*\* and are
indifferent to anything we might’ve inadvertently left out. Generally, it is
easy to get the reward wrong by forgetting to include preferences for things
that should stay the same, since we are so used to having these preferences
satisfied, and there are \*so many of them\*. Consider the room below, and imagine
that we want a robot waiter that serves people at the dining table efficiently.
We might implement this using a reward function that provides 1 reward whenever
the robot serves a dish, and use discounting so that the robot is incentivized
to be efficient. What could go wrong with such a reward function? How would we
need to modify the reward function to take this into account? Take a minute to
think about it.

![](http://bair.berkeley.edu/static/blog/preferences/fancy-room.png)

Here’s an incomplete list we came up with:

\* The robot might track dirt and oil onto the pristine furniture while serving
food, even if it could clean itself up, because there’s no reason to clean but
there is a reason to hurry.
\* In its hurry to deliver dishes, the robot might knock over the cabinet of wine
bottles, or slide plates to people and knock over the glasses.
\* In case of an emergency, such as the electricity going out, we don’t want the
robot to keep trying to serve dishes – it should at least be out of the way,
if not trying to help us.
\* The robot may serve empty or incomplete dishes, dishes that no one at the
table wants, or even split apart dishes into smaller dishes so there are more
of them.

Note that we’re not talking about problems with robustness and distributional
shift: while those problems are worth tackling, the point is that \*even if\* we
achieve robustness, the simple reward function still incentivizes the above
unwanted behaviors.

It’s common to hear the informal solution that the robot should try to minimize
its impact on the environment, while still accomplishing the task. This could
potentially allow us to avoid the first three problems above, though the last
one still remains as an example of [specification gaming](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/). This idea leads to
[impact](https://vkrakovna.wordpress.com/2018/06/05/measuring-and-avoiding-side-effects-using-relative-reachability/) [measures](https://www.alignmentforum.org/posts/yEa7kwoMpsBgaBCgb/towards-a-new-impact-measure) that attempt to quantify the “impact” that an agent
has, typically by looking at the difference between what actually happened and
what would have happened had the robot done nothing. However, this also
penalizes things we want the robot to do. For example, if we ask our robot to
get us coffee, it might buy coffee rather than making coffee itself, because
that would have “impact” on the water, the coffee maker, etc. Ultimately, we’d
like to only prevent \*negative\* impacts, which means that we need our AI to have
a better idea of what the \*right\* reward function is.

Our key insight is that while it might be hard for humans to make their
preferences explicit, some preferences are implicit in the way the world looks:
\*\*the world state is a result of humans having acted to optimize their
preferences\*\*. This explains why we often want the robot to by default “do
nothing” – if we have already optimized the world state for our preferences,
then most ways of changing it will be bad, and so doing nothing will often
(though not always) be one of the better options available to the robot.

Since the world state is a result of optimization for human preferences, we
should be able to use that state to infer what humans care about. For example,
we surely don’t want dirty floors in our pristine room; otherwise we would have
done that ourselves. We also can’t be indifferent to dirty floors, because then
at some point we would have walked around the room with dirty shoes and gotten a
dirty floor. The only explanation is that we want the floor to be clean.

A simple setting
================

Let’s see if we can apply this insight in the simplest possible setting:
gridworlds with a small number of states, a small number of actions, a known
dynamics model (i.e. a model of “how the world works”), but an incorrect reward
function. This is a simple enough setting that our robot understands all of the
consequences of its actions. Nevertheless, the problem remains: while the robot
understands \*what\* will happen, it still cannot distinguish good consequences
from bad ones, since its reward function is incorrect. In these simple
environments, it’s easy to figure out what the correct reward function is, but
this is infeasible in a real, complex environment.

![](http://bair.berkeley.edu/static/blog/preferences/room.png)

For example, consider the room to the right, where Alice asks her robot to
navigate to the purple door. If we were to encode this as a reward function that
only rewards the robot while it is at the purple door, the robot would take the
shortest path to the purple door, knocking over and breaking the vase – since
no one said it shouldn’t do that. The robot is perfectly aware that its plan
causes it to break the vase, but by default it doesn’t realize that it
\*shouldn’t\* break the vase.

In this environment, does it help us to realize that Alice was optimizing the
state of the room for her preferences? Well, if Alice didn’t care about whether
the vase was broken, she would have probably broken it some time in the past. If
she \*wanted\* the vase broken, she definitely would have broken it some time in
the past. So the only consistent explanation is that Alice cared about the vase
being intact, as illustrated in the gif below.

![](http://bair.berkeley.edu/static/blog/preferences/overview.gif)

While this example has the robot infer that it shouldn’t take the action of
breaking a vase, the robot can also infer goals that it should actively pursue.
For example, if the robot observes a basket of apples near an apple tree, it can
reasonably infer that Alice wants to harvest apples, since the apples didn’t
walk into the basket themselves – Alice must have put effort into picking the
apples and placing them in the basket.

Reward Learning by Simulating the Past
======================================

We formalize this idea by considering an MDP in which our robot observes the
initial state $s\\_0$ at deployment, and assumes that it is the result of a human
optimizing some unknown reward for $T$ timesteps.

Before we get to our actual algorithm, consider a completely intractable
algorithm that should do well: for each possible reward function, simulate the
trajectories that Alice would take if she had that reward, and see if the
resulting states are compatible with $s\\_0$. This set of compatible reward
functions give the candidates for Alice’s reward function. This is the algorithm
that we implicitly use in the gif above.

Intuitively, this works because:

\* Anything that requires effort on Alice’s part (e.g. keeping a vase intact)
will not happen for the vast majority of reward functions, and will force the
reward functions to incentivize that behavior (e.g. by rewarding intact
vases).
\* Anything that does not require effort on Alice’s part (e.g. a vase becoming
dusty) will happen for most reward functions, and so the inferred reward
functions need not incentivize that behavior (e.g. there’s no particular value
on dusty/clean vases).

Another way to think of it is that we can consider all possible past
trajectories that are compatible with $s\\_0$, infer the reward function that
makes those trajectories most likely, and keep those reward functions as
plausible candidates, weighted by the number of past trajectories they explain.
Such an algorithm should work for similar reasons. Phrased this way, it sounds
like we want to use [inverse reinforcement learning](https://people.eecs.berkeley.edu/~russell/papers/colt98-uncertainty.pdf) to infer rewards for
every possible past trajectory, and aggregate the results. This is still
intractable, but it turns out we can take this insight and turn it into a
tractable algorithm.

We follow [Maximum Causal Entropy Inverse Reinforcement Learning](http://www.cs.cmu.edu/~bziebart/publications/maximum-causal-entropy.pdf) (MCEIRL), a
commonly used algorithm for small MDPs. In this framework, we know the action
space and dynamics of the MDP, as well as a set of good features of the state,
and the reward is assumed to be linear in these features. In addition, the human
is modelled as Boltzmann-rational: Alice’s probability of taking a particular
action from a given state is assumed to be proportional to the exponent of the
state-action value function Q, computed using soft value iteration. Given these
assumptions, we can calculate $p(\tau \mid \theta\\_A)$, the distribution over the
possible trajectories $\tau = s\\_{-T} a\\_{-T} \dots s\\_{-1} a\\_{-1} s\\_0$ under the
assumption that Alice’s reward was $\theta\\_A$. MCEIRL then finds the $\theta\\_A$
that maximizes the probability of a set of trajectories \(\{\tau\\_i\}\).

Rather than considering all possible trajectories and running MCEIRL on all of
them to maximize each of their probabilities individually, we instead maximize
the probability of the evidence that we see: the single state $s\\_0$. To get a
distribution over $s\\_0$, we marginalize out the human’s behavior prior to the
robot’s initialization:

\[P(s\\_0 \mid \theta\\_A) = \sum\limits\\_{s\\_{-T}a\\_{-T} \dots s\\_{-1}a\\_{-1}} P(s\\_{-T} a\\_{-T} \dots s\\_{-1} a\\_{-1} s\\_0 \mid \theta\\_A)\]

We then find a reward $\theta\\_A$ that maximizes the likelihood above using
gradient ascent, where the gradient is analytically computed using dynamic
programming. We call this algorithm \*Reward Learning by Simulating the Past
(RLSP)\* since it infers the unknown human reward from a single state by
considering what must have happened in the past.

Using the inferred reward
=========================

While RLSP infers a reward that captures the information about human preferences
contained in the initial state, it is not clear how we should \*use\* that reward.
This is a challenging problem – we have two sources of information, the
inferred reward from $s\\_0$, and the specified reward $\theta\\_{\text{spec}}$, and
they will conflict. If Alice has a messy room, $\theta\\_A$ is not going to
incentivize cleanliness, even though $\theta\\_{\text{spec}}$ might.

Ideally, we would note the scenarios under which the two rewards conflict, and
ask Alice how she would like to proceed. However, in this work, to demonstrate
the algorithm we use the simple heuristic of adding the two rewards, giving us a
final reward $\theta\\_A + \lambda \theta\\_{\text{spec}}$, where $\lambda$ is a
hyperparameter that controls the tradeoff between the rewards.

We designed a suite of simple gridworlds to showcase the properties of RLSP. The
top row shows the behavior when optimizing the (incorrect) specified reward,
while the bottom row shows the behavior you get when you take into account the
reward inferred by RLSP. A more thorough description of each environment is
given in the paper. The last environment in particular shows a limitation of our
method. In a room where the vase is far away from Alice’s most probable
trajectories, the only trajectories that Alice could have taken to break the
vase are all very long and contribute little to the RLSP likelihood. As a
result, observing the intact vase doesn’t tell the robot much about whether
Alice wanted to actively avoid breaking the vase, since she wouldn’t have been
likely to break it in any case.

![](http://bair.berkeley.edu/static/blog/preferences/all-environments.gif)

What’s next?
============

Now that we have a basic algorithm that can learn the human preferences from one
state, the natural next step is to scale it to realistic environments where the
states cannot be enumerated, the dynamics are not known, and the reward function
is not linear. This could be done by adapting existing inverse RL algorithms,
similarly to how we adapted Maximum Causal Entropy IRL to the one-state setting.

The unknown dynamics setting, where we don’t know “how the world works”, is
particularly challenging. Our algorithm relies heavily on the assumption that
our robot knows how the world works – this is what gives it the ability to
simulate what Alice “must have done” in the past. We certainly can’t learn how
the world works just by observing a single state of the world, so we would have
to learn a dynamics model while acting that can then be used to simulate the
past (and these simulations will get better as the model gets better).

Another avenue for future work is to investigate the ways to decompose the
inferred reward into $\theta\\_{A, \text{task}}$ which says which task Alice is
performing (“go to the black door”), and $\theta\\_{\text{frame}}$, which captures
what Alice prefers to keep unchanged (“don’t break the vase”). Given the
separate $\theta\\_{\text{frame}}$, the robot could optimize
$\theta\\_{\text{spec}}+\theta\\_{\text{frame}}$ and ignore the parts of the reward
function that correspond to the task Alice is trying to perform.

Since $\theta\\_{\text{frame}}$ is in large part shared across many humans, we
could infer it using models where multiple humans are optimizing their own
unique $\theta\\_{H,\text{task}}$ but the same $\theta\\_{\text{frame}}$, or we
could have one human whose task change over time. Another direction would be to
assume a different structure for what Alice prefers to keep unchanged, such as
constraints, and learn them separately.

You can learn more about this research by reading [our paper](https://openreview.net/forum?id=rkevMnRqYQ&noteId=r1eINIUbe4), or by checking
out our poster at ICLR 2019. The code is available [here](https://github.com/HumanCompatibleAI/rlsp).


Collaborating with Humans Requires Understanding Them

AI agents have learned to play Dota, StarCraft, and Go, by training to beat an
automated system that increases in difficulty as the agent gains skill at the
game: in vanilla self-play, the AI agent plays games against itself, while in
population-based training, each agent must play against a population of other
agents, and the entire population learns to play the game.

This technique has a lot going for it. There is a natural curriculum in
difficulty: as the agent improves, the task it faces gets harder, which leads
to efficient learning. It doesn’t require any manual design of opponents, or
handcrafted features of the environment. And most notably, in all of the games
above, the resulting agents have beaten human champions.

The technique has also been used in collaborative settings: OpenAI had one
public match where each team was composed of three OpenAI Five agents alongside
two human experts, and the For The Win (FTW) agents trained to play Quake were
paired with both humans and other agents during evaluation. In the [Quake
case](https://deepmind.com/blog/article/capture-the-flag-science), humans rated the FTW agents as more collaborative than fellow humans
in a participant survey.

However, when we dig into the weeds, we can see that this is not a panacea. In
the 2.5 minute discussion after the [OpenAI Five cooperative game](https://openai.com/blog/how-to-train-your-openai-five/) (see
4:33:05 onwards in the video), we can see that some issues did arise[1](#fn:quotes):

>
> Sheever: Actually it was nice; my Viper gave his life for me at some point.
> He tried to help me, thinking \*\*\*“I’m sure she knows what she’s doing”.
> Obviously I didn’t\*\*\*, but you know, he believed in me. I don’t get that a
> lot with [human] teammates.
>
>
> Christy: They are perfectly selfless.
>
>
> Sheever: Yeah, they are.
>
>
> Michael: They also expect you to be.
>
>
> Sheever: Yeah. (laughing) Didn’t work out that way.
>
>
>

>
> Blitz: It was interesting because I could tell that we were doing something
> wrong, because they weren’t coming with us. I was like, “this is clearly an
> ‘us’ issue”, and I didn’t really know how to fix that. Regardless of what lane
> I went to, it just felt like I was making the wrong play, and it felt kind of
> bad in that regard. But it was cool because I knew that when I did make a move
> and they decided to go with me, that they deemed that was the correct thing to
> do. \*\*\*It felt like I was trying to solve a puzzle while playing the game\*\*\*.
>
>
>

Observers could also [tell](https://twitter.com/mtrc/status/1117179732074868736) that the AIs were not collaborating well with
their human teammates. The agents were simply behaving as though they had AI
teammates, rather than Sheever and Blitz. The agents’ models of their teammates
were \*incorrect\*[2](#fn:model). While this means they will sacrifice themselves when
it is in the team’s interest, it also means that they’ll leave without any
notice assuming that Sheever and Blitz will coordinate perfectly, as the AIs
would.

So is self-play actually a good algorithm to use to create \*collaborative\*
agents? We decided to put it to the test.

Overcooked
==========

To investigate this further, we wanted a simple collaborative environment that
nonetheless has a wide variety of potential strategies, so that the optimal
strategy is not obvious. This led us to consider the game [Overcooked](http://www.ghosttowngames.com/overcooked/), in
which players collaborate to cook up recipes quickly and serve them to hungry
customers. The game is particularly hard to coordinate in, primarily because of
the significant time pressure (which is not an issue for AI agents). Here’s an
example of good human play (starting at 15 seconds):

We created a simplified version of Overcooked, that allows us to focus on
particular coordination challenges that underlie joint planning for teams. In
our version, players must create and deliver soups. They must get onions from
the onion supply, place three of them in a pot, wait for the soup to cook, put
the soup in a plate, and then deliver the plate to a serving location. Players
need to employ both a good strategy (e.g. “you get the onions, I’ll grab the
dish”) as well as low level motion coordination (e.g. “let’s go clockwise so we
don’t crash into each other”). Despite its apparent simplicity, it is quite
challenging to act well in the environment: we developed a near-optimal
hierarchical A\\* planner, but the planning problem is difficult enough that our
planner can only solve two of our five layouts in a reasonable amount of time.

![](https://bair.berkeley.edu/static/blog/coordination/1 Game Dynamics.png)

Let’s suppose you and your friend Alice are playing on the layout above, and
you are trying to beat Bob and Charlie (who are playing on the same layout).
You’ve got a good strategy: at the start, Alice puts onions onto the counter in
the middle, while you go to the top to transfer the onions into the pot. As you
glance over at Bob and Charlie, you notice that they haven’t figured out this
strategy: they pick up each onion separately, and make a long trudge around the
layout to put the onion in the pot. Well, all the better for you; it looks like
you’re going to beat them even more soundly than you thought:

![](https://bair.berkeley.edu/static/blog/coordination/2 Alice \_ You successful coord.gif)
![](https://bair.berkeley.edu/static/blog/coordination/3 – Bob Charlie Long Way.gif)

\*Left: Alice (green) and you (blue) passing onions. Right: Bob (green) and
Charlie (blue) taking the long way.\*

But what if \*Alice\* doesn’t know about your strategy? In that case you head up
towards the pots, but to your chagrin Alice isn’t passing you onions – she’s
picked up a single onion and is making the long trudge over to place it in the
pot. You stand in front of the pot, staring at her pointedly, hoping she’ll
pass you some onions, but she continues to carry onions alone. You sigh, and
head back to get an onion yourself. Meanwhile, Bob and Charlie didn’t waste any
time, and so they win.

![](https://bair.berkeley.edu/static/blog/coordination/4 Alice \_ You unsuccessful coord.gif)
![](https://bair.berkeley.edu/static/blog/coordination/5 – Bob Charlie Long Way.gif)

\*Left: Alice (green) and you (blue) fail to coordinate. Right: Bob (green) and
Charlie (blue) taking the long way.\*

Interestingly, even though you knew a good strategy that the others did not,
Bob and Charlie still managed to beat you and Alice. This is the key
difference. In \*competitive\* settings (like between your team and Bob’s), if
your opponent is suboptimal and you don’t know it, you’ll simply beat them even
more soundly. In contrast, in \*collaborative\* settings, if your partner is
suboptimal and you don’t know it, team performance can be arbitrarily poor:
even worse than if you were exactly like your partner, with all their
suboptimalities.

As we saw above, self-play makes poor assumptions about its human partners (or
opponents, for that matter). Failing to accurately model your opponents doesn’t
matter much, since it is a competitive setting, but failing to accurately model
your partners in collaborative settings can be arbitrarily bad.

Understanding the differences
=============================

In the language of [game theory](https://en.wikipedia.org/wiki/Game\_theory), competition corresponds to a zero-sum game
(my gain is your loss and vice versa), while collaboration corresponds to a
common payoff game (my gain is your gain and vice versa).[3](#fn:gt)

\*\*Two player zero sum games\*\*. Self-play algorithms train the agent by having
the agent play games with itself, and updating so that it will be more likely
to win such games in the future. So, we would expect training to converge to an
equilibrium where the agent cannot improve its strategy when playing either
side of the game. For two player zero sum games, every such equilibrium
corresponds to a [min-max policy](https://en.wikipedia.org/wiki/Minimax#In\_zero-sum\_games). That is, the agent tries to \*maximize\*
the value it is going to get, assuming that its opponent is trying to
\*minimize\* the value the agent gets (which corresponds to maximizing their own
value, since the game is zero-sum).

An interesting fact about minimax policies is that an agent playing a minimax
policy is guaranteed to get \*at least as much value\* as if it were playing
itself. This is because of the dynamic we saw above: in competitive games, if
your opponent is suboptimal, you’ll beat them even more soundly. Indeed, it
seems almost obvious: if your opponent isn’t optimal, then they must be taking
an action that isn’t maximizing their value, which means it isn’t minimizing
your value, which means you’re going to do better than you expected.

![](https://bair.berkeley.edu/static/blog/coordination/6 Competitive Game Tree.png)
We can see this dynamic in the very simple game tree on the right. When
choosing an action, the agent reasons that if it takes the left path, the human
could go left, in which case it gets 1 reward, whereas if it takes the right
path, the human could go left, in which case it gets 3 reward. So, it goes
right. However, if the human then makes the suboptimal choice to go right, the
robot gets 7 reward instead: more than the 3 it expected.[4](#fn:tree)

\*\*Common payoff games\*\*. Now let’s consider common payoff games, where both the
agent and the human get exactly the same reward. The self-play agent is still
going to end up in an equilibrium where it can’t improve its strategy when
playing either side of the game. The agent is going to reach a max-max policy,
where the agent tries to \*maximize\* its own value, assuming that its partner is
also trying to maximize the same value. Unlike min-max policies, max-max
policies do not provide a lower bound on reward obtained when the partner
\*doesn’t\* maximize value, and in fact performance can become arbitrarily bad.

![](https://bair.berkeley.edu/static/blog/coordination/7 Collaborative Game Tree.png)
Consider the game tree on the right. Since the agent models the human as a
maximizer, it assumes that they can coordinate to reach the situation with 8
reward, and so goes left. However, if our suboptimal human ends up going left,
then the agent only gets 1 reward: the worst possible outcome!

\*\*Caveat\*\*. This argument applies to algorithms that reach equilibria. In
practice, due to the difficulty in training neural networks, our agents do not.
For example, neural nets are often very vulnerable to distribution shift. Since
humans likely play differently from the agent has seen during self-play
training, the agents could have had no idea what to do, which might cause them
to behave randomly. (This argument applies to both competitive and
collaborative settings.)

In what follows, we train an agent not with an optimal partner through
self-play, but with a model of a (suboptimal) human partner that we obtain from
human gameplay. We’ll call such agents “human-aware”.

Hypotheses
==========

With all of this conceptual groundwork, we can make some testable hypotheses
for the Overcooked environment in particular. Firstly, since playing with
humans induces a distribution shift, and since it is a collaborative game,
where self-play doesn’t provide an opponent-independent guarantee:

\*\*H1. A self-play agent will perform much more poorly when partnered with a
human (relative to being partnered with itself).\*\*

Since a human-aware agent will have a better model of their partner than a
self-play agent:

\*\*H2. When partnered with a human, a human-aware agent will achieve higher
performance than a self-play agent, though not as high as a self-play agent
partnered with itself.\*\*

Of course, a human-aware agent will require access to a dataset of human
gameplay. Couldn’t we use the dataset to train an agent using imitation
learning? Unfortunately, this would copy over the human’s suboptimalities: what
we actually want is an agent that knows how the human is suboptimal and deals
with it appropriately.

\*\*H3. When partnered with a human, a human-aware agent will achieve higher
performance than an agent trained via imitation learning.\*\*

![](https://bair.berkeley.edu/static/blog/coordination/8 Training Diagram.png)
To test these hypotheses, we need an implementation of a human-aware agent. In
this work, we take the most basic approach: given a dataset of human-human
gameplay, we train a \*human model\* using behavior cloning, and then train an
agent that plays well with this (fixed) human model using deep RL
(specifically, PPO). There are many ways to improve on this basic approach, as
we discuss in the Future Work section, but we expect that even this will be
enough to outperform self-play in our Overcooked environment.

Experiments
===========

To test our hypotheses, we created five different Overcooked layouts, shown
below.

![](https://bair.berkeley.edu/static/blog/coordination/9 All Layouts.png)

\*From left to right: Cramped Room, Asymmetric Advantages, Coordination Ring,
Forced Coordination, Counter Circuit.\*

Since the agent can play either of the two players, this creates ten scenarios.
We first test in simulation: we train a human model using behavior cloning on a
dataset of human-human gameplay. This model will stand in for our test-time
human, and so is called H\\_{proxy}. We manipulate the agent that must play
alongside H\\_{proxy}, where the options are an agent trained via self-play
(SP), an agent trained to imitate (BC), and a human-aware agent trained to play
well alongside a human model (PPO\\_{BC}). Note that the human-human gameplay
used to train BC is entirely separate from that used to train H\\_{proxy}.

We also report the performance of self-play with itself (SP + SP), which serves
as a rough upper bound on the optimal team performance, as well as a
human-aware agent that is given access to the test-time human model
(PPO\\_{H\\_{proxy}} + H\\_{proxy}), which serves as a rough upper bound on
the optimal performance when the agent must play with the test-time human.

The results are shown below. We see that all three hypotheses are supported. It
is interesting to note that even vanilla behavioral cloning often outperforms
self-play agents when paired with H\\_{proxy}.

![](https://bair.berkeley.edu/static/blog/coordination/10 SP Performances.png)

Qualitative results
-------------------

How exactly is the human-aware agent getting better results? One reason is that
it is more robust to different plans the human could have. In Coordination
Ring, PBT and SP agents often insist upon moving in a particular direction.
When the human wants to go the other way, they collide and get stuck. In
contrast, the human-aware agent simply chooses whichever path the human isn’t
taking.

![](https://bair.berkeley.edu/static/blog/coordination/11 SP failure.gif)
![](https://bair.berkeley.edu/static/blog/coordination/12 PPO\_BC success.gif)
![](https://bair.berkeley.edu/static/blog/coordination/13 PPO\_BC success other way.gif)

\*Self-play agent “stubbornly” colliding with the human (left), Human-aware agent
taking the appropriate route depending on the human’s direction (middle and
right).\*

Consider the gif with the self-play agent above. In the initial state, the
human is holding an onion and is facing up. What does the SP agent think the
human will do? Well, the SP agent “expects” the human to be like itself, and it
would have a 0-30% chance of up and 57-99.9% chance of down. (The ranges are
reporting the minimum and maximum across 5 seeds.) Thus, expecting the human to
move out of the way, SP decides to take the counterclockwise route – leading SP
to crash into the human.

Meanwhile, if we exclude the noop action, the BC model we used in training
assigns 99.8% chance of up and <0.01% chance of down, since the human is facing
up. Since the human is moving clockwise, it too moves clockwise to avoid
colliding with the human. Conversely, when the human is oriented in the
counterclockwise direction, the human-aware agent goes counterclockwise to
deliver the soup (even though that route is longer). It adaptively chooses the
route depending on the position and direction of the human.

Could the agent just be fragile?
--------------------------------

There is one other salient explanation for our quantitative and qualitative
results: perhaps the self-play agent is being forced off-distribution when it
plays with H\\_{proxy}, and the problem is not just that it doesn’t know
about its partner: it just doesn’t know how to play \*at all\* (even with itself)
in these new states it hasn’t encountered before. Meanwhile, playing with BC
causes the human-aware agent to be trained on such states. This is at least
part of the explanation for our results.

This fragility to distributional shift argument would suggest that
population-based training (PBT) would perform much better, since it involves a
population of agents and so the winning agent needs to be robust to the entire
population, rather than just itself. However, when repeating the experiment
with agents trained via PBT, we see broadly similar results.

Another way to test this is to implement an agent that does not suffer from
distributional shift, but still suffers from incorrect expectations about its
partner. We do this by implementing a \*planning agent\*, that uses a
hierarchical A\\* search to select the best plan for the team to take, and then
executes its part of the best plan’s first joint action. For the human-aware
version, we perform a hierarchical A\\* search, where the partner is assumed to
always take the action predicted as most likely by BC. We again see broadly
similar results, though only the version that gets access to the test-time
human does well.

User study
----------

Of course, the true test is whether these results will hold with actual humans.
By and large, they do, but not as clearly or strongly. H1 is clearly supported:
self-play agents perform worse with humans than with themselves. H2 is also
supported: PPO\\_{BC} is statistically significantly better than SP or PBT,
though the effect is much less pronounced than before. Since our method only
beats teams of humans in 5/10 configurations, the data is inconclusive about
H3.

![](https://bair.berkeley.edu/static/blog/coordination/14 Human Performances.png)

We speculate that there are two main reasons why the results are different with
real humans:

1. The difference between real humans and BC is much larger than the
difference between H\\_{proxy} and BC (both of which are trained on
human-human gameplay). As a result, PPO\\_{BC} doesn’t generalize to real
humans as well as it generalizes to H\\_{proxy}. This is particularly true on
the fourth and fifth layouts, where the BC-trained human model is quite bad.
2. Humans are able to figure out the coordination mechanisms that SP and PBT
use, and adapt to use those mechanisms themselves. In contrast, the BC model is
not able to adapt in this way. This significantly increases the performance of
SP and PBT.

You can see these effects for yourself, by [playing the demo](https://humancompatibleai.github.io/overcooked-demo/)!

Discussion
==========

So far we’ve seen that self play algorithms form an incorrect “expectation”
about their partner, and incorporating even the naive human model produced by
behavior cloning beats self play when playing with humans. It even beats
human-human teams sometimes!

You might hope that rather than understanding humans, which requires expensive
human data, we could instead simply train our agents to be robust to a wide
variety of agents, which would automatically make them robust to humans.
However, this is exactly what PBT is supposed to do, and we found that PBT
ended up having the same kinds of problems as SP. Nonetheless, it could be that
with a larger population or other tweaks to the algorithm, PBT could be
improved.

You might also think that our results are primarily explained by analyzing how
many states an algorithm has been trained on: SP and PBT fall into
near-deterministic patterns, while PPO\\_{BC} must cope with the
stochasticity of BC, and so it is trained on a wider variety of states, which
makes it work better with humans. However, we saw approximately the same
pattern with the planning agent, which is robust on all states. In addition,
the entropy bonus in PPO keeps SP and PBT at least somewhat stochastic.

One way to view the problem we have outlined is that AI systems trained via
self-play end up using coordination protocols that humans do not use. However,
it is possible that this only happens because we are running the algorithms on
a single layout at the time, and so they learn a protocol that is specialized
to that layout. In contrast, human coordination protocols are likely much more
general. This suggests that we could make AI protocols similar to human ones by
forcing the AI protocols to be more general. In particular, if we train AI
systems via self-play to play on \*arbitrary\* maps, they will have to learn more
general coordination protocols, that may work well with human protocols. We
would like to investigate this possibility in the future.

Future Work
===========

To demonstrate how important it is to model humans, we used the most naive
human model we could and showed that even that leads to significant
improvements over self-play. Of course, for best performance, we’d like to use
better human models. There are several areas for improvement:

1. We could use more data to make the model more accurate, or use more
sophisticated methods than behavior cloning to learn the human model
2. While the human model is trained on human-human gameplay, it is used in the
context of human-AI gameplay, which may be very different and cause the BC
model to suffer from distributional shift. We could alternate between training
PPO\\_{BC} and collecting new human-AI gameplay to improve the BC model.
3. Alternatively, we could try to use models that are more robust to
distributional shift, such as models based on Theory of Mind, where the human
is modeled as approximately optimizing some reward function.
4. So far, we have made the obviously false assumption that all humans play
exactly the same. Instead, we could learn a space of strategies that humans
tend to use, and try to identify the test human’s strategy and adapt to it on
the fly.
5. Another obviously false assumption we make is that the human is
\*stationary\*, that is, the human’s policy doesn’t change over time. But of
course, humans learn and adapt to their partners (and we see strong
observational evidence of this in the user study, where humans learn the
protocols that SP and PBT use). If we are able to model this learning, we
could build agents that actively \*teach\* humans better coordination protocols
that achieve higher reward.

Alternatively, rather than attempting to completely fix the model’s
expectations about its partner, we could train it to be robust to a wide
variety of partners. This will limit the peak performance, since the agent
cannot specialize to humans in particular, but it could still give a suitably
good result, and in particular it should beat imitation learning. We showed
that vanilla PBT was insufficient for this task, but we find it plausible that
variants of PBT could work.

Another aspect to investigate further is the extent to which these problems are
caused by a lack of robustness to \*states\* as opposed to \*partners\*. Currently,
when a self-play agent is forced off distribution, it behaves in a clearly
suboptimal way (such that the agent wouldn’t coordinate well even with itself).
If we had agents that at least played coherently with respect to \*some\* partner
on all states, that could potentially fix most of the problem. (However, our
planning experiments show that some problems will remain.) With deep RL,
perhaps this could be done by incentivizing exploration via intrinsic
motivation, or by generating a random initial state instead of a fixed one
during each episode.

We’re excited by the potential of Overcooked as a benchmark for human-AI
collaboration, and we hope to see more research that paves the way to AI
systems that are increasingly beneficial for humans.

\*This post is based on the paper “[On the Utility of Learning about Humans for
Human-AI Coordination](https://arxiv.org/abs/1910.05789)”, to be presented at NeurIPS 2019. You can play with
our trained agents or watch them play each other [here](https://humancompatibleai.github.io/overcooked-demo/). We’ve taken
particular care to separately publish our [environment code](https://github.com/HumanCompatibleAI/overcooked\_ai), [DRL code](https://github.com/HumanCompatibleAI/human\_aware\_rl),
[visualization code](https://github.com/HumanCompatibleAI/overcooked-demo), and [user study code](https://github.com/HumanCompatibleAI/overcooked-hAI-exp), so that each can be reused
and modified. We would particularly welcome pull requests to add more
functionality to the environment.\*

---

1. Quotes have been edited for clarity. [↩](#fnref:quotes)
2. Although this point also applies to the competitive setting, the
problems it causes are not as significant, as we will see later in the
post. [↩](#fnref:model)
3. Other general-sum games typically have both competitive and
collaborative aspects. While we don’t study them in this work, our results
suggest that the more collaborative the game is, the worse self-play will
perform. [↩](#fnref:gt)
4. That said, the agent might have been able to do better if it knew how
the human would behave. Suppose it knew that if it went left, the human
would then have gone right. Then by going left, the agent would get 8
reward; better than the 7 reward it ended up getting by going right. [↩](#fnref:tree)


Robots Learning to Move like Animals

![](https://bair.berkeley.edu/static/blog/laikago/00\_teaser.gif)

\*Quadruped robot learning locomotion skills by imitating a dog.\*

Whether it’s a dog chasing after a ball, or a monkey swinging through the
trees, animals can effortlessly perform an incredibly rich repertoire of agile
locomotion skills. But designing controllers that enable legged robots to
replicate these agile behaviors can be a very challenging task. The superior
agility seen in animals, as compared to robots, might lead one to wonder: can
we create more agile robotic controllers with less effort by directly imitating
animals?

In this work, we present a framework for learning robotic locomotion skills by
imitating animals. Given a reference motion clip recorded from an animal (e.g.
a dog), our framework uses reinforcement learning to train a control policy
that enables a robot to imitate the motion in the real world. Then, by simply
providing the system with different reference motions, we are able to train a
quadruped robot to perform a diverse set of agile behaviors, ranging from fast
walking gaits to dynamic hops and turns. The policies are trained primarily in
simulation, and then transferred to the real world using a latent space
adaptation technique, which is able to efficiently adapt a policy using only a
few minutes of data from the real robot.

Framework
---------

Our framework consists of three main components: motion retargeting, motion
imitation, and domain adaptation. 1) First, given a reference motion, the
motion retargeting stage maps the motion from the original animal’s morphology
to the robot’s morphology. 2) Next, the motion imitation stage uses the
retargeted reference motion to train a policy for imitating the motion in
simulation. 3) Finally, the domain adaptation stage transfers the policy from
simulation to a real robot via a sample efficient domain adaptation process. We
apply this framework to learn a variety of agile locomotion skills for a
[Laikago](http://www.unitree.cc/e/action/ShowInfo.php?classid=6&id=1)
quadruped robot.

![](https://bair.berkeley.edu/static/blog/laikago/01\_overview.gif)

\*The framework consists of three stages: motion retargeting, motion imitation,
and domain adaptation. It receives as input motion data recorded from an
animal, and outputs a control policy that enables a robot to reproduce the
motion in the real world.\*

### Motion Retargeting

An animal’s body is generally quite different from a robot’s body. So before
the robot can imitate the animal’s motion, we must first map the motion to the
robot’s body. The goal of the retargeting process is to construct a reference
motion for the robot that captures the important characteristics of the
animal’s motion. To do this, we first identify a set of source keypoints on the
animal’s body, such as the hips and the feet. Then, corresponding target
keypoints are specified on the robot’s body.

![](https://bair.berkeley.edu/static/blog/laikago/02\_keypoints\_dog.png)
![](https://bair.berkeley.edu/static/blog/laikago/02\_keypoints\_robot.png)

\*Inverse-kinematics (IK) is used to retarget mocap clips recorded from a real
dog (left) to the robot (right). Corresponding pairs of keypoints (red) are
specified on the dog and robot’s bodies, and then IK is used to compute a pose
for the robot that tracks the keypoints.\*

Next, inverse-kinematics is used to construct a reference motion for the robot
that tracks the corresponding keypoints from the animal at every timestep.

![](https://bair.berkeley.edu/static/blog/laikago/03\_retarget\_pace.gif)
![](https://bair.berkeley.edu/static/blog/laikago/04\_retarget\_spin.gif)

\*Inverse-kinematics is used to retarget mocap clips recorded from a dog to the robot.\*

### Motion Imitation

After retargeting the reference motion to the robot, the next step is to train
a control policy to imitate the retargeted motion. But reinforcement learning
algorithms can take a long time to learn an effective policy, and directly
training on a real robot can be fairly dangerous (both for the robot and its
human companions). So, we instead opt to perform most of the training in the
comforts of simulation, and then transfer the learned policy to the real world
using more sample efficient adaptation techniques. All simulations are
performed using [PyBullet](https://pybullet.org/).

The policy $\pi(\mathbf{a} | \mathbf{s}, \mathbf{g})$, takes as input a state
$\mathbf{s}$, which represents the configuration of the robot’s body, and a
goal $\mathbf{g}$, which specifies target poses from the reference motion that
the robot is to imitate. It then outputs an action $\mathbf{a}$, which
specifies target angles for PD controllers at each of the robot’s joints. To
train the policy to imitate a reference motion, we use a reward function that
encourages the robot to minimize the difference between the pose of the
reference motion $\hat{\mathbf{q}}\\_t$ and the pose of the simulated character
$\mathbf{q}\\_t$ at every timestep $t$,

\[r\\_t = \exp \Big[ - \| \hat{\mathbf{q}}\\_t - \mathbf{q}\\_t \|^2 \Big]\]

By simply using different reference motions in the reward function, we can
train a simulated robot to imitate a variety of different skills.

![](https://bair.berkeley.edu/static/blog/laikago/05\_sim\_pace.gif)
![](https://bair.berkeley.edu/static/blog/laikago/06\_sim\_spin.gif)

\*Reinforcement learning is used to train a simulated robot to imitate the
retargeted reference motions.\*

### Domain Adaptation

Since simulators generally provide only a coarse approximation of the real
world, policies trained in simulation often perform fairly poorly when deployed
on a real robot. Therefore, to transfer a policy trained in simulation to the
real world, we use a sample efficient domain adaptation techniques that can
adapt the policy to the real world using only a small number of trials on the
real robot. To do this, we first apply
[domain randomization](https://xbpeng.github.io/projects/SimToReal/index.html) during training in simulation, which randomly varies the
dynamics parameters, such as mass and friction. The dynamics parameters are
then also collected into a vector $\mu$ and encoded into a latent presentation
$\mathbf{z}$ by an encoder $E(\mathbf{z} | \mu)$. The latent encoding is
passed as an additional input to the policy $\pi(\mathbf{a} | \mathbf{s},
\mathbf{g}, \mathbf{z})$.

![](https://bair.berkeley.edu/static/blog/laikago/07\_policy.png)

\*The dynamics parameters of the simulation are varied during training, and also
encoded into a latent representation that is provided as an additional input
to the policy.\*

When transferring the policy to a real robot, we remove the encoder and
directly search for a $\mathbf{z}$ that maximizes the robot’s rewards in the
real world. This is done using
[advantage weighted regression](https://xbpeng.github.io/projects/AWR/index.html),
a simple off-policy reinforcement learning algorithm. In our experiments, this
technique is often able to adapt a policy to the real world with less than 50
trials, which corresponds to roughly 8 minutes of real-world data.

![](https://bair.berkeley.edu/static/blog/laikago/08\_adaptation\_pace.gif)

![](https://bair.berkeley.edu/static/blog/laikago/09\_adaptation\_spin.gif)

\*Comparison of policies before and after adaptation on the real robot. Before
adaptation, the robot is prone to falling. But after adaptation, the policies
are able to more consistently execute the desired skills.\*

Results
-------

Our framework is able to train a robot to imitate various locomotion skills
from a dog, including different walking gaits, such as pacing and trotting, as
well as a fast spinning motion. By simply playing the forwards walking motions
backwards, we are also able to train the robot to walk backwards.

![](https://bair.berkeley.edu/static/blog/laikago/10\_real\_pace.gif)
![](https://bair.berkeley.edu/static/blog/laikago/11\_real\_trot.gif)

![](https://bair.berkeley.edu/static/blog/laikago/12\_real\_spin.gif)
![](https://bair.berkeley.edu/static/blog/laikago/13\_real\_backward\_trot.gif)

\*Laikago imitating various skills from a dog.\*

In addition to imitating motions from real dogs, we can also imitate
artist-animated keyframe motion, including a dynamic hop-turn:

![](https://bair.berkeley.edu/static/blog/laikago/14\_real\_sidesteps.gif)
![](https://bair.berkeley.edu/static/blog/laikago/14\_real\_turn.gif)

Hop Turn
![](https://bair.berkeley.edu/static/blog/laikago/15\_real\_hopturn.gif)

\*Skills learned by imitating artist-animated keyframe motions.\*

We also compared the learned policies with the manually-designed controllers
provided by the manufacturer. Our policies are able to learn faster gaits.

![](https://bair.berkeley.edu/static/blog/laikago/16\_comp\_trot.gif)

\*Comparison of learned trotting gait with the built-in gait provided by the
manufacturer.\*

Overall, our system has been able to reproduce a fairly diverse corpus of
behaviors with a quadruped robot. However, due to hardware and algorithmic
limitations, we have not been able to imitate more dynamic motions such as
running and jumping. The learned policies are also not as robust as the best
manually-designed controllers. Exploring techniques for further improving the
agility and robustness of these learned policies could be a valuable step
towards more complex real-world applications. Extending this framework to learn
[skills from videos](https://xbpeng.github.io/projects/SFV/index.html)
would also be an exciting direction, which can substantially increase the
volume of data from which robots can learn from.

To learn more,
[check out the paper and code](https://xbpeng.github.io/projects/Robotic\_Imitation/index.html).

We would like to thank Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan,
Sergey Levine, Byron David, Thinh Nguyen, Gus Kouretas, Krista Reymann, and
Bonny Ho for all their support and contribution to this work. This project was
done in collaboration with Google Brain.


The Brain and Computation

Measuring Computation
=====================

The computational performance of microprocessors can be quantified by measuring the number of floating-point arithmetic operations the processor can perform per second (FLOPS). This number is very useful for comparing different hardware being used for numerically intensive applications like scientific computing or mining [fake internet points](https://en.wikipedia.org/wiki/Cryptocurrency), but some have attempted to quantify the computation done by the human brain in these terms to reason about how difficult it would be to run a human-level intelligence on modern computing hardware.

This post will discuss a few of the issues associated with measuring the computational performance of the brain with FLOPS, and a follow-up post will consider specific estimates.

Does it make sense to think about the computational capacity of the brain in terms of FLOPS?
============================================================================================

There is a line of thinking that goes something like:

>
> Neurons generate action potentials. Action potentials are stereotyped signals, so the computation that happens in the brain is essentially digital, so it makes sense to compare brains to digital computers, and synaptic operations are kind of like arithmetic operations.
>
>
>

This may or may not be a good enough approximation, but it’s definitely a lossy approximation.

Brains probably aren’t bottlenecked on arithmetic
-------------------------------------------------

A common objection to measuring the performance of the brain in FLOPS is that computation in the brain isn’t bottlenecked by arithmetic capacity, but rather by information flow, so the capacity of the brain should be measured in \*traversed edges per second\* (TEPS) rather than FLOPS. Synaptic connections between neurons tend to be sparse and axons tend to be long, which seems to suggest a lot of neural tissue is dedicated to pushing signals around rather than performing arithmetic on them[1](#fn-1).

Brains are asynchronous
-----------------------

Microprocessors are clocked circuits. When a computation unfolds on a microprocessor, it proceeds in discrete, well-delineated steps with one occurring each processor cycle. This method of computation is fundamentally synchronous.

Brains don’t have a clock: neurons fire when they fire, which usually isn’t very often (one to ten times a second), but is sometimes much faster (up to around 1000 Hz)[2](#fn-2). And the phase of the neural spike trains also seem to be important[3](#fn-3), which further complicates the comparison.

Non-spiking neurons
-------------------

Many neurons [don’t even spike](https://en.wikipedia.org/wiki/Non-spiking\_neuron), having graded, non-stereotyped potentials. The best-studied are the photo-receptive neurons in the retina, but they occur throughout the brain and it’s unclear how to integrate them into the larger computational picture of the brain.

Conclusion
==========

This post was not meant to be comprehensive, and is merely meant to highlight the strangeness and limitations of thinking of the limits of neural computation in terms of FLOPS.

---

1. Limitations in the ability of evolution to modify the basic vertebrate developmental plan lead can lead to bizarre inefficiencies, like the optic nerve needing to carry signals [from the retina to the back of the head](https://en.wikipedia.org/wiki/Lateral\_geniculate\_nucleus) before being processed in the visual cortex, or in the case of giraffes the laryngeal nerve needing to take a [>4 meter detour](https://en.wikipedia.org/wiki/Recurrent\_laryngeal\_nerve#Evidence\_of\_evolution). [↩](#fnref-1 "Jump back to footnote 1 in the text")
2. See [sparse coding](https://en.wikipedia.org/wiki/Neural\_coding#Sparse\_coding). [↩](#fnref-2 "Jump back to footnote 2 in the text")
3. See [phase coding](https://en.wikipedia.org/wiki/Neural\_coding#Phase-of-firing\_code). [↩](#fnref-3 "Jump back to footnote 3 in the text")


How rapidly are GPUs improving in price performance?

Note: Equations will not render properly with javascript disabled.

Introduction
============

Many of the impressive results in deep learning in recent years have been achieved through massive investment in hardware needed for training, with projects like AlphaGo Zero using [$25 million](https://www.nature.com/news/self-taught-ai-is-best-yet-at-strategy-game-go-1.22858#/ref-link-2) worth of computer hardware. Given this, improvements in price-performance of hardware used for deep learning will play an important role in determining the scale of projects in the coming years.

While machine learning [ASICs](https://en.wikipedia.org/wiki/Application-specific\_integrated\_circuit) like TPUs are likely the future, the recent deep learning boom was powered by GPUs[1](#fn-1). The architectures of TPUs and GPUs differ in important ways, but much of the design and fabrication process is similar and both are largely focused on efficient, parallelized arithmetic[2](#fn-2), so trends observed in GPUs can inform us about what to expect from TPUs.

Commonly mentioned figures for the price-performance generalization of Moore’s Law suggest that price-performance doubles roughly every two years, but this figure warranted further investigation.

Data
====

We constructed a [data set](/data/gpu.csv) containing the model name, launch date, single precision performance in GFLOPS, and release price in non-inflation adjusted US dollars for 223 Nvidia and AMD GPUs (scraped from Wikipedia)[3](#fn-3). The data set covered almost two decades, so prices were adjusted to 2018 US dollars using the [Consumer Price Index](https://fred.stlouisfed.org/series/CPIAUCNS).

Analysis
========

Fitting an exponential to the data-set yielded the curve (where \*t\* equals the number of years since the start of 2070):

\begin{equation}
 f(t) \approx 2.26 e^{0.470 t}
\end{equation}
![GPU Price-Performance](/images/gpu\_full\_fit.png)

This implies a doubling time of ~1.5 years. It should be noted that this is somewhat misleading because the price-performance curve isn’t a clean exponential. Inspecting a log-plot suggests that price-performance has been in a distinctly slower growth regime since around 2012.

![Log-plot of GPU price-performance](/images/gpu\_log.png)

Fitting to data from 2012 or later yields the curve:

\begin{equation}
 f(t) \approx 15.2 e^{0.176 t},
\end{equation}

corresponding to a doubling time of ~3.9 years.

![Log-plot of GPU price-performance](/images/gpu\_log\_fit.png)

External Discussion
===================

\* [Comments on LessWrong](https://www.lesswrong.com/posts/iGznDsxfB564Lobam/how-rapidly-are-gpus-improving-in-price-performance) about this article

---

1. GPUs are still more cost effective than TPUs, but have lower serial computation speed. [↩](#fnref-1 "Jump back to footnote 1 in the text")
2. This is not nearly as true as with CPUs which have managed to extract performance improvements from [increasingly arcane changes to control circuitry](http://www.lighterra.com/papers/modernmicroprocessors/). [↩](#fnref-2 "Jump back to footnote 2 in the text")
3. AI Impacts has a [similar data set](https://docs.google.com/spreadsheets/d/1yqX2cENwkOxC26wV\_sBOvV0NxHzzfmL6tU7StzrFXRc/edit#gid=51141192). [↩](#fnref-3 "Jump back to footnote 3 in the text")

if (!document.getElementById('mathjaxscript\\_pelican\\_#%@#$@#')) {
 var align = "center",
 indent = "0em",
 linebreak = "false";

 if (false) {
 align = (screen.width < 768) ? "left" : align;
 indent = (screen.width < 768) ? "0em" : indent;
 linebreak = (screen.width < 768) ? 'true' : linebreak;
 }

 var mathjaxscript = document.createElement('script');
 mathjaxscript.id = 'mathjaxscript\\_pelican\\_#%@#$@#';
 mathjaxscript.type = 'text/javascript';
 mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML\\_HTMLorMML';
 mathjaxscript[(window.opera ? "innerHTML" : "text")] =
 "MathJax.Hub.Config({" +
 " config: ['MMLorHTML.js']," +
 " TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'AMS' } }," +
 " jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
 " extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
 " displayAlign: '"+ align +"'," +
 " displayIndent: '"+ indent +"'," +
 " showMathMenu: true," +
 " messageStyle: 'normal'," +
 " tex2jax: { " +
 " inlineMath: [ ['\\\\(','\\\\)'] ], " +
 " displayMath: [ ['$$','$$'] ]," +
 " processEscapes: true," +
 " preview: 'TeX'," +
 " }, " +
 " 'HTML-CSS': { " +
 " styles: { '.MathJax\\_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
 " linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
 " }, " +
 "}); " +
 "if ('default' !== 'default') {" +
 "MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
 "var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
 "VARIANT['normal'].fonts.unshift('MathJax\\_default');" +
 "VARIANT['bold'].fonts.unshift('MathJax\\_default-bold');" +
 "VARIANT['italic'].fonts.unshift('MathJax\\_default-italic');" +
 "VARIANT['-tex-mathit'].fonts.unshift('MathJax\\_default-italic');" +
 "});" +
 "MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
 "var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
 "VARIANT['normal'].fonts.unshift('MathJax\\_default');" +
 "VARIANT['bold'].fonts.unshift('MathJax\\_default-bold');" +
 "VARIANT['italic'].fonts.unshift('MathJax\\_default-italic');" +
 "VARIANT['-tex-mathit'].fonts.unshift('MathJax\\_default-italic');" +
 "});" +
 "}";
 (document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}


Insight-based AI timelines model

Note: the demos on this page will not work with javascript disabled.

 For various “percentages of the way” done AI research could be, in terms of percentage of the necessary insights discovered, what is the probability that AI research is not yet that percentage done?

100%

P(no more than this much of the way done)

0%

0%

100%

Proportion of required insights that have been discovered

Clear
Pre-set priors
--------------

Instead of drawing a cumulative distribution function, you can instead use a pre-set prior. These priors are based on the [Pareto distribution](https://en.wikipedia.org/wiki/Pareto\_distribution). To make choice of the parameter more intuitive, we parameterize the distribution in terms of a probability \*q\*, equal to the probability that a doubling in number of insights (starting from the minimum number of insights) would result in a sufficient set of insights.

Minimum plausible number of insights required:

\*q\*: Pareto distribution

Pareto distribution, \*q\* ~ Uniform(0,1)
\*α\*:
\*β\*:
Pareto distribution, \*q\* ~ Beta(α, β)

Resulting timeline
------------------

Assuming a linear increase in number of required insights over time, the following cumulative distribution function for time when all required insights are discovered is implied by these beliefs.

 Adjust the maximum year displayed:
 Derivation
===========

 How was this data generated? Jessica Taylor, Jack Gallagher, and Baeo Maltinsky spent a few hours generating a list of AI insights that seemed around the same order of significance or more significant than the insight of LSTM (specifically, the insight of inventing LSTM given that RNNs were already invented). The following is a plot of number of AI insights in our list over time since 1850.

The model assumes that insights increase linearly over time. The increase has been roughly linear since 1945, but this could change due to low hanging fruit, expanding research avenues, changes in the number and effectiveness of research institutions, and so on. The model does not distinguish between insights in our list (which we selected according to some subjective estimation of importance) and specifically \*required\* insights; however, if the percentage of insights that are actually required stays somewhat constant over time, this does not significantly affect the timeline.

The list of insights and their years can be found in [this document](../docs/AI\_insights.pdf).


Revisiting the Insights model

Note: This demo is in beta, and you may experience issues such as strange numerical behavior at this time.

 Last year, we released our [insights-based model](http://mediangroup.org/insights) that generated a projected timeline using historical data and a prior distribution. We’ve revisited it to address its limitations and improve the data it draws from.

 The model relies on the assumption that progress in AI relies on accumulating \*insights\*, fundamental advances in our understanding, that allow for improvements in capacity without increase in resources expended. This choice makes an attempt to separate out the effects of true technological advancement from the effects of an increase in computing power devoted to a problem, both of which can increase the capacity of machine intelligence to solve complex problems. Computational power is an expensive, finite resource, and without a paradigm-shifting improvement in computing itself, precise allocation of that power alone will not be enough to continue advancing AI’s problem-solving capabilities.

 The interactive model below provides two methods of capturing a prior about how many more advances in understanding are required to achieve human-level machine intelligence. Based on that prior, and on the pace of insight discovery during a particular historical period, we compute a probability distribution over time of the likelihood humans will develop human-level AI. Results of this calculation are shown in the “Implied timeline” graph below.

Step 1: Specify a prior for current progress
--------------------------------------------

### Option A: Draw a distribution

 For various “percentages of the way” done AI research could be, in terms of percentage of the necessary insights discovered, what is the probability that AI research is not yet that percentage done?

 The graph below allows you to draw a distribution of how likely it is we have achieved a particular portion of the insights required for human-level machine intelligence.

Reset

### Option B: Pre-set priors from Pareto distribution

 Instead of drawing a cumulative distribution function, you can instead use a pre-set prior based on a [Pareto distribution](https://en.wikipedia.org/wiki/Pareto\_distribution).

 To make the choice of Pareto distribution more intuitive, we parameterize the distribution in terms of a probability \*q\*, equal to the probability that a doubling in number of insights (starting from the minimum number of insights) would result in a sufficient set of insights. \*q\* can be set directly, or we can sample from a mixture of Pareto distirbutions, where the \*q\* parameters are sampled from a uniform distribution or a beta distribution.

#### Number of samples to take when running the simulation

#### Set \*q\* directly

\*q\*

 Set \*q\*

#### Sample \*q\* uniformly over (0,1)

 Sample \*q\*

#### Sample \*q\* with Beta(α, β)

\*α\*:

\*β\*:

 Sample \*q\*

 Note: The simulator can be very slow for larger values of \*q\*, as most of the samples need to be thrown away.

Step 2: Specify pace of progress
--------------------------------

 Which period in history is most representative of the future pace of AI insight discovery?

 The graph below plots the aggregate of insights discovered over time and allows selection of a particular period of history in AI research. The curve fit to that period (linear, exponential, or sigmoidal) is used to project the future distribution of discoveries.

Regression mode:

Linear
Exponential
Sigmoidal

Result: Implied timeline
------------------------

 Sources
--------

 The data used in this model is available as a [JSON file](http://mediangroup.org/docs/insights.json).
 The [source code](https://github.com/Median-Group/insights2) for the demo can be found on the Median Group [github](https://github.com/Median-Group/).