Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration

During purely curious exploration, the JACO arm discovers how to pick up cubes, moves them around the workspace and even explores whether they can be balanced on their edges.

Curious exploration enables OP3 to walk upright, balance on one foot, sit down and even catch itself safely when leaping backwards - all without a specific target task to optimise for.

Intrinsic motivation [1, 2] can be a powerful concept to endow an agent with a mechanism to continuously explore its environment in the absence of task information. One common way to implement intrinsic motivation is via curiosity learning [3, 4]. With this method, a predictive model about the environment's response to an agent's actions is trained alongside the agent's policy. This model can also be called a world model. When an action is taken, the world model makes a prediction about the agent's next observation. This prediction is then compared to the true observation made by the agent. Crucially, the reward given to the agent for taking this action is scaled by the error it made when predicting the next observation. This way, the agent is rewarded for taking actions whose outcomes are not yet well predictable. Simultaneously, the world model is updated to better predict the outcome of said action.

This mechanism has been applied successfully in on-policy settings, e.g. to beat 2D computer games in an unsupervised way [4] or to train a general policy which is easily adaptable to concrete downstream tasks [5]. However, we believe that the true strength of curiosity learning lies in the diverse behaviour which emerges during the curious exploration process: As the curiosity objective changes, so does the resulting behaviour of the agent thereby discovering many complex policies which could be utilised later on, if they were retained and not overwritten.

[In this paper](https://arxiv.org/abs/2109.08603), we make two contributions to study curiosity learning and harness its emergent behaviour: First, we introduce *SelMo*, an off-policy realisation of a self-motivated, curiosity-based method for exploration. We show that using SelMo, meaningful and diverse behaviour emerges solely based on the optimisation of the curiosity objective in simulated manipulation and locomotion domains. Second, we propose to extend the focus in the application of curiosity learning towards the identification and retention of emerging intermediate behaviours. We support this conjecture with an experiment which reloads self-discovered behaviours as pretrained, auxiliary skills in a hierarchical reinforcement learning setup.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335aefce7a171a10eeec26_unnamed.jpg)The control flow of the SelMo method: The agent (actor) collects trajectories in the environment using its current policy and stores them in the model replay buffer on the left. The connected world model samples uniformly that buffer and updates its parameters for forward prediction using stochastic gradient descent (SGD). The sampled trajectories are assigned curiosity rewards scaled by their respective prediction error under the current world model. The labeled trajectories are then passed on to the policy replay buffer on the right. Maximum a posteriori policy optimisation (MPO) [6] is used to fit Q-function and policy based on samples from the policy replay. The resulting, updated policy is then synced back into the actor.We run SelMo in two simulated continuous control robotic domains: On a 6-DoF JACO arm with a three-fingered gripper and on a 20-DoF humanoid robot, the OP3. The respective platforms present challenging learning environments for object manipulation and locomotion, respectively. While only optimising for curiosity, we observe that complex human-interpretable behaviour emerges over the course of the training runs. For instance, JACO learns to pick up and move cubes without any supervision or the OP3 learns to balance on a single foot or sit down safely without falling over.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335b093f091c2012d2af1f_unnamed%20(1).jpg)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335b11e94a89e257b024bf_unnamed%20(3).jpg)Example training timelines for JACO and the OP3. While optimising for the curiosity objective, complex, meaningful behaviour emerges in both manipulation and locomotion settings. The full videos can be found at the top of this page.However, the impressive behaviours observed during curious exploration have one crucial drawback: They are not persistent as they keep changing with the curiosity reward function. As the agent keeps repeating a certain behaviour, e.g. JACO lifting the red cube, the curiosity rewards accumulated by this policy are diminishing. Consequently, this leads to the learning of a modified policy which acquires higher curiosity rewards again, e.g. moving the cube outside the workspace or even attending to the other cube. But this new behaviour overwrites the old one. However, we believe that retaining the emergent behaviours from curious exploration equips the agent with a valuable skill set to learn new tasks more quickly. In order to investigate this conjecture, we set up an experiment to probe the utility of the self-discovered skills.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335b268209b6493fd3dc43_unnamed%20(2).jpg)We treat randomly sampled snapshots from different phases of the curious exploration as auxiliary skills in a modular learning framework [7] and measure how quickly a new target skill can be learned by using those auxiliaries. In the case of the JACO arm, we set the target task to be "lift the red cube" and use five randomly sampled self-discovered behaviours as auxiliaries. We compare the learning of this downstream task to an SAC-X baseline [8] which uses a curriculum of reward functions to reward reaching and moving the red cube which ultimately facilitates to learn lifting as well. We find that even this simple setup for skill-reuse already speeds up the learning progress of the downstream task commensurate with a hand designed reward curriculum. The results suggest that the automatic identification and retention of useful emerging behaviour from curious exploration is a fruitful avenue of future investigation in unsupervised reinforcement learning.

Oliver Groth
Markus Wulfmeier
Giulia Vezzani
Vibhavari Dasagi
Tim Hertweck
Roland Hafner
Nicolas Heess
and Martin Riedmiller


Challenges in Detoxifying Language Models

#### **Undesired Behavior from Language Models**

Language models trained on large text corpora can generate [fluent text](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), and show promise as [few/zero shot learners](https://arxiv.org/abs/2005.14165) and code generation tools, amongst other capabilities. However, prior research has also identified several issues with LM use that should be addressed, including [distributional biases](https://arxiv.org/abs/1911.03064), [social stereotypes](https://arxiv.org/abs/2004.09456), potentially revealing [training samples](https://arxiv.org/abs/2012.07805), and other [possible LM harms](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922). One particular type of LM harm is the generation of [toxic language](https://arxiv.org/abs/2009.11462), which includes hate speech, insults, profanities and threats.

In our paper, we focus on LMs and their [propensity](https://arxiv.org/abs/2009.11462) to generate toxic language. We study the effectiveness of different methods to mitigate LM toxicity, and their side-effects, and we investigate the reliability and limits of classifier-based automatic toxicity evaluation.

Following the definition of toxicity developed by [Perspective API](https://perspectiveapi.com/), we here consider an utterance to be *toxic if it is rude, disrespectful, or unreasonable language that is likely to make someone leave a discussion*. However, we note two important caveats. First, toxicity judgements are subjective—they depend both on the raters evaluating toxicity and their cultural background, as well as the inferred context. While not the focus of this work, it is important for future work to continue to develop this above definition, and clarify how it can be fairly applied in different contexts. Second, we note that toxicity covers only one aspect of possible LM harms, excluding e.g. harms arising from distributional model bias.

#### **Measuring and Mitigating Toxicity**

To enable safer language model use, we set out to measure, understand the origins of, and mitigate toxic text generation in LMs. There has been prior work which has considered various approaches towards reducing LM toxicity, either by [fine-tuning](https://arxiv.org/abs/2004.10964) [pre-trained LMs](https://arxiv.org/abs/2009.11462), by [steering model generations](https://openreview.net/forum?id=H1edEyBKDS), or through direct [test-time filtering](https://arxiv.org/abs/2104.06390). Further, prior [work](https://arxiv.org/abs/2009.11462) has introduced automatic metrics for measuring LM toxicity, both when prompted with different kinds of prompts, as well as in unconditional generation. These metrics rely on the toxicity scores of the widely used [Perspective API](https://perspectiveapi.com/) model, which is trained on online comments annotated for toxicity.

In our study we first show that a combination of relatively simple baselines leads to a drastic reduction, as measured by previously introduced LM toxicity [metrics](https://arxiv.org/abs/2009.11462). Concretely, we find that a combination of i) filtering the LM training data annotated as toxic by [Perspective API](https://perspectiveapi.com/), ii) filtering generated text for toxicity based on a separate, fine-tuned BERT classifier trained to detect toxicity, and iii) [steering](https://arxiv.org/abs/1912.02164) the generation towards being less toxic, is highly effective at reducing LM toxicity, as measured by automatic toxicity metrics. When prompted with toxic (or non-toxic) prompts from the [RealToxicityPrompts](https://arxiv.org/abs/2009.11462) dataset, we see a 6-fold (or 17-fold) reduction compared with the previously reported state-of-the-art, in the aggregate *Probability of Toxicity* metric. We reach a value of zero in the unprompted text generation setting, suggesting that we have exhausted this metric. Given how low the toxicity levels are in absolute terms, as measured with automatic metrics, the question arises to what extent this is also reflected in human judgment, and whether improvements on these metrics are still meaningful, especially since they are derived from an imperfect automatic classification system. To gather further insights, we turn towards evaluation by humans.

#### **Evaluation by Humans**

We conduct a human evaluation study where raters annotate LM-generated text for toxicity. The results of this study indicate that there is a direct and largely monotonic relation between average human and classifier-based results, and LM toxicity reduces according to human judgment.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335c02f2c1ba641d73d637_unnamed.jpg)We found inter-annotator agreement comparable to other studies measuring toxicity, and that annotating toxicity has aspects that are subjective and ambiguous. For example, we found that ambiguity frequently arose as a result of sarcasm, news-style text about violent behavior, and quoting toxic text (either neutrally or in order to disagree with it).

In addition, we find that automatic evaluation of LM toxicity becomes less reliable once detoxification measures have been applied. While initially coupled very well, for samples with a high (automatic) toxicity score, the link between human ratings and Perspective API scores disappears once we apply and increase the strength of LM toxicity reduction interventions.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335c127ba0ef7c524b59da_unnamed%20(1).jpg)Further manual inspection also reveals that false positive texts mention some identity terms at disproportionate frequencies. For example, for one detoxified model, we observe that within the high automatic toxicity bucket, 30.2% of texts mention the word “gay”, reflecting previously observed biases in automatic toxicity classifiers (which the community is already [working on](https://research.google/pubs/pub46743/) improving). Together, these findings suggest that when judging LM toxicity, a reliance on automatic metrics alone could lead to potentially misleading interpretations.

#### **Unintended Consequences of Detoxification**

We further study possible unintended consequences resulting from the LM toxicity reduction interventions. For detoxified language models, we see a marked increase in the language modeling loss, and this increase correlates with the strength of the detoxification intervention. However, the increase is larger on documents that have higher automatic toxicity scores, compared to documents with lower toxicity scores. At the same time, in our human evaluations we did not find notable differences in terms of grammar, comprehension, and in how well the style of prior conditioning text is preserved.

Another consequence of detoxification is that it can disproportionately reduce the ability of the LM to model texts related to certain identity groups *(i.e. topic coverage)*, and also text by people from different identity groups and with different dialects *(i.e. dialect coverage)*. We find that there is a larger increase in the language modeling loss for text in African-American English (AAE) when compared to text in White-Aligned English.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335c28e94a8979eeb0abf9_unnamed%20(2).jpg)We see similar disparities in LM-loss degradation for text related to female actors when compared to text about male actors. For text about certain ethnic subgroups (such as Hispanic American), the degradation in performance is again relatively higher when compared to other subgroups.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335c313ff32a5db0279c02_unnamed%20(3).jpg)#### **Takeaways**

Our experiments on measuring and mitigating language model toxicity provide us valuable insights into potential next steps towards reducing toxicity-related language model harms.

From our automated and human evaluation studies, we find that existing mitigation methods are indeed very effective at reducing automatic toxicity metrics, and this improvement is largely matched with reductions in toxicity as judged by humans. However, we might have reached an exhaustion point for the use of automatic metrics in LM toxicity evaluation: after the application of toxicity reduction measures, the majority of remaining samples with high automatic toxicity scores are not actually judged as toxic by human raters, indicating that automatic metrics become less reliable for detoxified LMs. This motivates efforts towards designing more challenging benchmarks for automatic evaluation, and to consider human judgment for future studies on LM toxicity mitigation.

Further, given the ambiguity in human judgements of toxicity, and noting that judgements can vary across users and applications (e.g. language describing violence, that might otherwise be flagged as toxic, might be appropriate in a news article), future work should continue to develop and adapt the notion of toxicity for different contexts, and refine it for different LM applications. We hope the list of phenomena which we found annotator disagreement for is helpful in this regard.

Finally, we also noticed unintended consequences of LM toxicity mitigation, including a deterioration in LM loss, and an unintended amplification of social biases - measured in terms of topic and dialect coverage - potentially leading to decreased LM performance for marginalized groups. Our findings suggest that alongside toxicity, it is key for future work to not rely on just a single metric, but to consider an “ensemble of metrics” which capture different issues. Future interventions, such as further reducing bias in toxicity classifiers will potentially help prevent trade-offs like the ones we observed, enabling safer language model use.

Johannes Welbl
Mia Glaese
Jonathan Uesato
Sumanth Dathathri
John Mellor
Lisa Anne Hendricks
Kirsty Anderson *
Pushmeet Kohli
Ben Coppin
Po-Sen Huang


Enabling high-accuracy protein structure prediction at the proteome scale

#### The AlphaFold method

Many novel machine learning innovations contribute to AlphaFold’s current level of accuracy. We give a high-level overview of the system below; for a technical description of the network architecture see our AlphaFold [methods paper](https://www.nature.com/articles/s41586-021-03819-2) and especially its extensive Supplementary Information.

The AlphaFold network consists of two main stages. Stage 1 takes as input the amino acid sequence and a multiple sequence alignment (MSA). Its goal is to learn a rich “pairwise representation” that is informative about which residue pairs are close in 3D space.

Stage 2 uses this representation to directly produce atomic coordinates by treating each residue as a separate object, predicting the rotation and translation necessary to place each residue, and ultimately assembling a structured chain. The design of the network draws on our intuitions about protein physics and geometry, for example, in the form of the updates applied and in the choice of loss.

Interestingly, we can produce a 3D structure based on the representation at intermediate layers of the network. The resulting “trajectory” videos show how AlphaFold’s belief about the correct structure develops during inference, layer by layer. Typically a hypothesis emerges after the first few layers followed by a lengthy process of refinement, although some targets require the full depth of the network to arrive at a good prediction.

![](https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/enabling-high-accuracy-protein-structure-prediction-at-the-proteome-scale/fig_1.gif)
![](https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/enabling-high-accuracy-protein-structure-prediction-at-the-proteome-scale/fig_2.gif)
![](https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/enabling-high-accuracy-protein-structure-prediction-at-the-proteome-scale/fig_3.gif)

Predicted structure for the CASP14 targets T1044, T1024 and T1064 at successive layers of the network. Structures are colored by residue number and the counter shows the current layer.
#### Accuracy and confidence

AlphaFold was stringently assessed in the [CASP14](https://predictioncenter.org/casp14/zscores_final.cgi) experiment, in which participants blindly predict protein structures that have been solved but not yet made public. The method achieved high accuracy in a majority of cases, with an average 95% RMSD-Cα to the experimental structure of less than 1Å. In our papers, we further evaluate the model on a much larger set of recent PDB entries. Among the findings are strong performance on large proteins and good side chain accuracy where the backbone is well-predicted.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335d9cade0573a437b9e08_unnamed.jpg)AlphaFold’s CASP14 accuracy relative to other methods. RMSD-Cα based on the best-predicted 95% of residues for each target.An important factor in the utility of structure predictions is the quality of the associated confidence measures. Can the model identify the parts of its prediction likely to be reliable? We have developed two confidence measures on top of the AlphaFold network to address this question.

The first is pLDDT ([predicted lDDT-Cα](https://doi.org/10.1093/bioinformatics/btt473)), a per-residue measure of local confidence on a scale from 0 - 100. pLDDT can vary dramatically along a chain, enabling the model to express high confidence on structured domains but low confidence on the linkers between them, for example. In our [paper](https://www.nature.com/articles/s41586-021-03828-1), we present evidence that some regions with low pLDDT may be unstructured in isolation; either intrinsically disordered or structured only in the context of a larger complex. Regions with pLDDT < 50 should not be interpreted except as a possible disorder prediction.

The second metric is PAE (Predicted Aligned Error), which reports AlphaFold’s expected position error at residue x, when the predicted and true structures are aligned on residue y. This is useful for assessing confidence in global features, especially domain packing. For residues x and y drawn from two different domains, a consistently low PAE at (x, y) suggests AlphaFold is confident about the relative domain positions. Consistently high PAE at (x, y) suggests the relative positions of the domains should not be interpreted. The general approach used to produce PAE can be adapted to predict a variety of superposition-based metrics, including [TM-score](https://doi.org/10.1002/prot.20264) and [GDT](https://doi.org/10.1093/nar/gkg571).

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335db2dd9d1c2cb07aa2b0_unnamed%20(2).jpg)Per-residue confidence (pLDDT) and Predicted Aligned Error (PAE) for two example proteins (P54725, Q5VSL9). Both have confident individual domains, but the latter also has confident relative domain positions. Note: Q5VSL9 was solved after this prediction was produced.To emphasise, AlphaFold models are ultimately predictions: while often highly accurate they will sometimes be in error. Predicted atomic coordinates should be interpreted carefully, and in the context of these confidence measures.

#### Open sourcing

Alongside our [method paper](https://www.nature.com/articles/s41586-021-03819-2), we have made the AlphaFold source code available on [GitHub](https://github.com/deepmind/alphafold). This includes access to a trained model and a script for making predictions on novel input sequences. We believe this is an important step that will enable the community to use and build on our work. The easiest way to fold a single new protein with AlphaFold is to use our [Colab notebook](https://bit.ly/alphafoldcolab).

The open source code is an updated version of our CASP14 system based on the [JAX framework](https://github.com/google/jax), and it achieves equally high accuracy. It also incorporates some recent performance improvements. AlphaFold’s speed has always depended heavily on the input sequence length, with short proteins taking minutes to process and only very long proteins running into hours. Once the MSA has been assembled, the open source version can now predict the structure of a 400 residue protein in just over a minute of GPU time on a V100.

#### Proteome scale and AlphaFold DB

AlphaFold’s fast inference times allow the method to be applied at whole-proteome scale. In our [paper](https://www.nature.com/articles/s41586-021-03828-1), we discuss AlphaFold’s predictions for the human proteome. However, we have since generated predictions for the reference proteomes of a number of [model organisms, pathogens and economically significant species](https://alphafold.ebi.ac.uk/download), and large scale prediction is now routine. Interestingly, we observe a difference in the pLDDT distribution between species, with generally higher confidence on bacteria and archaea and lower confidence on eukaryotes, which we hypothesize may be related to the prevalence of disorder in these proteomes.

No single research group can fully explore such a large dataset, and so we partnered with [EMBL-EBI](https://www.ebi.ac.uk/) to make the predictions freely available via the [AlphaFold DB](https://alphafold.ebi.ac.uk/). Each prediction can be viewed alongside the confidence metrics described above. A bulk download is also provided for each species, and all data is covered by a CC-BY-4.0 license (making it freely available for both academic and commercial use). We are extremely grateful to EMBL-EBI for their work with us to develop this new resource. Over the course of the coming months we plan to expand the dataset to cover the over 100 million proteins in [UniRef90](https://www.uniprot.org/uniref/?query=&fil=identity:0.9).

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335e12dbf2046bb6893f14_unnamed%20(3).jpg)Example: AlphaFold DB predictions from a variety of organisms.![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335de67ba0ef82024c93af_unnamed%20(1).jpg)Distribution of per-residue confidence for 14 species; left to right: bacteria / archaea, animals, and protists.In AlphaFold DB, we have chosen to share predictions of full protein chains up to 2700 amino acids in length, rather than cropping to individual domains. The rationale is that this avoids missing structured regions that have yet to be annotated. It also provides context from the full amino acid sequence, and allows the model to attempt a domain packing prediction. AlphaFold’s intra-domain accuracy was more extensively evaluated in CASP14 and is expected to be higher than its inter-domain accuracy. However, AlphaFold was the top ranked method in the inter-domain assessment, and we expect it to produce an informative prediction in some cases. We encourage users to view the PAE plot to determine whether domain placement is likely to be meaningful.

#### Future work

We are excited about the future for computational structural biology. There remain many important topics to address: predicting the structure of complexes, incorporating non-protein components, and capturing dynamics and the response to point mutations. The development of network architectures like AlphaFold that excel at the task of understanding protein structure is a cause for optimism that we can make progress on related problems.

We see AlphaFold as a complementary technology to experimental structural biology. This is perhaps best illustrated by its role in helping to solve experimental structures, through molecular replacement and docking into cryo-EM volumes. Both applications can accelerate existing research, saving months of effort. From a bioinformatics perspective, AlphaFold’s speed enables the generation of predicted structures on a massive scale. This has the potential to unlock new avenues of research, by supporting structural investigations of the contents of large sequence databases.

Ultimately, we hope AlphaFold will prove a useful tool for illuminating protein space, and we look forward to seeing how it is applied in the coming months and years.

We would love to hear your feedback and understand how AlphaFold and the AlphaFold DB have been useful in your research. Share your stories at [alphafold@deepmind.com](mailto:alphafold@deepmind.com).

Kathryn Tunyasuvunakool
Jonas Adler
Zachary Wu
Tim Green
Michal Zielinski
Augustin Žídek
Alex Bridgland
Andrew Cowie
Clemens Meyer
Agata Laydon
Sameer Velanka *
Gerard J Kleywegt *
Alex Bateman *
Richard Evans
Alexander Pritzel
Michael Figurnov
Olaf Ronneberger
Russ Bates
Simon A. A. Kohl
Anna Potapenko
Andrew J Ballard
Bernardino Romera-Paredes
Stanislav Nikolov
Rishub Jain
Ellen Clancy
David Reiman
Stig Petersen
Andrew Senior
Koray Kavukcuoglu
Ewan Birney *
Pushmeet Kohli
John Jumper
Demis Hassabis


Melting Pot: an evaluation suite for multi-agent reinforcement learning

Technology deployed in the real world inevitably faces unforeseen challenges. These challenges arise because the environment where the technology was developed differs from the environment where it will be deployed. When a technology transfers successfully we say it generalises. In a *multi-agent system*, such as autonomous vehicle technology, there are two possible sources of generalisation difficulty: (1) physical-environment variation such as changes in weather or lighting, and (2) social-environment variation: changes in the behaviour of other interacting individuals. Handling social-environment variation is at least as important as handling physical-environment variation, however it has been much less studied.

As an example of a social environment, consider how self-driving cars interact on the road with other cars. Each car has an incentive to transport its own passenger as quickly as possible. However, this competition can lead to poor coordination (road congestion) that negatively affects everyone. If cars work cooperatively, more passengers might get to their destination more quickly. This conflict is called a *social dilemma.*

However, not all interactions are social dilemmas. For instance, there are *synergistic* interactions in open-source software, there are *zero-sum* interactions in sports, and *coordination problems* are at the core of supply chains. Navigating each of these situations requires a very different approach.

Multi-agent reinforcement learning provides tools that allow us to explore how artificial agents may interact with one another and with unfamiliar individuals (such as human users). This class of algorithms is expected to perform better when tested for their social generalisation abilities than others. However, until now, there has been no systematic evaluation benchmark for assessing this.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6232127f110f7a5984f6e7a6_fig_1.jpg)Blue: focal populations of trained agents, Red: background population of pre-trained botsHere we introduce Melting Pot, a scalable evaluation suite for multi-agent reinforcement learning. Melting Pot assesses generalization to novel social situations involving both familiar and unfamiliar individuals, and has been designed to test a broad range of social interactions such as: cooperation, competition, deception, reciprocation, trust, stubbornness and so on. Melting Pot offers researchers a set of 21 MARL “substrates” (multi-agent games) on which to train agents, and over 85 unique test scenarios on which to evaluate these trained agents. The performance of agents on these held-out test scenarios quantifies whether agents:

* Perform well across a range of social situations where individuals are interdependent,
* Interact effectively with unfamiliar individuals not seen during training,
* Pass a universalisation test: answering positively to the question "what if everyone behaved like that?''

The resulting score can then be used to rank different multi-agent RL algorithms by their ability to *generalise* to novel social situations.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623212cd392aa8e603bdfd6e_fig_2.jpg)We hope Melting Pot will become a standard benchmark for multi-agent reinforcement learning. We plan to maintain it, and will be extending it in the coming years to cover more social interactions and generalisation scenarios.

Learn more from our [GitHub page](https://github.com/deepmind/meltingpot).

Joel Z. Leibo
Edgar Duéñez-Guzmán
Alexander Vezhnevets
John Agapiou
Peter Sunehag
Raphael Koster
Jayd Matyas
Charlie Beattie
Igor Mordatch *
Thore Graepel


Data, Architecture, or Losses: What Contributes Most to Multimodal Transformer Success?

The ability to ground language to vision is a fundamental aspect of real-world AI systems; it is useful across a range of tasks (*e.g.*, visual question answering) and applications (*e.g.*, generating descriptions for visually impaired). Multimodal models (pre-trained on image-language pairs) aim to address this grounding problem. A recent family of models, multimodal transformers (e.g., Lu et al., 2019; Chen et al., 2020; Tan and Bansal, 2019; Li et al., 2020), have achieved state-of-the-art performance in a range of multimodal benchmarks, suggesting that the joint-encoder transformer architecture is better suited for capturing the alignment between image-language pairs than previous approaches (such as dual encoders).

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320e2a337d937f992f130e_fig_1.jpg)In particular, compared to the dual-encoder architecture where there is no cross-talk between the modalities, multimodal transformers (joint encoders) are more sample efficient. In the plot below, we see that, when tested on zero-shot image retrieval, an existing multimodal transformer (UNITER) performs similar to a large-scale dual encoder (CLIP) which is trained on 100 times more data.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320e43fb63b51eb51e2185_fig_2.jpg)BOW-DE: Miech & Alayrac et al. Arxiv 2021, MMT: Hendricks et al. TACL 2021, UNITER: Chen et al. ECCV 2020, CLIP: Radford et al. Arxiv 2021, ALIGN: Jia et al. Arxiv 2021In this work, we examine what aspects of multimodal transformers – attention, losses, and pretraining data – are important in their success at multimodal pretraining. We find that Multimodal attention, where both language and image transformers attend to each other, is crucial for these models’ success. Models with other types of attention (even with more depth or parameters) fail to achieve comparable results to shallower and smaller models with multimodal attention. Moreover, comparable results can be achieved without the image (masked region modelling) loss originally proposed for multimodal transformers. This suggests that our current models are not tapping into the useful signal in the image modality, presumably because of the image loss formulation.

We also study different properties of multimodal datasets such as their size and the degree to which the language describes its corresponding image (noisiness). We find that a dataset’s size does not always predict multimodal transformers’ performance; its noise level and language similarity to the evaluation task are both important contributing factors. These suggest curating less noisy image–text datasets to be important despite the current trend of harvesting noisy datasets from the web.

Overall, our analysis shows that multimodal transformers are stronger than dual encoder architecture (given the same amount of pretraining data), mainly due to the cross-talk through multimodal attention. However, there are still many open problems when designing multimodal models, including better losses for the image modality and robustness to dataset noise.

Aida Nematzadeh
Lisa Anne Hendricks
Jean-baptiste Alayrac
Rosalia Schneider
John Mellor


Imitating Interactive Intelligence

Two questions must be answered at the outset of any artificial intelligence research. What do we want AI systems to do? And how will we evaluate when we are making progress toward this goal? Alan Turing, in his seminal paper describing the Turing Test, which he more modestly named the imitation game, argued that for a certain kind of AI, these questions may be one and the same. Roughly, if an AI’s behaviour resembles human-like intelligence when a person interacts with it, then the AI has passed the test and can be called intelligent. An AI that is designed to interact with humans should be tested via interaction with humans.

At the same time, interaction is not just a test of intelligence but also the point. For AI agents to be generally helpful, they should assist us in diverse activities and communicate with us naturally. In science fiction, the vision of robots that we can speak to is commonplace. And intelligent digital agents that can help accomplish large numbers of tasks would be eminently useful. To bring these devices into reality, we therefore must study the problem of how to create agents that can capably interact with humans and produce actions in a rich world.

Building agents that can interact with humans and the world poses a number of important challenges. How can we provide appropriate learning signals to teach artificial agents such abilities? How can we evaluate the performance of the agents we develop, when language itself is ambiguous and abstract? As the wind tunnel is to the design of the airplane, we have created a virtual environment for researching how to make interacting agents.

We first create a simulated environment, the Playroom, in which virtual robots can engage in a variety of interesting interactions by moving around, manipulating objects, and speaking to each other. The Playroom’s dimensions can be randomised as can its allocation of shelves, furniture, landmarks like windows and doors, and an assortment of children's toys and domestic objects. The diversity of the environment enables interactions involving reasoning about space and object relations, ambiguity of references, containment, construction, support, occlusion, partial observability. We embedded two agents in the Playroom to provide a social dimension for studying joint intentionality, cooperation, communication of private knowledge, and so on.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320d57fb7cb612609e1378_fig_1.jpg)Agents interacting in the Playroom. The blue agent instructs the yellow agent to “Put the helicopter into the box.”![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320d6327179e737d345d3f_fig_2.jpg)The configuration of the Playroom is randomised to create diversity in data collection.We harness a range of learning paradigms to build agents that can interact with humans, including imitation learning, reinforcement learning, supervised, and unsupervised learning. As Turing may have anticipated in naming “the imitation game,” perhaps the most direct route to create agents that can interact with humans is through imitation of human behaviour. Large datasets of human behaviour along with algorithms for imitation learning from those data have been instrumental for making agents that can interact with textual language or play games. For grounded language interactions, we have no readily available, pre-existing data source of behaviour, so we created a system for eliciting interactions from human participants interacting with each other. These interactions were elicited primarily by prompting one of the players with a cue to improvise an instruction about, e.g., “Ask the other player to position something relative to something else.” Some of the interaction prompts involve questions as well as instructions, like “Ask the other player to describe where something is.” In total, we collected more than a year of real-time human interactions in this setting.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320d74337d9337ba2f12ed_fig_3.jpg)Our agents each consume images and language as inputs and produce physical actions and language actions as outputs. We built reward models with the same input specifications.![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320d836e04ba63f81c6e26_fig_4.jpg)Left: Over the course of a 2 minute interaction, the two players (setter & solver) move around, look around, grab and drop objects, and speak. Right: The setter is prompted to “Ask the other player to lift something.” The setter instructs the solver agent to “Lift the plane which is in front of the dining table”. The solver agent finds the correct object and completes the task.Imitation learning, reinforcement learning, and auxiliary learning (consisting of supervised and unsupervised representation learning) are integrated into a form of interactive self-play that is crucial to create our best agents. Such agents can follow commands and answer questions. We call these agents “solvers.” But our agents can also provide commands and ask questions. We call these agents “setters.” Setters interactively pose problems to solvers to produce better solvers. However, once the agents are trained, humans can play as setters and interact with solver agents.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320d98f724cb999b9eb83d_fig_5.jpg)From human demonstrations we train policies using a combination of supervised learning (behavioural cloning), inverse RL to infer reward models, and forward RL to optimise policies using the inferred reward model. We use semi-supervised auxiliary tasks to help shape the representations of both the policy and reward models.![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320dc06a323a2219564a9f_fig_6.jpg)The setter agent asks the solver agent to “Take the white robot and place it on the bed.” The solver agent finds the robot and accomplishes the task. The reward function learned from demonstrations captures key aspects of the task (blue), and gives less reward (grey) when the same observations are coupled with the counterfactual instruction, “Take the red robot and place it on the bed.”Our interactions cannot be evaluated in the same way that most simple reinforcement learning problems can. There is no notion of winning or losing, for example. Indeed, communicating with language while sharing a physical environment introduces a surprising number of abstract and ambiguous notions. For example, if a setter asks a solver to put something near something else, what exactly is “near”? But accurate evaluation of trained models in standardised settings is a linchpin of modern machine learning and artificial intelligence. To cope with this setting, we have developed a variety of evaluation methods to help diagnose problems in and score agents, including simply having humans interact with agents in large trials.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320dd4c8ddbba2a8d4c9ac_fig_7.jpg)Humans evaluated the performance of agents and other humans in completing instructions in the Playroom on both instruction-following and question-answering tasks. Randomly initialised agents were successful ~0% of the time. An agent trained with supervised behavioural cloning alone (B) performed somewhat better, at ~10-20% of the time. Agents trained with semi-supervised auxiliary tasks as well (B·A) performed better. Those trained with supervised, semi-supervised, and reinforcement learning using interactive self-play were judged to perform best (BG·A & BGR·A).A distinct advantage of our setting is that human operators can set a virtually infinite set of new tasks via language, and quickly understand the competencies of our agents. There are many tasks that they cannot cope with, but our approach to building AIs offers a clear path for improvement across a growing set of competencies. Our methods are general and can be applied wherever we need agents that interact with complex environments and people.

Josh Abramson
Arun Ahuja
Arthur Brussee
Federico Carnevale
Mary Cassin
Stephen Clark
Andrew Dudzik
Petko Georgiev
Aurelia Guy
Tim Harley
Felix Hill
Alden Hung
Zac Kenton
Jessica Landon
Timothy Lillicrap
Kory W. Mathewson
Alistair Muldal
Adam Santoro
Nikolay Savinov
Vikrant Varma
Gregory Wayne
Nathaniel Wong
Chen Yan
Rui Zhu


Using Unity to Help Solve Intelligence

#### A wide range of environments

In the pursuit of artificial general intelligence (AGI), we seek to create agents that can achieve goals in a wide range of environments. As our agents master the environments we create, we must continually create new environments that probe as-yet-untested cognitive abilities.

Games have always provided a challenge for artificial intelligence (AI) research, most famously board games such as backgammon, chess, and Go. Video games such as Space Invaders, Quake III Arena, Dota 2 and StarCraft II have also more recently become popular for AI research. Games are ideal because they have a clear measure of success, allowing progress to be reviewed empirically and to be directly benchmarked against humans.

As AGI research progresses, so too does the research community’s interest in more complex games. At the same time, the engineering efforts needed for transforming individual video games into research environments become hard to manage. Increasingly, general-purpose game engines become the most scalable way to create a wide range of interactive environments.

#### General-purpose game engines

Much AGI research has already happened in game engines such as [Project Malmo](https://www.microsoft.com/en-us/research/project/project-malmo/), based on Minecraft; [ViZDoom](http://vizdoom.cs.put.edu.pl/), based on Doom; and [DeepMind Lab](https://github.com/deepmind/lab), based on Quake III Arena. These engines can be scripted to quickly create new environments – and since many were written for older hardware, they’re able to run extremely fast on modern hardware, eliminating the environment as a performance bottleneck.

But these game engines are missing some important features. DeepMind Lab, for example, is excellent for learning navigation but poor for learning common sense notions like how objects move and interact with each other.

#### Unity

At DeepMind we use Unity, a flexible and feature-rich game engine. Unity’s realistic physics simulation allows agents to experience an environment more closely grounded in the real world. The modern rendering pipeline provides more subtle visual clues such as realistic lighting and shadows. Unity scripts are written in C#, which is easy to read, and unlike with bespoke engines, provides access to all game-engine features. Multiplatform support lets us run environments at home on our laptops or at scale on Google’s data-centres. Finally, as the Unity engine continues to evolve, we can future-proof ourselves without expending a large amount of our own engineering time.

Unity includes a ready-to-use machine learning toolkit called [ML-Agents](https://unity.com/products/machine-learning-agents) that focuses on simplifying the process of making an existing game available as a learning environment. DeepMind focuses on constructing a wide variety of heterogeneous environments which are run at scale, and as such we instead use dm\_env\_rpc (see below).

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62320bed766cbabbba55a2e1_fig_1.jpg)Screen captures of Unity environments created at DeepMind#### Differences from conventional games

Traditional video games render themselves in real-time: one second on-screen is equal to one second in a simulation. But to AI researchers, a game is just a stream of data. Games can often be processed much more quickly than in real-time, and there’s no problem if the game speed varies wildly from moment to moment.

Additionally, many reinforcement learning algorithms scale with multiple instances. That is, one AI can play thousands of games simultaneously and learn from them all at once.

Because of this, we optimise for throughput instead of latency. That is, we update our games as many times as we can and don’t worry about generating those updates at a consistent rate. We run multiple games on a single computer, with one game per processor core. Stalls caused by features such as garbage collection - a common headache for traditional game makers - are not a concern to us as long as the game generally runs quickly.

#### Containerisation and dm\_env\_rpc

Games output images, text, and sound for the player to see and hear, and also take input commands from a game controller of some kind. The structure of this data is important for AI researchers. For example, text is normally presented separately instead of being drawn onto the screen. Since flexibility in this data format is so important, we created a new open-source library called [*dm\_env\_rpc*](https://github.com/deepmind/dm_env_rpc), which functions as the boundary between environments and agents.

By using dm\_env\_rpc, we can containerise our environments and release them publicly. Containerisation means using technology like [Docker](https://www.docker.com/) to package precompiled environment binaries. Containerisation allows our research to be independently verified. It’s a more reliable and convenient way to reproduce experiments than open sourcing, which can be confused by compiler or operating system differences. For more details on how we containerise an environment, please see our work on [dm\_memorytasks.](https://github.com/deepmind/dm_memorytasks)

Simon Carter
Manuel Sanchez
Ricardo Barreira
Seb Noury
Keith Anderson
Jay Lemmon
Jonathan Coe
Piotr Trochim
Tom Handley
Adrian Bolton


![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6231efbede9e9af7f611ff68_fig%201.gif)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6231efcc337d93e52e2e23b2_fig%202.gif)Many of the successes of RL rely heavily on repeated online interactions of an agent with an environment, which we call online RL. Despite its success in simulation, the uptake of RL for real-world applications has been limited. Power plants, robots, healthcare systems, or self-driving cars are expensive to run and inappropriate controls can have dangerous consequences. They are not easily compatible with the crucial idea of exploration in RL and the data requirements of online RL algorithms. Nevertheless, most real-world systems produce large amounts of data as part of their normal operation, and the goal of offline RL to learn a policy directly from that logged data without interacting with the environment.

Offline RL methods (e.g Agarwal et al., 2020; Fujimoto et al., 2018) have shown promising results on well-known benchmark domains. However, non-standardised evaluation protocols, differing datasets, and ack of baselines make algorithmic comparisons difficult. Nevertheless, some important properties of potential real-world application domains such as partial observability, high-dimensional sensory streams (i.e., images), diverse action spaces, exploration problems, non-stationarity, and stochasticity, are underrepresented in the current offline RL literature.

[INSERT GIF + CAPTION]

We introduce a novel collection of task domains and associated datasets together with a clear evaluation protocol. We include widely-used domains such as the DM Control Suite (Tassa et al., 2018) and Atari 2600 games (Bellemare et al., 2013), but also domains that are still challenging for strong online RL algorithms such as real-world RL (RWRL) suite tasks (Dulac-Arnold et al., 2020) and DM Locomotion tasks (Heess et al., 2017; Merel et al., 2019a,b, 2020). By standardizing the environments, datasets, and evaluation protocols, we hope to make research in offline RL more reproducible and accessible. We call our suite of benchmarks “RL Unplugged”, because offline RL methods can use it without any actors interacting with the environment. Our paper offers four main contributions: (i) a unified API for datasets (ii) a varied set of environments (iii) clear evaluation protocols for offline RL research, and (iv) reference performance baselines.

##### RL Unplugged: Benchmarks for Offline Reinforcement Learning

Caglar Gülçehre
Ziyu Wang
Alexander Novikov
Tom Le Paine
Sergio Gómez Colmenarejo
K Zolna
Rishabh Agarwal*
Josh Merel
Daniel Mankowitz
Cosmin Paduraru
Gabriel Dulac-Arnold*
Jerry Li
Mohammad Norouzi *
Matt Hoffman
Ofir Nachum *
George Tucker *
Nicolas Heess
Nando de Freitas


dm_control: Software and Tasks for Continuous Control

#### Overview

A public colab notebook with a tutorial for dm\_control software is available [here](https://colab.sandbox.google.com/github/deepmind/dm_control/blob/master/tutorial.ipynb).

##### Infrastructure

* An autogenerated MuJoCo Python wrapper provides full access to the underlying engine.
* PyMJCF is a Document Object Model, wherein a hierarchy of Python *Entity* objects corresponds to MuJoCo model elements.
* Composer is the high-level “game engine” which streamlines the composing of Entities into scenes and the defining observations, rewards, terminations and general game logic.
* The Locomotion framework introduces several abstract Composer entities such as the Arena and Walker, facilitating locomotion-like tasks.

##### Environments

* The [Control Suite](https://www.youtube.com/watch?v=rAai4QzcYbs), including a new [quadruped](https://www.youtube.com/watch?v=RhRLjbb7pBE&t=5s) and [dog](https://www.youtube.com/watch?v=i0_OjDil0Fg) environment.
* Several locomotion tasks, including soccer.
* Single arm robotic manipulation tasks using snap-together bricks.

#### Highlights

##### Named Indexing

Exploiting MuJoCo’s support of *names* for all model elements, we allow strings to index and slice into arrays. So instead of writing:

"fingertip\_height = physics.data.geom\_xpos[7, 2]"

...using obscure, fragile numerical indexing, you can write:

"fingertip\_height = physics.named.data.geom\_xpos['fingertip', 'z']"

leading to a much more robust, readable codebase.

#### PyMJCF

The PyMJCF library creates a Python object hierarchy with 1:1 correspondence to a MuJoCo model. It introduces the attach() method which allows models to be attached to one another. For example, in our tutorial we create procedural multi-legged creatures by attaching legs to bodies and creatures to the scene.

##### Composer

Composer is the “game engine“ framework, which defines a particular order of runtime function calls, and abstracts the affordances of *reward*, *termination* and *observation*. These abstractions allowed us to create useful submodules:

composer.Observable: An abstract observation wrapper which can add noise, delays, buffering and filtering to any sensor.

composer.Variation: A set of tools for randomising simulation quantities, allowing for agent robustification and sim-to-real via model variation.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6231ee766a323a1793376c3b_diagram3.svg)Diagram showing the life-cycle of Composer callbacks. Rounded rectangles represent callbacks that Tasks and Entities may implement. Blue rectangles represent built-in Composer operations.#### Locomotion

The Locomotion framework introduced the abstractions:

Walker: A controllable entity with common locomotion-related methods, like projection of vectors into an egocentric frame.

Arena: A self-scaling randomised scene, in which the walker can be placed and given a task to perform.

For example, using just 4 function calls, we can instantiate a humanoid walker, a WallsCorridor arena and combine them in a RunThroughCorridor task.

#### New Control Suite domains

##### Quadruped

* A generic quadruped domain with a passively stable body.
* Several pure locomotion tasks (e.g. walk, run).
* An escape task requiring rough terrain navigation.
* A fetch task requiring ball dribbling.

##### Dog

* An elaborate model based on a skeleton commissioned from [leo3Dmodels](https://www.turbosquid.com/Search/Artists/leo3Dmodels).
* A challenging ball-fetching task that requires precision grasping with the mouth.

##### Showcase

A fast-paced montage of dm\_control based tasks from DeepMind:

Yuval Tassa
Saran Tunyasuvunakool
Alistair Muldal
Yotam Doron
Siqi Liu
Steven Bohez
Josh Merel
Tom Erez
Timothy Lillicrap
Nicolas Heess


Acme: A new framework for distributed reinforcement learning

Overall, the high-level goals of Acme are as follows:

1. To enable the reproducibility of our methods and results  — this will help clarify what makes an RL problem hard or easy, something that is seldom apparent.
2. To simplify the way we (and the community at large) design new algorithms — we want that next RL agent to be easier for everyone to write!
3. To enhance the readability of RL agents — there should be no hidden surprises when transitioning from a paper to code.

In order to enable these goals, the design of Acme also bridges the gap between large-, medium-, and small-scale experiments. We have done so by carefully thinking about the design of agents at many different scales.

At the highest level, we can think of Acme as a classical RL interface (found in any introductory RL text) which connects an actor (i.e. an action-selecting agent) to an environment. This actor is a simple interface which has methods for selecting actions, making observations, and updating itself. Internally, learning agents further split the problem up into an “acting” and a “learning from data” component. Superficially, this allows us to re-use the acting portions across many different agents. However, more importantly this provides a crucial boundary upon which to split and parallelize the learning process. We can even scale down from here and seamlessly attack the batch RL setting where there exists *no environment* and only a fixed dataset. Illustrations of these different levels of complexity are shown below:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228c6b6b755e529e96ea6c6_Fig%201.gif)This design allows us to easily create, test, and debug novel agents in small-scale scenarios before scaling them up — all while using the same acting and learning code. Acme also provides a number of useful utilities from checkpointing, to snapshotting, to low-level computational helpers. These tools are often the unsung heroes of any RL algorithm, and in Acme we strive to keep them as simple and understandable as possible.

To enable this design Acme also makes use of [Reverb](https://deepmind.com/research/open-source/Reverb): a novel, efficient data storage system purpose built for machine learning (and reinforcement learning) data. Reverb is primarily used as a system for experience replay in distributed reinforcement learning algorithms, but it also supports other data structure representations such as FIFO and priority queues. This allows us to use it seamlessly for on- and off-policy algorithms. Acme and Reverb were designed from the beginning to play nicely with one another, but Reverb is also fully usable on its own, so go check it out!

Along with our infrastructure, we are also releasing single-process instantiations of a number of agents we have built using Acme. These run the gamut from continuous control (D4PG, MPO, etc.), discrete Q-learning (DQN and R2D2), and more. With a minimal number of changes — by splitting across the acting/learning boundary — we can run these same agents in a distributed manner. Our first release focuses on single-process agents as these are the ones mostly used by students and research practitioners.

We have also carefully benchmarked these agents on a number of environments, namely the [control suite](https://deepmind.com/research/publications/deepmind-control-suite), [Atari](https://github.com/mgbellemare/Arcade-Learning-Environment), and [bsuite](https://deepmind.com/research/open-source/bsuite).

###### Playlist of videos showing agents trained using Acme framework

While additional results are readily available in our [paper](https://arxiv.org/abs/2006.00979), we show a few plots comparing the performance of a single agent (D4PG) when measured against both actor steps and wall clock time for a continuous control task. Due to the way in which we limit the rate at which data is inserted into replay — refer to the paper for a more in-depth discussion — we can see roughly the same performance when comparing the rewards an agent receives versus the number of interactions it has taken with the environment (actor steps). However, as the agent is further parallelised we see gains in terms of how fast the agent is able to learn. On relatively small domains, where the observations are constrained to small feature spaces, even a modest increase in this parallelisation (4 actors) results in an agent that takes under half the time to learn an optimal policy:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228c7165eaff527cb3d52f4_Fig%202.jpg)But for even more complex domains where the observations are images that are comparatively costly to generate we see much more extensive gains:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228c723cd9c1976c2f08330_Fig%203.jpg)And the gains can be even bigger still for domains such as Atari games where the data is more expensive to collect and the learning processes generally take longer. However, it is important to note that these results share the same acting and learning code between both the distributed and non-distributed setting. So it is perfectly feasible to experiment with these agents and results at a smaller scale — in fact this is something we do all the time when developing novel agents!

For a more detailed description of this design, along with further results for our baseline agents, see our [paper](https://arxiv.org/abs/2006.00979). Or better yet, take a look at our [GitHub repository](https://github.com/deepmind/acme) to see how you can start using Acme to simplify your own agents!

Matt Hoffman
Bobak Shahriari
John Aslanides
Gabriel Barth-Maron
Feryal Behbahani
Tamara Norman
Abbas Abdolmaleki
Albin Cassirer
Fan Yang
Kate Baumli
Sarah Henderson
Alex Novikov
Sergio Gómez Colmenarejo
Serkan Cabi
Caglar Gülçehre
Tom Le Paine
Andrew Cowie
Ziyu Wang
Bilal Piot
Nando de Freitas


Simple Sensor Intentions for Exploration

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228c5d829a96fa1b5f53db5_Fig%201.gif)By simple color-masking, high-level image statistics can be derived. Rewarding an agent for deliberately changing these statistics, leads to diverse exploration and interesting behavior like, for example, grasping or lifting objects.#### Example Skills

Learned from scratch and from pixels and proprioception only. The external rewards are sparse and Simple Sensor Intentions (SSIs) are used as the only auxiliary tasks.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228c5ead7858131e4ad7dd2_Fig%202.gif)#### Video

Tim Hertweck
Martin Riedmiller
Michael Bloesch
Jost Tobias Springenberg
Noah Siegel
Markus Wulfmeier
Roland Hafner
Nicolas Heess


Learning to Segment Actions from Observation and Narration

Complex tasks that people carry out in the world, for example *making pancakes*, have multiple action steps (e.g., *pouring the mixture, flipping the pancake, removing the pancake*), and are structured. When we observe people carrying out tasks, we recognize where the action steps begin and end (*pouring the mixture* now, *flipping the pancake* later), and distinguish the important steps from the insignificant ones. Identifying important action steps and associating them with intervals of time is known as *action segmentation*, and is a crucial process for human cognition and planning. When people, and in particular, children, learn to segment actions, they rely on a number of cues, including descriptions narrated by the person carrying out the task (“now I’ll stir everything”..) and structural regularities in the task (mixing ingredientstypically happens after adding the ingredients).

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bfca149e456ff57380c2_Fig%201%20copy.jpg)In this work, inspired by how people learn to segment actions, we examine how effective language descriptions and task regularities are in improving systems for action segmentation. Action segmentation is an important first step for processing and cataloguing video: knowing which actions are occurring, and when, makes it easier to search for relevant videos and parts of video from a large, web-scale collection. However, standard, supervised, machine learning methods for predicting action segments in videos would require videos to be annotated with the action segments that occur in them. Since these annotations would be expensive and difficult to collect, we are interested in *weakly-supervised* action segmentation: training without annotated action segments.

We focus on a challenging dataset of instructional videos taken from YouTube [CrossTask, Zhukov et al. 2019], involving everyday household tasks such as cooking and assembling furniture. While these videos are naturally-occurring, they consist of tasks that have some structural regularities across videos, and have language descriptions (transcriptions of the person’s narration), which both provide a noisy source of weak supervision. We develop a flexible unsupervised model for action segmentation that can be trained without action labels, and can optionally use this weak supervision from the *task regularities* and *language descriptions*. Our model, and models from past work, both benefit substantially from both of these sources of supervision, even on top of rich features from state-of-the-art neural action and object classifiers. We also find that generativemodels of the video features typically have better performance than discriminative models on the segmentation task.

Our findings suggest that using language to guide action segmentation is a promising direction for future work, when annotations for the action segments are not available.

Daniel Fried*
Jean-Baptiste Alayrac
Phil Blunsom
Chris Dyer
Stephen Clark
Aida Nematzadeh


Visual Grounding in Video for Unsupervised Word Translation

#### Translating Words Through Unpaired Narrated Videos

The most common approach for machine translation relies on supervision through paired or parallel corpus where each sentence in the source language is paired with its translation in the target language. This is limiting as we do not have access to such a paired corpus for most languages in the world. Interestingly, bilingual children can learn two languages without being exposed to them at the same time. Instead, they can leverage visual similarity across situations: what they observe while hearing "the dog is eating'' on Monday is similar to what they see as they hear "le chien mange'' on Friday.

In this work, inspired by bilingual children, we develop a model that learns to translate words from one language to another by tapping into the visual similarity of situations in which words occur. More specifically, our training dataset consists of disjoint sets of videos narrated in different languages. These videos share similar topics (e.g., cooking pasta or changing a tire); for example, the dataset consists of some videos on how to cook pasta narrated in Korean and a different set of videos on the same topic but in English. Note that the videos in different languages are not *paired*.

Our model leverages the visual similarity of videos by associating videos with their corresponding narrations in a shared embedding space between languages. The model is trained by alternating between videos narrated in one language and those in the second language. Thanks to such a training procedure, and since we share the video representation between both languages, our model learns a joint bilingual-visual space that aligns words in two different languages.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bf8f779b4968925406fe_Fig%201.gif)#### MUVE: improving language only methods with vision

We demonstrate that our method, MUVE (Multilingual Unsupervised Visual Embeddings), can complement existing translation techniques that are trained on unpaired corpus but do not use vision. By doing so, we show that the quality of unsupervised word translation improves, most notably in situations where language-only methods suffer the most, e.g., when: (i) languages are very different (such as English and Korean or English and Japanese), (ii) the initial corpora have different statistics in the two languages, or (iii)  a limited amount of training data is available.

Our findings suggest that using visual data such as videos is a promising direction to improve bilingual translation models when we do not have paired data.

Gunnar Sigurdsson*
Jean-Baptiste Alayrac
Aida Nematzadeh
Lucas Smaira
Mateusz Malinowski
Joao Carreira
Phil Blunsom
Andrew Zisserman


Artificial Intelligence, Values and Alignment

The question of ‘value alignment’ centres upon how to ensure that AI systems are properly aligned with human values. It can be broken down into two parts. The first part is *technical* and focuses on how to encode values or principles in artificial agents, so that they reliably do what they ought to do. The second part is *normative*, and focuses on what values or principles it would be right to encode in AI.

This paper focuses on the second question, paying particular attention to the fact that we live in a pluralistic world where people have a variety of different beliefs about value. Ultimately, I suggest that we need to devise principles for alignment that treat people fairly and command widespread support despite this difference of opinion.

#### Moral considerations

Any new technology generates moral considerations. Yet the task of imbuing artificial agents with moral values becomes particularly important as computer systems operate with greater autonomy and at a speed that ‘increasingly prohibits humans from evaluating whether each action is performed in a responsible or ethical manner’.

The first part of the paper notes that while technologists have an important role to play in building systems that respect and embody human values, the task of selecting appropriate values is not one that can be settled by technical work alone. This becomes clear when we look at the different ways in which value alignment could be achieved, at least within the reinforcement learning paradigm.

One set of approaches try to specify a reward function for an agent that would lead it to promote the right kind of outcome and act in ways that are broadly thought to be ethical. For this approach to succeed, we need to specify appropriate goals for artificial agents and encode them in AI systems – which is far from straightforward. A second family of approaches proceeds differently. Instead of trying to specify the correct reward function for the agent upfront, it looks at ways in which an agent could learn the correct reward from examples of human behavior or human feedback. However, the question then becomes what data or feedback to train the agent on – and how this decision can be justified.

Either way, important normative questions remain.

#### Alignment with what?

A key concern among AI researchers is that the systems they build are properly responsive to human direction and control. Indeed, as Stuart Russell notes, it is important that artificial agents understand the real meaning of the instructions they are given, and that they do not interpret them in an excessively literal way – with the story of King Midas serving as a cautionary tale.

At the same time, there is growing recognition that AI systems may need to go beyond this – and be designed in a way that leads them to do the right thing by default, even in the absence of direct instructions from a human operator.

One promising approach holds that AI should be designed to align with human preferences. In this way, AI systems would learn to avoid outcomes that very few people wanted or desired. However, this approach also has certain weaknesses. Revealed preferences can be irrational or based on false information. They may also be malicious. Furthermore, preferences are sometimes ‘adaptive’: people who lead lives affected by poverty or discrimination may revise their hopes and expectations downwards in order to avoid disappointment. By aligning itself with existing human preferences, AI could therefore come to act on data that is heavily compromised.

To address this weakness, I suggest that AI systems need to be properly responsive to underlying human interests and values. A principle-based approach to AI alignment, which takes into account both of these factors, would yield agents that are less likely to do harm and more likely to promote human well-being. A principle-based approach to alignment could also be sensitive to other considerations, such as the welfare of future generations, non-human animals and the environment.

#### Three approaches

The final part of the paper looks at the ways in which principles for AI alignment might be identified.

In this context, I suggest that the main challenge is not to identify ‘true’ moral principles and encode them in AI – for even if we came to have great confidence in the truth of a single moral theory there would still be people with different beliefs and opinions who disagreed with us. Instead, we should try to identify principles for alignment that are acceptable to people who ascribe to a wide range of reasonable points of view. Principles of this kind could be arrived at in at least three different ways.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bf1adb861b5521f64c63_Fig%201%20copy.jpg)One approach looks at the possibility that there is an *overlapping consensus* between the moral beliefs held by people around the world. If such a consensus exists, then AI could be aligned with it – and potentially command widespread support – without encountering the problem of value imposition. In this regard, human rights are particularly promising. For while the idea of universal human rights is not wholly uncontested, the principles they embody command significant international support in practice. They also find justification in African, Islamic, Western, and Confucian philosophical traditions.

A second approach to pluralistic value alignment seeks to model fair principles for AI using the idea of a *‘veil of ignorance’*. The veil of ignorance is a device proposed by the philosopher John Rawls, to help people with different values and perspectives agree upon principles of justice for a society. The central claim is that when choosing principles of this kind, people should do so from an imaginary position where they do not know who they will be in that society, or what specific moral view they will hold. As a result, they will deliberate impartially and choose principles that do not unduly favour themselves. A similar approach could be used to model principles for AI.

Although it is difficult to say what people would choose in this situation without knowing more about the specific form of AI in question, it seems plausible that they would want to ensure that this technology is safe, amenable to human control, and that its benefits are distributed widely.

The final approach looks at ways in which *social choice theory* can be used to combine different viewpoints and inform the direction AI should take. One school of thought focuses on mathematical integration of individual preferences into a single ranking – which could be used to guide AI. More promising still are democratic methods such as voting and broad-based deliberation. When used successfully, these approaches reflect the value of equality and have the potential to ensure that principles for AI alignment enjoy widespread legitimacy.

#### Further research

Each proposal discussed here is tentative. They can be developed and combined in many different ways. This paper has benefited from feedback provided by over fifty people, including from audiences at workshops convened at Stanford University, Princeton University, PAI, the University of Warwick, and the University of California, Berkeley. Moving forward, our hope is that this paper can contribute to the growing conversation about AI systems and their alignment with human values.


International evaluation of an AI system for breast cancer screening

Breast cancer is the second leading cause of death from cancer in women, but outcomes have been shown to improve if caught and treated early. This is why many countries around the world have set up breast cancer screening programmes, aiming to identify breast cancer at earlier stages of the disease, when treatment can be more successful.

However, interpreting mammograms (breast x-rays) remains challenging, as evidenced by the high variability of experts’ performance in detecting cancer. In this collaborative research with [Google Health](https://health.google/) & [Cancer Research UK](https://www.cancerresearchuk.org/) Imperial Centre, [Northwestern University](https://www.northwestern.edu/), and [Royal Surrey County Hospital](https://www.royalsurrey.nhs.uk/) now [published in Nature](https://www.nature.com/articles/s41586-019-1799-6.epdf?author_access_token=V_LKV2xpSv9G1dhANYeWM9RgN0jAjWel9jnR3ZoTv0M5zwPVx5jT4z_z-YkUZTBT6_1AtRXi8QouJM7xB-oSN-cVBoH7f_QTgx-yQN3UBEVfkvO1_5urNT-CZHGCEQNGlCuO69tMQYak4SmdoDqyzg%3D%3D), we developed an AI system capable of surpassing clinical specialists from the UK and US in predicting breast cancer from mammograms, as confirmed by biopsy.

#### Breast cancer screening datasets

Breast cancer screening programmes vary from country to country. In the US, women are typically screened every one to two years, and their mammograms are interpreted by a single radiologist. In the UK, women are screened every three years, but each mammogram is interpreted by two radiologists, with an arbitration process in case of disagreement. We utilised large datasets collected in both countries to develop and evaluate this AI system.

The UK evaluation dataset consisted of a random sample of 10% of all women with screening mammograms at two sites in London between 2012 and 2015. It included 25,856 women, 785 of which had a biopsy, and 414 women with cancer that was diagnosed within three years of imaging. These de-identified data was collected as part of the [OPTIMAM](http://commercial.cancerresearchuk.org/optimam-mammography-image-database-and-viewing-software) database effort by Cancer Research UK, and are subject to strict privacy constraints.

The US evaluation dataset consisted of de-identified screening mammograms of 3,097 women collected between 2001 and 2018 from one academic medical centre. We included images from all 1,511 women who were biopsied during this time period and a random subset of women who never underwent biopsy. Among the women who received a biopsy, 686 were diagnosed with cancer within 2 years of imaging.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228be8cea4387c56ae3cd5f_Fig%201.jpg)Each mammogram has four images - two of each breast from different angles.#### Assessing the performance of the AI system

We compared the performance of the AI system against decisions made by individual human specialists in the original screening visit. In this evaluation, we found that the AI had an absolute reduction in false positives (women incorrectly referred for further investigation) of 5.7% for US subjects and 1.2% for UK subjects, and a reduction in false negatives (women incorrectly missed for further investigation)  of 9.4% for US subjects and 2.7% for UK subjects, compared to human experts. See the [paper](https://www.nature.com/articles/s41586-019-1799-6.epdf?author_access_token=V_LKV2xpSv9G1dhANYeWM9RgN0jAjWel9jnR3ZoTv0M5zwPVx5jT4z_z-YkUZTBT6_1AtRXi8QouJM7xB-oSN-cVBoH7f_QTgx-yQN3UBEVfkvO1_5urNT-CZHGCEQNGlCuO69tMQYak4SmdoDqyzg%3D%3D) for more extensive results.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bea6779b4947615344dd_Fig%202.jpg)The AI system accurately predicts, solely from screening mammograms, whether a patient will have a biopsy positive for breast cancer (note that only a small fraction of screening visits will result in a biopsy). It does so more accurately than individual human specialists, with lower false positive rates (for cancer prediction within three years for the UK dataset, and within two years for the US dataset). The top left indicates peak performance, with no false positives or false negatives. Credit: McKinney et al, Nature#### Generalisation across populations

To evaluate whether the AI system was able to generalise across populations and screening settings, we ran an experiment in which the AI was only allowed to learn from data from UK subjects, and then evaluated it on data from US subjects. This experiment showed that the AI system still surpassed human expert performance on US data.

This is an encouraging avenue for future research and gives more confidence about the robustness of the AI system. It might be possible that an AI diagnostic system could be beneficial even when used in areas where there is not a significant history of screening mammography on which to train it.

#### Future research & potential applications

We’ve yet to determine how to best deploy an AI system for clinical use in mammography. However, we investigated one possible such scenario by using the AI system as a “second reader”. We simulated this by treating the prediction of the AI system as an independent second opinion for every mammogram, taking the place of the ‘second reader’ in the UK ‘double reading’ system. When the AI and the clinician disagreed, the existing arbitration process would take place. In these simulated experiments, we showed that an AI-aided double-reading system could achieve non-inferior performance to the UK system with only 12% of the current second reader workload.

Further research, including prospective clinical studies, will be required to understand the full extent to which this technology can benefit breast cancer screening programmes.

Scott Mayer McKinney *
Marcin T. Sieniek *
Varun Godbole *
Jonathan Godwin
Natasha Antropova
Hutan Ashrafian *
Trevor Back
Mary Chesus
Greg C Corrado *
Ara Darzi *
Mozziyar Etemadi *
Florencia Garcia-Vicente *
Fiona J Gilbert *
Mark Halling-Brown *
Demis Hassabis
Sunny Jansen *
Alan Karthikesalingam
Christopher J Kelly
Dominic King
Joseph Ledsam
David Melnick *
Hormuz Mostofi *
Bernardino Romera Paredes
Lily Peng *
Joshua Jay Reicher *
Richard Sidebottom *
Mustafa Suleyman
Daniel Tse *
Kenneth C. Young *
Jeffrey De Fauw
Shravya Shetty *


Restoring ancient text using deep learning: a case study on Greek epigraphy

Historians rely on different sources to reconstruct the thought, society and history of past civilisations. Many of these sources are text-based – whether written on scrolls or carved into stone, the preserved records of the past help shed light on ancient societies. However, these records of our ancient cultural heritage are often incomplete: due to deliberate destruction, or erosion and fragmentation over time. This is the case for inscriptions: texts written on a durable surface (such as stone, ceramic, metal) by individuals, groups and institutions of the past, and which are the focus of the discipline called [epigraphy](https://en.wikipedia.org/wiki/Epigraphy). Thousands of inscriptions have survived to our day; but the majority have suffered damage over the centuries, and parts of the text are illegible or lost (Figure 1). The reconstruction ("restoration") of these documents is complex and time consuming, but necessary for a deeper understanding of civilisations past.

One of the issues with discerning meaning from incomplete fragments of text is that there are often multiple possible solutions. In many word games and puzzles, players guess letters to complete a word or phrase – the more letters that are specified, the more constrained the possible solutions become. But unlike these games, where players have to guess a phrase in isolation, historians restoring a text can estimate the likelihood of different possible solutions based on other context clues in the inscription – such as grammatical and linguistic considerations, layout and shape, textual parallels, and historical context. Now, by using machine learning trained on ancient texts, we’ve built a system that can furnish a more complete and systematically ranked list of possible solutions, which we hope will augment historians’ understanding of a text.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bcb8d5510c7928a6a8ee_Fig%201.jpg)Figure 1: Damaged inscription: a decree of the Athenian Assembly relating to the management of the Acropolis (dating 485/4 BCE). IG I3 4B. (CC BY-SA 3.0, WikiMedia)
#### Pythia

Pythia – which takes its name from the woman who delivered the god Apollo's oracular responses at the Greek sanctuary of Delphi – is the first ancient text restoration model that recovers missing characters from a damaged text input using deep neural networks. Bringing together the disciplines of ancient history and deep learning, the present work offers a fully automated aid to the text restoration task, providing ancient historians with multiple textual restorations, as well as the confidence level for each hypothesis.

Pythia takes a sequence of damaged text as input, and is trained to predict character sequences comprising hypothesised restorations of ancient Greek inscriptions (texts written in the Greek alphabet dating between the seventh century BCE and the fifth century CE). The architecture works at both the character- and word-level, thereby effectively handling long-term context information, and dealing efficiently with incomplete word representations (Figure 2). This makes it applicable to all disciplines dealing with ancient texts ([philology](https://en.wikipedia.org/wiki/Philology), [papyrology](https://en.wikipedia.org/wiki/Papyrology), [codicology](https://en.wikipedia.org/wiki/Codicology)) and applies to any language (ancient or modern).

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bdabafc10511b9bf2878_Fig%202.jpg)Figure 2: Pythia processing the phrase μηδέν ἄγαν (Mēdèn ágan) "nothing in excess," a fabled maxim inscribed on Apollo’s temple in Delphi. The letters "γα" are the characters to be predicted, and are annotated with ‘?’. Since ἄ??ν is not a complete word, its embedding is treated as unknown (‘unk’). The decoder outputs correctly "γα".
#### Experimental evaluation

To train Pythia, we wrote a non-trivial pipeline to convert the largest digital corpus of ancient Greek inscriptions ([PHI Greek Inscriptions](https://epigraphy.packhum.org/)) to machine actionable text, which we call PHI-ML. As shown in Table 1, Pythia’s predictions on PHI-ML achieve a 30.1% character error rate, compared to the 57.3% of evaluated human ancient historians (specifically, these were PhD students from Oxford). Moreover, in 73.5% of cases the ground-truth sequence was among the Top-20 hypotheses of Pythia, which effectively demonstrates the impact of this assistive method on the field of digital epigraphy, and sets the state-of-the-art in ancient text restoration.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bdbd5484b9bbe1f8bca4_Fig%203.jpg)Table 1: Pythia's Predictive performance of on PHI-ML.#### The importance of context

To evaluate Pythia’s receptiveness to context information and visualise the attention weights at each decoding step, we experimented with the modified lines of an inscription from the city of Pergamon (in modern-day Turkey)\*. In the text of Figure 3, the last word is a Greek personal name ending in -ου. We set ἀπολλοδώρου ("Apollodorou") as the personal name, and hid its first 9 characters. This name was specifically chosen because it already appeared within the input text. Pythia attended to the contextually-relevant parts of the text - specifically, ἀπολλοδώρου. The sequence ἀπολλοδώρ was predicted correctly. As a litmus test, we substituted ἀπολλοδώρου in the input text with another personal name of the same length: ἀρτεμιδώρου ("Artemidorou"). The predicted sequence changed accordingly to ἀρτεμιδώρ, thereby illustrating the importance of context in the prediction process.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bdda39cc63339ecee0f3_Fig%204.jpg)Figure 3: Visualisation of the attention weights for the decoding of the first 4 missing characters. To aid visualisation, the weights within the area of the characters to be predicted (‘?’) are in green, and in blue for the rest of the text; the magnitude of the weights  is represented by the colour intensity. The ground-truth text ἀπολλοδώρ appears in the input text, and Pythia attends to the relevant parts of the sequence.
#### Future research

The combination of machine learning and epigraphy has the potential to impact meaningfully  the study of inscribed texts, and widen the scope of the historian’s work. For this reason, we have open-sourced an online Python notebook, Pythia, and PHI-ML’s processing pipeline at <https://github.com/sommerschield/ancient-text-restoration>, collaborating with scholars at the University of Oxford. By so doing, we hope to aid future research and inspire further interdisciplinary work.

Yannis Assael
Thea Sommerschield*
Jonathan Prag*


Making Efficient Use of Demonstrations to Solve Hard Exploration Problems

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228bc30b71ebc05d2d0b6cb_Fig%201.gif)We propose a new agent, which we call Recurrent Replay Distributed DQN from Demonstrations (R2D3). R2D3 is designed to make efficient use of demonstrations to solve sparse reward tasks in partially observed environments with highly variable initial conditions. The architecture of the R2D3 agent is shown below. There are several actor processes, each running independent copies of the behavior against an instance of the environment. Each actor streams its experience to a shared agent replay buffer, where experience from all actors is aggregated and globally prioritized. The actors periodically request the latest network weights from the learner process in order to update their behavior.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228ba8915232a5a9d7138d0_Fig%202.jpg)As shown in the figure, R2D3 has two replay buffers: an agent replay and a demo replay buffer, which is populated with expert demonstrations of the task to be solved.  Maintaining separate replay buffers for agent experience and expert demonstrations allows us to prioritize the sampling of agent and expert data separately. The learner process samples batches of data from both the agent and demo replay buffers simultaneously. The demo ratio (ρ) controls the proportion of data coming from the expert demonstrations vs from the agent’s own experience. The demo ratio is implemented at a batch level by randomly choosing whether to sample from the expert replay buffer independently with probability ρ. When ρ=0, R2D3 performs standard RL, when ρ=1, R2D3 performs batch RL on the data in demo buffer. The loss is optimized by the learner by using n-step double Q-learning (with n=5) and a dueling architecture.

In each replay buffer, we store fixed-length sequences of *(s, a, r)* tuples where adjacent sequences overlap by 40 time-steps. These sequences never cross episode boundaries. Given a single batch of trajectories we unroll both online and target networks on the same sequence of states to generate value estimates with the recurrent state initialized to zero.

#### Hard Eight Task Suite

The tasks in the Hard Eight task suite require the agent to perform a sequence of high level skills in order to gain access to a large apple which gives the reward and terminates the episode. In the picture below, we give an example from the Baseball task. The agent must learn to execute these high level skills as a sequence of low level actions in the environment. The sequence of low-level actions can be quite long and consequently it is unlikely that the task will be solved by random exploration. Let us note that each step in this task involves interaction with physical objects in the environment which are shown in bold.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228ba9778f7e36c267110e5_Fig%203.jpg)

In the figure below, for each task the agent (blue triangle) must interact with objects in its environment in order to gain access to a large apple (red triangle) that provides reward. Our 3D environments are procedurally generated such that at every episode, the state of the world such as shapes of the objects, colors and positions are different. The environment is partially observable which means that the agent can only see the part of the environment at every timestep. Since the agent receives the reward only at the end of the episode and needs to execute a long sequence of actions, the exploration can be difficult. Furthermore, highly variable initial conditions and the objects that the agents can interact with can make the exploration even more difficult.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6228baa5b920ee7df8857dd2_Fig%204.jpg)#### Human demonstrating tasks

Below is a playlist of eight videos of humans performing each of the tasks to demonstrate the steps involved.

#### R2D3 on Hard Eight Tasks

Below is a playlist of eight representative videos of the R2D3 agent after training on each of these tasks.

#### Additional R2D3 Results

We ran a few additional experiments - shown in the playlist below - to get more information about the tasks R2D3 did not solve, or solved incorrectly.

##### Remember Sensor

****This task requires a long memory, and has the longest episode length of any task in the suite. In an attempt to mitigate these issues, we trained the agent using a higher action repeat of 4 which reduces the episode length, and used stale lstm states instead of zero lstm states which provides information about earlier in the episode. This allows R2D3 to learn policies that display reasonable behavior.

##### Throw Across

The demonstrations collected for this task had a very low success rate of 54%. We attempted to compensate for this by collecting an additional 30 demos. When we trained R2D3 with all 130 demos all seed solved the task.

##### **Wall Sensor Stack**

The original Wall Sensor Stack environment had a bug that the R2D3 agent was able to exploit. We fixed the bug and verified the agent can learn the proper stacking behaviour.

Caglar Gülçehre
Tom Le Paine
Bobak Shahriari
Misha Denil
Matt Hoffman
Hubert Soyer
Richard Tanburn
Steven Kapturowski
Neil Rabinowitz
Duncan Williams
Gabriel Barth-Maron
Ziyu Wang
Nando de Freitas
Worlds Team