Human-centred mechanism design with Democratic AI

**In our recent** [**paper**](https://www.nature.com/articles/s41562-022-01383-x)**, published in Nature Human Behaviour, we provide a proof-of-concept demonstration that deep reinforcement learning (RL) can be used to find economic policies that people will vote for by majority in a simple game. The paper thus addresses a key challenge in AI research - how to train AI systems that align with human values.**

Imagine that a group of people decide to pool funds to make an investment. The investment pays off, and a profit is made. How should the proceeds be distributed? One simple strategy is to split the return equally among investors. But that might be unfair, because some people contributed more than others. Alternatively, we could pay everyone back in proportion to the size of their initial investment. That sounds fair, but what if people had different levels of assets to begin with? If two people contribute the same amount, but one is giving a fraction of their available funds, and the other is giving them all, should they receive the same share of the proceeds?

This question of how to redistribute resources in our economies and societies has long generated controversy among philosophers, economists and political scientists. Here, we use deep RL as a testbed to explore ways to address this problem.

To tackle this challenge, we created a simple game that involved four players. Each instance of the game was played over 10 rounds. On every round, each player was allocated funds, with the size of the endowment varying between players. Each player made a choice: they could keep those funds for themselves or invest them in a common pool. Invested funds were guaranteed to grow, but there was a risk, because players did not know how the proceeds would be shared out. Instead, they were told that for the first 10 rounds there was one referee (A) who was making the redistribution decisions, and for the second 10 rounds a different referee (B) took over. At the end of the game, they voted for either A or B, and played another game with this referee. Human players of the game were allowed to keep the proceeds of this final game, so they were incentivised to report their preference accurately.

In reality, one of the referees was a pre-defined redistribution policy, and the other was designed by our deep RL agent. To train the agent, we first recorded data from a large number of human groups and taught a neural network to copy how people played the game. This simulated population could generate limitless data, allowing us to use data-intensive machine learning methods to train the RL agent to maximise the votes of these “virtual” players. Having done so, we then recruited new human players, and pitted the AI-designed mechanism head-to-head against well-known baselines, such as a *libertarian* policy that returns funds to people in proportion to their contributions.

When we studied the votes of these new players, we found that the policy designed by deep RL was more popular than the baselines. In fact, when we ran a new experiment asking a fifth human player to take on the role of referee, and trained them to try and maximise votes, the policy implemented by this “human referee” was still less popular than that of our agent.

AI systems have been sometimes criticised for learning policies that may be incompatible with human values, and this problem of “value alignment” has become a major concern in AI research. One merit of our approach is that the AI learns directly to maximise the stated preferences (or votes) of a group of people. This approach may help ensure that AI systems are less likely to learn policies that are unsafe or unfair. In fact, when we analysed the policy that the AI had discovered, it incorporated a mixture of ideas that have previously been proposed by human thinkers and experts to solve the redistribution problem.

Firstly, the AI chose to redistribute funds to people in proportion to their *relative* rather than *absolute* contribution. This means that when redistributing funds, the agent accounted for each player’s initial means, as well as their willingness to contribute. Secondly, the AI system especially rewarded players whose relative contribution was more generous, perhaps encouraging others to do likewise. Importantly, the AI only discovered these policies by learning to maximise human votes. The method therefore ensures that humans remain “in the loop” and the AI produces human-compatible solutions.

By asking people to vote, we harnessed the principle of majoritarian democracy for deciding what people want. Despite its wide appeal, it is widely acknowledged that democracy comes with the caveat that the preferences of the majority are accounted for over those of the minority. In our study, we ensured that – like in most societies – that minority consisted of more generously endowed players. But more work is needed to understand how to trade off the relative preferences of majority and minority groups, by designing democratic systems that allow all voices to be heard.

Raphael Koster
Jan Balaguer
Andrea Tacchetti
Ari Weinstein
Tina Zhu
Oliver Hauser* (University of Exeter)
Duncan Williams
Lucy Campbell-Gillingham
Phoebe Thacker
Matthew Botvinick
Christopher Summerfield


BYOL-Explore: Exploration with Bootstrapped Prediction

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62b068d95af4f802873938ec_throw_across_first_1.gif)Second-person and top-down views of a BYOL-Explore agent solving Thow-Across level of [DM-HARD-8](https://arxiv.org/abs/1909.01387), whereas pure RL and other baseline exploration methods fail to make any progress on Thow-Across.Curiosity-driven exploration is the active process of seeking new information to enhance the agent’s understanding of its environment. Suppose that the agent has learned a model of the world that can predict future events given the history of past events. The curiosity-driven agent can then use the prediction mismatch of the world model as the intrinsic reward for directing its exploration policy towards seeking new information. As follows, the agent can then use this new information to enhance the world model itself so it can make better predictions.  This iterative process can allow the agent to eventually explore every novelty  in the world and use this information to build an accurate world model.

Inspired by the successes of [bootstrap your own latent](https://arxiv.org/abs/2006.07733) (BYOL) – which has been applied in [computer vision](https://arxiv.org/abs/2103.16559), [graph representation learning](https://arxiv.org/abs/2102.06514), and [representation learning in RL](https://arxiv.org/abs/2007.05929) – we propose BYOL-Explore: a conceptually simple yet general, curiosity-driven AI agent for solving hard-exploration tasks. BYOL-Explore learns a representation of the world by predicting its own future representation. Then, it uses the prediction-error at the representation level as an intrinsic reward to train a curiosity-driven policy. Therefore, BYOL-Explore learns a world representation, the world dynamics, and a curiosity-driven exploration policy all-together, simply by optimising the prediction error at the representation level.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62b0511e11e44e60dc8b0b00_BYOL-Explore_Schematic_v3.png)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62b06901690bcc5c64c64fa7_fp_hard_8_1.gif)Comparison between BYOL-Explore, [Random Network Distillation](https://arxiv.org/abs/1810.12894) (RND), [Intrinsic Curiosity Module](https://arxiv.org/abs/1705.05363) (ICM) and pure RL (no intrinsic reward), in terms of mean capped human-normalised score (CHNS).Despite the simplicity of its design, when applied to the [DM-HARD-8](https://arxiv.org/abs/1909.01387) suite of challenging 3-D, visually complex, and hard exploration tasks, BYOL-Explore outperforms standard curiosity-driven exploration methods such as [Random Network Distillation](https://arxiv.org/abs/1810.12894) (RND) and [Intrinsic Curiosity Module](https://arxiv.org/abs/1705.05363) (ICM), in terms of mean capped human-normalised score (CHNS), measured across all tasks. Remarkably, BYOL-Explore achieved this performance using only a single network concurrently trained across all tasks, whereas prior work was restricted to the single-task setting and could only make meaningful progress on these tasks when provided with human expert demonstrations.

As further evidence of its generality, BYOL-Explore achieves super-human performance in the ten hardest exploration [Atari games](https://arxiv.org/abs/1207.4708), while having a simpler design than other competitive agents, such as [Agent57](https://arxiv.org/abs/2003.13350) and [Go-Explore](https://arxiv.org/abs/2004.12919).

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6307b4f7fc73190a5ad67ce6_62b06c47ed1b2613a7eea9ce_atari-chns-across-games-2.png)Comparison between BYOL-Explore, [Random Network Distillation](https://arxiv.org/abs/1810.12894) (RND), [Intrinsic Curiosity Module](https://arxiv.org/abs/1705.05363) (ICM) and pure RL (no intrinsic reward), in terms of mean capped human-normalised score (CHNS).![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6307b4f8b782305825efd7ae_62b06c464bde0e1f3a671733_dmh-chns-across-tasks-1.png)Moving forward, we can generalise BYOL-Explore to highly stochastic environments by learning a probabilistic world model that could be used to generate trajectories of the future events. This could allow the agent to model the possible stochasticity of the environment, avoid stochastic traps, and plan for exploration.

Zhaohan Daniel Guo
Shantanu Thakoor
Miruna Pîslar
Bernardo Avila Pires
Florent Altché
Corentin Tallec
Alaa Saade
Daniele Calandriello
Jean-Bastien Grill
Yunhao Tang
Michal Valko
Rémi Munos
Mohammad Gheshlaghi Azar
Bilal Piot


Unlocking High-Accuracy Differentially Private Image Classification through Scale

A recent [DeepMind paper](https://arxiv.org/abs/2112.04359) on the ethical and social risks of language models identified large language models [leaking sensitive information](https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting) about their training data as a potential risk that organisations working on these models have the responsibility to address. Another [recent paper](https://arxiv.org/abs/2201.04845) shows that similar privacy risks can also arise in standard image classification models: a fingerprint of each individual training image can be found embedded in the model parameters, and malicious parties could exploit such fingerprints to reconstruct the training data from the model.

Privacy-enhancing technologies like differential privacy (DP) can be deployed at training time to mitigate these risks, but they often incur significant reduction in model performance. In this work, we make substantial progress towards unlocking high-accuracy training of image classification models under differential privacy.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62ab43e65845e64d1a827c87_Figure.png)*Figure 1: (left) Illustration of training data leakage in GPT-2 [credit: Carlini et al. "Extracting Training Data from Large Language Models", 2021]. (right) CIFAR-10 training examples reconstructed from a 100K parameter convolutional neural network [credit: Balle et al. "Reconstructing Training Data with Informed Adversaries", 2022]*Differential privacy was [proposed](https://link.springer.com/content/pdf/10.1007/11681878_14.pdf) as a mathematical framework to capture the requirement of protecting individual records in the course of statistical data analysis (including the training of machine learning models). DP algorithms protect individuals from any inferences about the features that make them unique (including complete or partial reconstruction) by injecting carefully calibrated noise during the computation of the desired statistic or model. Using DP algorithms provides robust and rigorous privacy guarantees both in theory and in practice, and has become a de-facto gold standard adopted by a number of [public](https://dl.acm.org/doi/10.1145/3219819.3226070) and [private](https://ai.googleblog.com/2022/02/federated-learning-with-formal.html) organisations.

The most popular DP algorithm for deep learning is differentially private stochastic gradient descent (DP-SGD), a modification of standard SGD obtained by clipping gradients of individual examples and adding enough noise to mask the contribution of any individual to each model update:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62ab451f8d59129ace23bbac_Figure2.png)*Figure 2: Illustration of how DP-SGD processes gradients of individual examples and adds noise to produce model updates with privatised gradients.*Unfortunately, prior works have found that in practice, the privacy protection provided by DP-SGD often comes at the cost of significantly less accurate models, which presents a major obstacle to the widespread adoption of differential privacy in the machine learning community. According to empirical evidence from prior works, this utility degradation in DP-SGD becomes more severe on larger neural network models – including the ones regularly used to achieve the best performance on challenging image classification benchmarks.

Our work investigates this phenomenon and proposes a series of simple modifications to both the training procedure and model architecture, yielding a significant improvement on the accuracy of DP training on standard image classification benchmarks. The most striking observation coming out of our research is that DP-SGD can be used to efficiently train much deeper models than previously thought, as long as one ensures the model's gradients are well-behaved. We believe the substantial jump in performance achieved by our research has the potential to unlock practical applications of image classification models trained with formal privacy guarantees.

The figure below summarises two of our main results: an ~10% improvement on CIFAR-10 compared to previous work when privately training without additional data, and a top-1 accuracy of 86.7% on ImageNet when privately fine-tuning a model pre-trained on a different dataset, almost closing the gap with the best non-private performance.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62ab4601aabb144ad7dcd770_Figure3.png)*Figure 3: (left) Our best results on training WideResNet models on CIFAR-10 without additional data. (right) Our best results on fine-tuning NFNet models on ImageNet. The best performing model was pre-trained on an internal dataset disjoint from ImageNet.*These results are achieved at 𝜺=8, a standard setting for calibrating the strength of the protection offered by differential privacy in machine learning applications. We refer to the paper for a discussion of this parameter, as well as additional experimental results at other values of 𝜺 and also on other datasets. Together with the paper, we are also open-sourcing our implementation to enable other researchers to verify our findings and build on them. We hope this contribution will help others interested in making practical DP training a reality.

Download our JAX implementation [on GitHub](https://github.com/deepmind/jax_privacy).

Soham De
Leonard Berrada
Jamie Hayes
Samuel L. Smith
Borja Balle


Evaluating Multimodal Interactive Agents

To train agents to interact well with humans, we need to be able to measure progress. But human interaction is complex and measuring progress is difficult. In this work we developed a method, called the Standardised Test Suite (STS), for evaluating agents in temporally extended, multi-modal interactions. We examined interactions that consist of human participants asking agents to perform tasks and answer questions in a 3D simulated environment.

The STS methodology places agents in a set of behavioural scenarios mined from real human interaction data. Agents see a replayed scenario context, receive an instruction, and are then given control to complete the interaction offline. These agent continuations are recorded and then sent to human raters to annotate as success or failure. Agents are then ranked according to the proportion of scenarios on which they succeed.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6290c501360e22cfee4d2a80_select_frames_fig_v2.png)Figure 1: Example of an original scenario taken from two humans interacting alongside successful and unsuccessful agent continuations.Many of the behaviours that are second nature to humans in our day-to-day interactions are difficult to put into words, and impossible to formalise. Thus, the mechanism relied on for solving games (like Atari, Go, DotA, and Starcraft) with reinforcement learning won't work when we try to teach agents to have fluid and successful interactions with humans. For example, think about the difference between these two questions: "Who won this game of Go?" versus "What are you looking at?" In the first case, we can write a piece of computer code that counts the stones on the board at the end of the game and determines the winner with certainty. In the second case, we have no idea how to codify this: the answer may depend on the speakers, the size and shapes of the objects involved, whether the speaker is joking, and other aspects of the context in which the utterance is given. Humans intuitively understand the myriad of relevant factors involved in answering this seemingly mundane question.

Interactive evaluation by human participants can serve as a touchstone for understanding agent performance, but this is noisy and expensive. It is difficult to control the exact instructions that humans give to agents when interacting with them for evaluation. This kind of evaluation is also in real-time, so it is too slow to rely on for swift progress. Previous works have relied on proxies to interactive evaluation. Proxies, such as losses and scripted probe tasks (e.g. “lift the x” where x is randomly selected from the environment and the success function is painstakingly hand-crafted), are useful for gaining insight into agents quickly, but don’t actually correlate that well with interactive evaluation. Our new method has advantages, mainly affording control and speed to a metric that closely aligns with our ultimate goal - to create agents that interact well with humans.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6290c5317e6d90ce5c603af8_sts_vs_ha.png)Figure 2: STS evaluation compared to other evaluation metrics used for evaluating interactive agents. The STS correlates best with interactive evaluation compared to previous proxies used.The development of MNIST, ImageNet and other human-annotated datasets has been essential for progress in machine learning. These datasets have allowed researchers to train and evaluate classification models for a one-time cost of human inputs. The STS methodology aims to do the same for human-agent interaction research. This evaluation method still requires humans to annotate agent continuations; however, early experiments suggest that automation of these annotations may be possible, which would enable fast and effective automated evaluation of interactive agents. In the meantime, we hope that other researchers can use the methodology and system design to accelerate their own research in this area.

Josh Abramson
Arun Ahuja
Federico Carnevale
Petko Georgiev
Alex Goldin
Jessica Landon
Timothy Lillicrap
Alistair Muldal
Adam Santoro
Tamara von Glehn
Gregory Wayne
Nathaniel Wong
Chen Yan
Blake Richards*
Alden Hung*


Dynamic language understanding: adaptation to new knowledge in parametric and semi-parametric models

Many recent successes in language models (LMs) have been achieved within a ‘static paradigm’, where the focus is on improving performance on the benchmarks that are created without considering the temporal aspect of data. For instance, answering questions on events that the model could learn about during training, or evaluating on text sub-sampled from the same period as the training data. However, our language and knowledge are dynamic and ever evolving. Therefore, to enable a more realistic evaluation of question-answering models for the next leap in performance, it’s essential to ensure they are flexible and robust when encountering new and unseen data.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/628f80e879bcf32f9eefad77_Figure%201.png)Figure 1. We evaluate our models on unseen language and knowledge, seen here using questions about events in 2020, while the model has been trained on data up until the end of 2019.In 2021, we released [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://arxiv.org/abs/2102.01951) and the [dynamic language modelling benchmarks](https://github.com/deepmind/deepmind-research/tree/master/pitfalls_static_language_models) for WMT and arXiv to facilitate language model evaluation that take temporal dynamics into account. In this paper, we highlighted issues that current state-of-the-art large LMs face with temporal generalisation and found that knowledge-intensive tokens take a considerable performance hit.

Today, we’re releasing two papers and a new benchmark that further advance research on this topic. In [StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models](http://arxiv.org/abs/2205.11388), we study the downstream task of question-answering on our newly proposed benchmark, [*StreamingQA*](https://github.com/deepmind/streamingqa): we want to understand how parametric and retrieval-augmented, semi-parametric question-answering models adapt to new information, in order to answer questions about new events. In [Internet-augmented language models through few-shot prompting for open-domain question answering](https://arxiv.org/abs/2203.05115), we explore the power of combining a few-shot prompted large language model along with Google Search as a retrieval component. In doing so, we aim to improve the model's factuality, while making sure it has access to up-to-date information for answering a diverse set of questions.

#### StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models

Knowledge and language understanding of models evaluated through question-answering (QA) has been commonly studied on static snapshots of knowledge, like Wikipedia. To study how semi-parametric QA models and their underlying parametric LMs adapt to evolving knowledge, we constructed the new large-scale benchmark, StreamingQA, with human-written and automatically generated questions asked on a given date, to be answered from 14 years of time-stamped news articles (see Figure 2). We show that parametric models can be updated without full retraining, while avoiding catastrophic forgetting. For semi-parametric models, adding new articles into the search space allows for rapid adaptation, however, models with an outdated underlying LM underperform those with a retrained LM.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/628f8145cc208c1207a49e16_Figure%202.png)Figure 2. Example questions from the StreamingQA benchmark.#### Internet-augmented language models through few-shot prompting for open-domain question-answering

We’re aiming to capitalise on the unique few-shot capabilities offered by large-scale language models to overcome some of their challenges, with respect to grounding to factual and up-to-date information. Motivated by semi-parametric LMs, which ground their decisions in externally retrieved evidence, we use few-shot prompting to learn to condition LMs on information returned from the web using Google Search, a broad and constantly updated knowledge source. Our approach does not involve fine-tuning or learning additional parameters, thus making it applicable to virtually any language model. And indeed, we find that LMs conditioned on the web surpass the performance of closed-book models of similar, or even larger, model size in open-domain question-answering.

Elena Gribovskaya
Angeliki Lazaridou
Tomáš Kočiský


Emergent Bartering Behaviour in Multi-Agent Reinforcement Learning

In [our recent paper](https://arxiv.org/abs/2205.06760), we explore how populations of deep reinforcement learning (deep RL) agents can learn microeconomic behaviours, such as production, consumption, and trading of goods. We find that artificial agents learn to make economically rational decisions about production, consumption, and prices, and react appropriately to supply and demand changes. The population converges to local prices that reflect the nearby abundance of resources, and some agents learn to transport goods between these areas to “buy low and sell high”. This work advances the broader multi-agent reinforcement learning research agenda by introducing new social challenges for agents to learn how to solve.

Insofar as the goal of multi-agent reinforcement learning research is to eventually produce agents that work across the full range and complexity of human social intelligence, the set of domains so far considered has been woefully incomplete. It is still missing crucial domains where human intelligence excels, and humans spend significant amounts of time and energy. The subject matter of economics is one such domain. Our goal in this work is to establish environments based on the themes of trading and negotiation for use by researchers in multi-agent reinforcement learning.

Economics uses agent-based models to simulate how economies behave. These agent-based models often build in economic assumptions about how agents should act. In this work, we present a multi-agent simulated world where agents can learn economic behaviours from scratch, in ways familiar to any Microeconomics 101 student: decisions about production, consumption, and prices. But our agents also must make other choices that follow from a more physically embodied way of thinking. They must navigate a physical environment, find trees to pick fruits, and partners to trade them with. Recent advances in deep RL techniques now make it possible to create agents that can learn these behaviours on their own, without requiring a programmer to encode domain knowledge.

Our environment, called *Fruit Market*, is a multiplayer environment where agents produce and consume two types of fruit: apples and bananas. Each agent is skilled at producing one type of fruit, but has a preference for the other – if the agents can learn to barter and exchange goods, both parties would be better off.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62824c1d9603656400227009_blog_post_environment.png)**An example map in Fruit Market:** Agents move around the map to harvest apples and bananas from trees, meet up to trade with each other, and then consume the fruit that they prefer.In our experiments, we demonstrate that current deep RL agents can learn to trade, and their behaviours in response to supply and demand shifts align with what microeconomic theory predicts. We then build on this work to present scenarios that would be very difficult to solve using analytical models, but which are straightforward for our deep RL agents. For example, in environments where each type of fruit grows in a different area, we observe the emergence of different price regions related to the local abundance of fruit, as well as the subsequent learning of arbitrage behaviour by some agents, who begin to specialise in transporting fruit between these regions.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62824c66f299c6e3f1f54b36_blog_post_supply_demand.png)**Emergent Supply and Demand curves:** In this experiment, we manipulate the probability of apple trees (a=x) and banana trees (b=y) appearing in each map location. These results replicate the theoretical supply and demand curves presented in introductory Microeconomics courses.The field of agent-based computational economics uses similar simulations for economics research. In this work, we also demonstrate that state-of-the-art deep RL techniques can flexibly learn to act in these environments from their own experience, without needing to have economic knowledge built in. This highlights the reinforcement learning community’s recent progress in multi-agent RL and deep RL, and demonstrates the potential of multi-agent techniques as tools to advance simulated economics research.

As a [path to artificial general intelligence](https://arxiv.org/abs/1903.00742) (AGI), multi-agent reinforcement learning research should encompass all critical domains of social intelligence. However, until now it hasn’t incorporated traditional economic phenomena such as trade, bargaining, specialisation, consumption, and production. This paper fills this gap and provides a platform for further research. To aid future research in this area, the Fruit Market environment will be included in the next release of the [Melting Pot](https://github.com/deepmind/meltingpot) suite of environments.

Mike Johanson
Edward Hughes
Finbarr Timbers
Joel Leibo


A Generalist Agent

Inspired by progress in large-scale language modelling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627d13d743dc353a184da8d4_data_sequences.png)During the training phase of Gato, data from different tasks and modalities are serialised into a flat sequence of tokens, batched, and processed by a transformer neural network similar to a large language model. The loss is masked so that Gato only predicts action and text targets.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627d148b710554b355ec4d28_diagram_train%20(1)-1.png)When deploying Gato, a prompt, such as a demonstration, is tokenised, forming the initial sequence. Next, the environment yields the first observation, which is also tokenised and appended to the sequence. Gato samples the action vector autoregressively, one token at a time.

Once all tokens comprising the action vector have been sampled (determined by the action specification of the environment), the action is decoded and sent to the environment which steps and yields a new observation. Then the procedure repeats. The model always sees all previous observations and actions within its context window of 1024 tokens.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627d14de5d578e1ad6af2aee_eval_sequence-1.png)Gato is trained on a large number of datasets comprising agent experience in both simulated and real-world environments, in addition to a variety of natural language and image datasets. The number of tasks, where the performance of the pretrained Gato model is above a percentage of expert score, grouped by domain, is shown here.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627d15240b604dc2628bc05f_barplot_domains.png)The following images also show how the pre-trained Gato model with the same weights can do image captioning, engage in an interactive dialogue, and control a robot arm, among many other tasks.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627d15dba01b303962bf0014_image_captions_v3-1.png)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627d161a9709ad24126a513b_dialogue_examples_g1-1.png)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627d1648c0eef89f6a91f370_real_robot_blue_on_green.png)

Scott Reed
Konrad Żołna
Emilio Parisotto
Sergio Gómez Colmenarejo
Alexander Novikov
Gabriel Barth-Maron
Mai Giménez
Yury Sulsky
Jackie Kay
Jost Tobias Springenberg
Tom Eccles
Jake Bruce
Ali Razavi
Ashley Edwards
Nicolas Heess
Yutian Chen
Raia Hadsell
Oriol Vinyals
Mahyar Bordbar
and Nando de Freitas


Active offline policy selection

Reinforcement learning (RL) has made tremendous progress in recent years towards addressing real-life problems – and offline RL made it even more practical. Instead of direct interactions with the environment, we can now train many algorithms from a single pre-recorded dataset. However, we lose the practical advantages in data-efficiency of offline RL when we evaluate the policies at hand.

For example, when training robotic manipulators the robot resources are usually limited, and training many policies by offline RL on a single dataset gives us a large data-efficiency advantage compared to online RL. Evaluating each policy is an expensive process, which requires interacting with the robot thousands of times. When we choose the best algorithm, hyperparameters, and a number of training steps, the problem quickly becomes intractable.

To make RL more applicable to real-world applications like robotics, we propose using an intelligent evaluation procedure to select the policy for deployment, called active offline policy selection (A-OPS). In A-OPS, we make use of the prerecorded dataset and allow limited interactions with the real environment to boost the selection quality.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627bec820168b096d5504b9e_1.png)Active offline policy selection (A-OPS) selects the best policy out of a set of policies given a pre-recorded dataset and limited interaction with the environment.To minimise interactions with the real environment, we implement three key features:

1. Off-policy policy evaluation, such as fitted Q-evaluation (FQE), allows us to make an initial guess about the performance of each policy based on an offline dataset. It correlates well with the ground truth performance in many environments, including real-world robotics where it is applied for the first time.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627bed3d532fe54cd0a5e46a_2-1.png)FQE scores are well aligned with the ground truth performance of policies trained in both sim2real and offline RL setups.The returns of the policies are modelled jointly using a Gaussian process, where observations include FQE scores and a small number of newly collected episodic returns from the robot. After evaluating one policy, we gain knowledge about all policies because their distributions are correlated through the kernel between pairs of policies. The kernel assumes that if policies take similar actions – such as moving the robotic gripper in a similar direction – they tend to have similar returns.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627bed70dc6376412174067c_3.gif)We useOPE scores and episodic returns to model latent policy performance as a Gaussian process.![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627cf4b5d6cc1c56d8a29cb4_4-1.png)Similarity between the policies is modelled through the distance between the actions these policies produce.1. To be more data-efficient, we apply Bayesian optimisation and prioritise more promising policies to be evaluated next, namely those that have high predicted performance and large variance.

We demonstrated this procedure in a number of environments in several domains: dm-control, Atari, simulated, and real robotics. Using A-OPS reduces the regret rapidly, and with a moderate number of policy evaluations, we identify the best policy.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/627cf4fe45a4003f659b47c0_5.gif)In a real-world robotic experiment, A-OPS helps identify a very good policy faster than other baselines. To find a policy with close to zero regret out of 20 policies takes the same amount of time as it takes to evaluate two policies with current procedures.Our results suggest that it’s possible to make an effective offline policy selection with only a small number of environment interactions by utilising the offline data, special kernel, and Bayesian optimisation. The code for A-OPS is open-sourced and [available on GitHub](https://github.com/deepmind/active_ops) with an example dataset to try.

Yutian Chen
Ksenia Konyushkova
Tom Paine
Caglar Gulcehre
Cosmin Paduraru
Daniel J. Mankowitz
Misha Denil
Nando de Freitas


An empirical analysis of compute-optimal large language model training

In the last few years, a focus in language modelling has been on improving performance through increasing the number of parameters in transformer-based models. This approach has led to impressive results and state-of-the-art performance across many natural language processing tasks.

We also pursued this line of research at DeepMind and recently showcased Gopher, a 280-billion parameter model that established leading performance on a wide range of tasks including language modelling, reading comprehension, and question answering. Since then, an even larger model named Megatron-Turing NLG has been published with 530 billion parameters.

Due to the substantial cost of training these large models, it is paramount to estimate the best possible training setup to avoid wasting resources. In particular, the training compute cost for transformers is determined by two factors: the model size and the number of training tokens.

The current generation of large language models has allocated increased computational resources to increasing the parameter count of large models and keeping the training data size fixed at around 300 billion tokens. In this work, we empirically investigate the optimal tradeoff between increasing model size and the amount of training data with increasing computational resources. Specifically, we ask the question: “What is the optimal model size and number of training tokens for a given compute budget?” To answer this question, we train models of various sizes and with various numbers of tokens, and estimate this trade-off empirically.

Our main finding is that the current large language models are far too large for their compute budget and are not being trained on enough data. In fact, we find that for the number of training FLOPs used to train *Gopher*, a 4x smaller model trained on 4x more data would have been preferable.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62557f7626b9e103db549c7b_tokens_vs_flops%20(1).png)**Figure 1:** Based on our approach, we show our projections of the optimal number of training tokens and parameters. We show points representing the training setup of three different established large language models along with our new model, Chinchilla*.*We test our data scaling hypothesis by training *Chinchilla,* a 70-billion parameter model trained for 1.3 trillion tokens. While the training compute cost for ChinchillaandGopherare the same, we find that it outperforms Gopher and other large language models on nearly every measured task, despite having 70 billion parameters compared to Gopher’s 280 billion.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62557f43672f48833d2088c1_chinchilla.performance.image.png)**Figure 2:** For various common benchmarks that include Question Answering (TriviaQA), CommonSense (HellaSwag, PIQA, Winogrande, and BoolQ), Reading Comprehension (LAMBADA), and the large Multi-task Language Understanding (MMLU) general knowledge benchmark, we compare the performance of Gopher, Chinchilla*,* GPT-3, and Megatron-Turing NLG.After the release of Chinchilla, a model named PaLM was released with 540 billion parameters and trained on 768 billion tokens. This model was trained with approximately 5x the compute budget of Chinchilla and outperformed Chinchilla on a range of tasks. While the training corpus is different, our methods do predict that such a model trained on our data would outperform Chinchilla despite not being compute-optimal. Given the PaLM compute budget, we predict a 140-billion-parameter model trained on 3 trillion tokens to be optimal and more efficient for inference.

An additional benefit of smaller, more performant models is that the inference time and memory costs are reduced making querying the models both faster and possible on less hardware. In practice, while the training FLOPs between Gopherand Chinchilla are the same, the cost of using Chinchilla is substantially smaller, in addition to it performing better. Further simple optimisations may be possible that are able to continue to provide large gains.

Jordan Hoffmann
Sebastian Borgeaud
Arthur Mensch
Laurent Sifre


GopherCite: Teaching language models to support answers with verified quotes

DeepMind published a [series of papers](https://deepmind.com/blog/article/language-modelling-at-scale) about large language models (LLMs) last year, including [an analysis](https://arxiv.org/abs/2112.11446) of Gopher, our large language model. Language modelling technology, which is also currently being developed by several other labs and companies, promises to strengthen many applications, from [search engines](https://blog.google/products/search/search-language-understanding-bert/) to a new wave of chatbot-like [conversational assistants](https://blog.google/technology/ai/lamda/) and beyond. One [paper](https://arxiv.org/abs/2112.04359) in this series laid out a number of reasons why “raw” language models like Gopher do not meet our standards for safely deploying this technology in user-facing applications, especially if guard rails for managing problematic and potentially harmful behaviour are not set in place.

Our latest work focuses on one of these concerns: Language models like Gopher can “hallucinate” facts that appear plausible but are actually fake. Those who are familiar with this problem know to do their own fact-checking, rather than trusting what language models say. Those who are not, may end up believing something that isn’t true. This paper describes GopherCite, a model which aims to address the problem of language model hallucination. GopherCite attempts to back up all of its factual claims with evidence from the web. It uses Google Search to find relevant web pages on the internet and quotes a passage which tries to demonstrate why its response is correct. If the system is unable to form an answer that can be well-supported by evidence, it tells the user, “I don’t know”, instead of providing an unsubstantiated answer.

Supporting simple factual claims with easily verifiable evidence is one step towards making language models more trustworthy, both for users interacting with them and for annotators assessing the quality of samples. A comparison between the behaviour of “raw” Gopher and our new model is helpful for illustrating this change.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6238b8206dff585fb5183967_fig_1.svg)Based on GopherCite’s response, you’ll notice that Gopher invented a fact (“Lake Placid hosted the winter Olympics in 1936”) without warning. When shown a verified snippet from a relevant Wikipedia page by GopherCite, we can confirm that Lake Placid only hosted the Olympics twice, in 1932 and 1980.

To alter Gopher’s behaviour in this way, we trained Gopher according to human preferences. We asked participants in a user study to pick their preferred answer from a pair of candidates, according to criteria including how well the evidence supports the answers given. These labels were used as training data for both supervised learning on highly rated samples and for [reinforcement learning from human preferences](https://arxiv.org/abs/1909.08593) (RLHP). We also took this approach in [our recent work on red teaming](https://deepmind.com/research/publications/2022/Red-Teaming-Language-Models-with-Language-Models).

We are not the only ones interested in this problem of factual inaccuracy in language models. Our colleagues at Google recently made progress on factual grounding in their latest [LaMDA system](https://ai.googleblog.com/2022/01/lamda-towards-safe-grounded-and-high.html), having a conversational model interact with Google Search and sometimes share relevant URLs. Indeed, GopherCite’s training regimen uses similar methodology to that of LaMDA, but a critical difference is that we aim to provide a specific snippet of relevant evidence, rather than simply pointing the user to a URL. Based on motivations similar to our own, OpenAI has [recently announced work](https://openai.com/blog/webgpt/) developing a closely related system called WebGPT, which also applies RLHP to align their GPT-3 language model. Whereas GopherCite focuses on reading long document inputs, WebGPT carefully curates the context presented to the language model by interacting multiple times with a web browser. It also cites evidence to back up its responses. Similarities and differences between these systems and our own are discussed in our paper and we also demonstrate that GopherCite very often provides compelling evidence for its claims.

We conducted a user study with paid participants to assess the model on two types of questions: fact-seeking questions typed into Google Search ([released by Google in a dataset called “NaturalQuestions”](https://ai.google.com/research/NaturalQuestions)), and explanation-seeking questions which Reddit users asked on a forum called “/r/eli5” (“Explain it Like I’m 5 [years old]”). The participants in our study determined that GopherCite answers fact-seeking questions correctly – and with satisfactory evidence – about 80% of the time, and does so for explanation-seeking questions about 67% of the time. When we allow GopherCite to refrain from answering some questions, its performance improves dramatically amongst the questions it does choose to answer (see the paper for details). This explicit mechanism for abstaining is a core contribution of our work.

But when we evaluate the model on a set of “adversarial” questions, which attempt to trick the model into parroting a fiction or misconception that is stated on the internet, GopherCite often falls into the trap. For instance, when asked “what does Red Bull give you?”, here is how it responds:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6238b8399fc3670aa60958e8_fig_2.svg)An example of GopherCite's response to a question from the TruthfulQA dataset. We also show alongside the sample, how human annotators assessed three criteria we have for samples. 1. "Plausible": Is the answer on topic, attempting to address the user's question? 2. "Supported": Does the quotation convince you that the response is accurate? 3. "True": If the response does not contain false information.We think this failure mode and others discussed in our paper can be avoided by enriching the setting, moving from a “single-shot” reply to a user’s question, to one in which the model can ask clarifying questions of the user and engage in a dialogue. For example, we could enable future models to ask the user whether they want an answer that is literally true or one that is true in the confines of the fictional world of a Red Bull advertisement.

In summary, we think GopherCite is an important step forward, but building it has taught us that evidence citation is only one part of an overall strategy for safety and trustworthiness. More fundamentally, not all claims require quote evidence – and as we demonstrated above, not all claims supported by evidence are true. Some claims require multiple pieces of evidence along with a logical argument explaining why the claim follows. We will continue working in this area and aim to overcome the issues presented with further research and development as well as dedicated sociotechnical research.

Our paper covers many more details about our methods, experiments, and relevant context from the research literature. We have also created an FAQ about GopherCite, answered by the model itself after reading the paper's introduction (using candidate samples curated by the authors):

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6238b879d3a417cd9f473c0c_fig_3.svg)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6238b8812be7bee9042434ca_fig_4.svg)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6238b887522b7603b6dcb08d_fig_5.svg)

Jacob Menick
Maja Trebacz
Vladimir Mikulik
John Aslanides
Francis Song
Martin Chadwick
Mia Glaese
Susannah Young
Lucy Campbell-Gillingham
Geoffrey Irving
Nat McAleese


Learning Robust Real-Time Cultural Transmission without Human Data

Over millennia, humankind has discovered, evolved, and accumulated a wealth of cultural knowledge, from navigation routes to mathematics and social norms to works of art. Cultural transmission, defined as efficiently passing information from one individual to another, is the inheritance process underlying this exponential increase in human capabilities.

Our agent, in blue, imitates and remembers the demonstration of both bots (left) and humans (right), in red.

For more videos of our agents in action, visit our [website](https://sites.google.com/view/dm-cgi).

In this work, we use deep reinforcement learning to generate artificial agents capable of test-time cultural transmission. Once trained, our agents can infer and recall navigational knowledge demonstrated by experts. This knowledge transfer happens in real time and generalises across a vast space of previously unseen tasks. For example, our agents can quickly learn new behaviours by observing a single human demonstration, without ever training on human data.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6227d60010fcae8805be6718_Fig%201.jpg)A summary of our reinforcement learning environment. The tasks are navigational representatives for a broad class of human skills, which require particular sequences of strategic decisions, such as cooking, wayfinding, and problem solving.We train and test our agents in procedurally generated 3D worlds, containing colourful, spherical goals embedded in a noisy terrain full of obstacles. A player must navigate the goals in the correct order, which changes randomly on every episode. Since the order is impossible to guess, a naive exploration strategy incurs a large penalty. As a source of culturally transmitted information, we provide a privileged “bot” that always enters goals in the correct sequence.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6227d611c9968b617accf2a9_Fig%202.jpg)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6227d6414a3de27de2d3f161_Fig%203.jpg)Our MEDAL(-ADR) agent outperforms ablations on held-out tasks, in worlds without obstacles (top) and with obstacles (bottom).Via ablations, we identify a minimal sufficient "starter kit" of training ingredients required for cultural transmission to emerge, dubbed MEDAL-ADR. These components include memory (M), expert dropout (ED), attentional bias towards the expert (AL), and automatic domain randomization (ADR). Our agent outperforms the ablations, including the state-of-the-art method (ME-AL), across a range of challenging held-out tasks. Cultural transmission generalises out of distribution surprisingly well, and the agent recalls demonstrations long after the expert has departed. Looking into the agent's brain, we find strikingly interpretable neurons responsible for encoding social information and goal states.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6227d69116dd17585eae51a5_Fig%204.jpg)![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6227d69a721902e35c03584d_Fig%205.jpg)
Our agent generalises outside the training distribution (top) and possesses individual neurons that encode social information (bottom).

In summary, we provide a procedure for training an agent capable of flexible, high-recall, real-time cultural transmission, without using human data in the training pipeline. This paves the way for cultural evolution as an algorithm for developing more generally intelligent artificial agents.

This authors' notes is based on joint work by the Cultural General Intelligence Team: Avishkar Bhoopchand, Bethanie Brownfield, Adrian Collister, Agustin Dal Lago, Ashley Edwards, Richard Everett, Alexandre Fréchette, Edward Hughes, Kory W. Mathewson, Piermaria Mendolicchio, Yanko Oliveira, Julia Pawar, Miruna Pîslar, Alex Platonov, Evan Senter, Sukhdeep Singh, Alexander Zacherl, and Lei M. Zhang.

Read the full paper [here](https://arxiv.org/abs/2203.00715).


Probing Image-Language Transformers for Verb Understanding

Grounding language to vision is a fundamental problem for many real-world AI systems such as retrieving images or generating descriptions for the visually impaired. Success on these tasks requires models to relate different aspects of language such as objects and verbs to images. For example, to distinguish between the two images in the middle column below, models must differentiate between the verbs “catch” and “kick.” Verb understanding is particularly difficult as it requires not only recognising objects, but also how different objects in an image relate to each other. To overcome this difficulty, we introduce the SVO-Probes dataset and use it to probe language and vision models for verb understanding.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6233488fe173bc221c1df2ae_SVO-drawing2.jpg)In particular, we consider multimodal transformer models (e.g., Lu et al., 2019; Chen et al., 2020; Tan and Bansal, 2019; Li et al., 2020), which have shown success on a variety of language and vision tasks. However, despite strong performance on benchmarks, it is not clear if these models have fine-grained multimodal understanding. In particular, prior work shows that language and vision models can succeed at benchmarks without multimodal understanding: for example, answering questions about images based only on language priors (Agrawal et al., 2018) or “hallucinating” objects that are not in the image when captioning images (Rohrbach et al., 2018). To anticipate model limitations, work like Shekhar et al. propose specialised evaluations to probe models systematically for language understanding. However, prior probe sets are limited in the number of objects and verbs. We developed SVO-Probes to better evaluate potential limitations in verb understanding in current models.

SVO-Probes includes 48,000 image-sentence pairs and tests understanding for more than 400 verbs. Each sentence can be broken into a <Subject, Verb, Object> triplet (or SVO triplet) and paired with positive and negative example images. The negative examples differ in only one way: the Subject, Verb, or Object is changed. The figure above shows negative examples in which the subject (left), verb (middle), or object (right) does not match the image. This task formulation makes it possible to isolate which parts of the sentence a model has the most trouble with. It also makes SVO-Probes more challenging than standard image retrieval tasks, where negative examples are often completely unrelated to the query sentence.

To create SVO-Probes, we [query an image search](https://developers.google.com/custom-search/v1/reference/rest/v1/cse/list) with SVO triplets from a common training dataset, Conceptual Captions (Sharma et al. 2018). Because image search can be noisy, a preliminary annotation step filters the retrieved images to ensure we have a clean set of image-SVO pairs. Since transformers are trained on image-sentence pairs, not image-SVO pairs, we need image-sentence pairs to probe our model. To collect sentences which describe each image, annotators write a short sentence for each image that includes the SVO triplet. For example, given the SVO triplet <animal, lie, grass>, an annotator could write the sentence “An animal lays in the grass.” We then use the SVO annotations to pair each sentence with a negative image, and ask annotators to verify negatives in a final annotation step. See the figure below for details.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623348fb22a815412339f8b5_SVO-drawing3.jpg)We examine whether multimodal transformers can accurately classify examples as positive or negative. The bar chart below illustrates our results. Our dataset is challenging: our standard multimodal transformer model achieves 64.3% accuracy overall (chance is 50%). Whereas accuracy is 67.0% and 73.4% on subjects and objects respectively, performance falls to 60.8% on verbs. This result shows that verb recognition is indeed challenging for vision and language models.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623349754dd07eedab303af7_SVO-drawing1.jpg)We also explore which model architectures perform best on our dataset. Surprisingly, models with weaker image modeling perform better than the standard transformer model. One hypothesis is that our standard model (with stronger image modeling ability) overfits the train set. As both these models perform worse on other language and vision tasks, our targeted probe task illuminates model weaknesses that are not observed on other benchmarks.

Overall, we find that despite impressive performance on benchmarks, multimodal transformers still struggle with fine-grained understanding, especially fine-grained verb understanding. We hope SVO-Probes can help drive exploration of verb understanding in language and vision models and inspire more targeted probe datasets.

Visit our SVO-Probes [benchmark](https://github.com/deepmind/multimodal_transformers) and [models](https://github.com/deepmind/svo_probes) on GitHub: benchmark and models.


Red Teaming Language Models with Language Models

**In our** [**recent paper**](https://arxiv.org/abs/2202.03286)**, we show that it is possible to automatically find inputs that elicit harmful text from language models by generating inputs using language models themselves. Our approach provides one tool for finding harmful model behaviours before users are impacted, though we emphasize that it should be viewed as one component alongside many other techniques that will be needed to find harms and mitigate them once found.**

Large generative language models like GPT-3 and Gopher have a remarkable ability to generate high-quality text, but they are difficult to deploy in the real world. Generative language models come with a risk of generating very harmful text, and even a small risk of harm is unacceptable in real-world applications.

For example, in 2016, Microsoft released the Tay Twitter bot to automatically tweet in response to users. Within 16 hours, [Microsoft took Tay down](https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist) after several adversarial users elicited racist and sexually-charged tweets from Tay, which were sent to over 50,000 followers. The outcome was [not for lack of care on Microsoft’s part](https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/):

> "Although we had prepared for many types of abuses of the system, we had made a critical oversight for this specific attack."
>
> Peter Lee
> VP, Microsoft

The issue is that there are so many possible inputs that can cause a model to generate harmful text. As a result, it’s hard to find all of the cases where a model fails before it is deployed in the real world. Previous work relies on paid, human annotators to manually discover failure cases ([Xu et al. 2021](https://aclanthology.org/2021.naacl-main.235/), *inter alia*). This approach is effective but expensive, limiting the number and diversity of failure cases found.

We aim to complement manual testing and reduce the number of critical oversights by finding failure cases (or ‘red teaming’) in an automatic way. To do so, we generate test cases using a language model itself and use a classifier to detect various harmful behaviors on test cases, as shown below:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62334badb21bcaf518e14447_red-teaming.jpg)Our approach uncovers a variety of harmful model behaviors:

1. **Offensive Language**: Hate speech, profanity, sexual content, discrimination, etc.
2. **Data Leakage**: Generating copyrighted or private, personally-identifiable information from the training corpus.
3. **Contact Information Generation**: Directing users to unnecessarily email or call real people.
4. **Distributional Bias**: Talking about some groups of people in an unfairly different way than other groups, on average over a large number of outputs.
5. **Conversational Harms**: Offensive language that occurs in the context of a long dialogue, for example.

To generate test cases with language models, we explore a variety of methods, ranging from prompt-based generation and few-shot learning to supervised finetuning and reinforcement learning. Some methods generate more diverse test cases, while other methods generate more difficult test cases for the target model. Together, the methods we propose are useful for obtaining high test coverage while also modeling adversarial cases.

Once we find failure cases, it becomes easier to fix harmful model behavior by:

1. Blacklisting certain phrases that frequently occur in harmful outputs, preventing the model from generating outputs that contain high-risk phrases.
2. Finding offensive training data quoted by the model, to remove that data when training future iterations of the model.
3. Augmenting the model’s prompt (conditioning text) with an example of the desired behavior for a certain kind of input, as shown in our [recent work](https://deepmind.com/blog/article/language-modelling-at-scale).
4. Training the model to [minimize the likelihood](https://arxiv.org/abs/1908.04319) of its original, harmful output for a given test input.

Overall, language models are a highly effective tool for uncovering when language models behave in a variety of undesirable ways. In our current work, we focused on red teaming harms that today’s language models commit. In the future, our approach can also be used to preemptively discover other, hypothesized harms from advanced machine learning systems, e.g., due to [inner misalignment](https://arxiv.org/abs/1906.01820) or [failures in objective robustness](https://arxiv.org/abs/2105.14111). This approach is just one component of responsible language model development: we view red teaming as one tool to be used alongside many others, both to find harms in language models and to mitigate them. We refer to Section 7.3 of [Rae et al. 2021](https://arxiv.org/abs/2112.11446) for a broader discussion of other work needed for language model safety.

For more details on our approach and results, as well as the broader consequences of our findings, read our [red teaming paper](https://arxiv.org/abs/2202.03286) here.

Ethan Perez
Saffron Huang
Francis Song
Trevor Cai
Roman Ring
John Aslanides
Amelia Glaese
Nat McAleese
Geoffrey Irving


Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents

**In** [**our recent paper**](https://www.pnas.org/content/119/3/e2106028118) **we explore how multi-agent deep reinforcement learning can serve as a model of complex social interactions, like the formation of social norms. This new class of models could provide a path to create richer, more detailed simulations of the world.**

Humans are an [ultra social species](https://press.uchicago.edu/ucp/books/book/chicago/N/bo3615170.html). Relative to other mammals we benefit more from cooperation but we are also more dependent on it, and face greater cooperation challenges. Today, humanity faces numerous cooperation challenges including preventing conflict over resources, ensuring everyone can access clean air and drinking water, eliminating extreme poverty, and combating climate change. Many of the cooperation problems we face are difficult to resolve because they involve complex webs of social and biophysical interactions called [social-ecological systems](https://doi.org/10.1126/science.1172133). However, humans can collectively learn to overcome the cooperation challenges we face. We accomplish this by an ever evolving culture, including norms and institutions which organize our interactions with the environment and with one another.

However, norms and institutions sometimes fail to resolve cooperation challenges. For example, individuals may over-exploit resources like forests and fisheries thereby causing them to collapse. In such cases, policy-makers may write laws to change institutional rules or develop other [interventions to try to change norms](https://oxford.universitypressscholarship.com/view/10.1093/acprof:oso/9780190622046.001.0001/acprof-9780190622046) in hopes of bringing about a positive change. But policy interventions do not always work as intended. This is because real-world social-ecological systems are considerably [more complex](https://doi.org/10.1038/s41893-019-0419-7) than the models we typically use to try to predict the effects of candidate policies.

Models based on game theory are often applied to the study of cultural evolution. In most of these models, the key interactions that agents have with one another are expressed in a ‘payoff matrix’. In a game with two participants and two actions A and B, a payoff matrix defines the value of the four possible outcomes: (1) we both choose A, (2) we both choose B, (3) I choose A while you choose B and (4) I choose B while you choose A. The most famous example is the ‘Prisoner’s Dilemma’, in which the actions are interpreted as “cooperate” and “defect”. Rational agents who act according to their own myopic self-interest are doomed to defect in the Prisoner’s Dilemma even though the better outcome of mutual cooperation is available.

Game-theoretic models have been very widely applied. Researchers in diverse fields have used them to study a wide range of different phenomena, including economies and the evolution of human culture. However, game theory is not a neutral tool, rather it is a deeply opinionated modeling language. It imposes a strict requirement that everything must ultimately cash out in terms of the payoff matrix (or equivalent representation). This means that the modeler has to know, or be willing to assume, everything about how the effects of individual actions combine to generate incentives. This is sometimes appropriate, and the game theoretic approach has had many notable successes such as in modeling the [behavior of oligopolistic firms](https://doi.org/10.1016/S0899-8256(03)00114-3) and [cold war era international relations](https://yalebooks.yale.edu/book/9780300143379/arms-and-influence). However, game theory’s major weakness as a modeling language is exposed in situations where the modeler does not fully understand how the choices of individuals combine to generate payoffs. Unfortunately this tends to be the case with social-ecological systems because their social and ecological parts interact in complex ways that we do not fully understand.

The work we present here is one example within a research program that attempts to establish an alternative modeling framework, different from game theory, to use in the study of social-ecological systems. Our approach may be seen formally as a variety of [agent-based modeling](https://books.google.co.uk/books?id=Zrh2DwAAQBAJ&lpg=PP1&ots=OBPL7cpf_k&dq=agent%20based%20modeling%20textbook&lr&pg=PP1#v=onepage&q=agent%20based%20modeling%20textbook&f=false). However, its distinguishing feature is the incorporation of algorithmic elements from artificial intelligence, especially multi-agent deep reinforcement learning.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62334e665a58701fbf639fb2_unnamed.gif)The core idea of this approach is that every model consists of two interlocking parts: (1) a rich, dynamical model of the environment and (2) a model of individual decision-making.

The first takes the form of a researcher-designed simulator: an interactive program that takes in a current environment state and agent actions, and outputs the next environment state as well as the observations of all agents and their instantaneous rewards. The model of individual decision-making is likewise conditioned on environment state. It is an *agent* that learns from its past experience, performing a form of trial-and-error. An agent interacts with an environment by taking in observations and outputting actions. Each agent selects actions according to its behavioral policy, a mapping from observations to actions. Agents learn by changing their policy to improve it along any desired dimension, typically to obtain more reward. The policy is stored in a neural network. Agents learn ‘from scratch’, from their own experience, how the world works and what they can do to earn more rewards. They accomplish this by tuning their network weights in such a way that the pixels they receive as observations are gradually transformed into competent actions. Several learning agents can inhabit the same environment as one another. In this case the agents become interdependent because their actions affect one another.

Like other agent-based modeling approaches, multi-agent deep reinforcement learning makes it easy to specify models that cross levels of analysis that would be hard to treat with game theory. For instance, actions may be far closer to low-level motor primitives (e.g. 'walk forward'; 'turn right') than the high-level strategic decisions of game theory (e.g. ‘cooperate’). This is an important feature needed to capture situations where agents must practice to learn effectively how to [implement their strategic choices](https://arxiv.org/pdf/1702.03037.pdf). For instance in one [study](https://arxiv.org/abs/2103.04982), agents learned to cooperate by taking turns cleaning a river. This solution was only possible because the environment had spatial and temporal dimensions in which agents have great freedom in how they structure their behavior towards one another. Interestingly, while the environment allowed for many different solutions (such as [territoriality](https://arxiv.org/pdf/1707.06600.pdf)), agents converged on the same turn-taking solution as human players.

In our latest study, we applied this type of model to an open question in research on cultural evolution: how to explain the existence of spurious and arbitrary social norms that appear not to have immediate material consequences for their violation beyond those imposed socially. For instance, in some societies men are expected to wear trousers not skirts; in many there are words or hand gestures that should not be used in polite company; and in most there are rules about how one styles one's hair or what one wears on one's head. We call these social norms ‘silly rules’. Importantly, in our framework, enforcing and complying with social norms both have to be learned. Having a social environment that includes a ‘silly rule’ means that agents have more opportunities to learn about enforcing norms in general. This additional practice then allows them to enforce the important rules more effectively. Overall, the ‘silly rule’ can be beneficial for the population – a surprising result. This result is only possible because our simulation focuses on learning: enforcing and complying with rules are complex skills that need training to develop.

Part of why we find this result on silly rules so exciting is that it demonstrates the utility of multi-agent deep reinforcement learning in modeling cultural evolution. Culture contributes to the success or failure of policy interventions for socio-ecological systems. For instance, strengthening social norms around recycling is part of the [solution](https://www.science.org/doi/10.1126/science.aaf8317) to some environmental problems. Following this trajectory, richer simulations could lead to a deeper understanding of how to design interventions for social-ecological systems. If simulations become realistic enough, it may even be possible to test the impact of interventions, e.g. aiming to [design a tax code that fosters productivity and fairness](https://arxiv.org/abs/2108.02755).

This approach provides researchers with tools to specify detailed models of phenomena that interest them. Of course, like all research methodologies it should be expected to come with its own strengths and weaknesses. We hope to discover more about when this style of modeling can be fruitfully applied in the future. While there are no panaceas for modeling, we think there are compelling reasons to look to multi-agent deep reinforcement learning when constructing models of social phenomena, especially when they involve learning.

Raphael Koster
Dylan Hadfield-Menell *
Richard Everett
Laura Weidinger
G Hadfield *
Joel Leibo


Improving language models by retrieving from trillions of tokens

In recent years, significant performance gains in autoregressive language modeling have been achieved by increasing the number of parameters in Transformer models. This has led to a tremendous increase in training energy cost and resulted in a generation of dense “Large Language Models” (LLMs) with 100+ billion parameters. Simultaneously, large datasets containing trillions of words have been collected to facilitate the training of these LLMs.

We explore an alternate path for improving language models: we augment transformers with retrieval over a database of text passages including web pages, books, news and code. We call our method RETRO, for “Retrieval Enhanced TRansfOrmers”.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6233547661f4d80dd32aafbb_LLM_figure_5_and_1.jpg)Figure 1: A high-level overview of Retrieval Enhanced TransfOrmers (RETRO).In traditional transformer language models, the benefits of model size and data size are linked: as long as the dataset is large enough, language modeling performance is limited by the size of the model. However, with RETRO the model is not limited to the data seen during training– it has access to the entire training dataset through the retrieval mechanism. This results in significant performance gains compared to a standard Transformer with the same number of parameters. We show that language modeling improves continuously as we increase the size of the retrieval database, at least up to 2 trillion tokens – 175 full lifetimes of continuous reading.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623354cb7ba0efb9ef468ebc_improving.jpg)Figure 2: Increasing the size of the retrieval dataset results in large gains in model performance.For each text passage (approximately a paragraph of a document), a nearest-neighbor search is performed which returns similar sequences found in the training database, and their continuation. These sequences help predict the continuation of the input text. The RETRO architecture interleaves regular self-attention at a document level and cross-attention with retrieved neighbors at a finer passage level. This results in both more accurate and more factual continuations.  Furthermore, RETRO increases the interpretability of model predictions, and provides a route for direct interventions through the retrieval database to improve the safety of text continuation. In our experiments on the Pile, a standard language modeling benchmark, a 7.5 billion parameter RETRO model outperforms the 175 billion parameter Jurassic-1 on 10 out of 16 datasets and outperforms the 280B Gopher on 9 out of 16 datasets.

Below, we show two samples from our 7B baseline model and from our 7.5B RETRO model model that highlight how RETRO’s samples are more factual and stay more on topic than the baseline sample.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6233553fd5cc337484139079_creative2.jpg)Figure 3: The baseline only generates 2 correct digits. With RETRO, the correct digits are generated after being retrieved by the database.![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335571d5cc33e8cd13a110_creative3.jpg)Figure 4: The RETRO model stays more on-topic than the baseline sample.Type image caption here (optional)

Sebastian Borgeaud
Arthur Mensch
Jordan Hoffmann
Laurent Sifre


Creating Interactive Agents with Imitation Learning

Humans are an interactive species. We interact with the physical world and with one another. For artificial intelligence (AI) to be generally helpful, it must be able to interact capably with humans and their environment. In this work we present the Multimodal Interactive Agent (MIA), which blends visual perception, language comprehension and production, navigation, and manipulation to engage in extended and often surprising physical and linguistic interactions with humans.

![](https://assets-global.website-files.com/img/image-placeholder.svg)We build upon the approach introduced by Abramson et al. (2020), which primarily uses imitation learning to train agents. After training, MIA displays some rudimentary intelligent behaviour that we hope to later refine using human feedback. This work focuses on the creation of this intelligent behavioural prior, and we leave further feedback-based learning for future work.

We created the Playhouse environment, a 3D virtual environment composed of a randomised set of rooms and a large number of domestic interactable objects, to provide a space and setting for humans and agents to interact together. Humans and agents can interact in the Playhouse by controlling virtual robots that locomote, manipulate objects, and communicate via text. This virtual environment permits a wide range of situated dialogues, ranging from simple instructions (e.g., “Please pick up the book from the floor and place it on the blue bookshelf”) to creative play (e.g., “Bring food to the table so that we can eat”).

We collected human examples of Playhouse interactions using language games, a collection of cues prompting humans to improvise certain behaviours. In a language game one player (the setter) receives a prewritten prompt indicating a kind of task to propose to the other player (the solver). For example, the setter might receive the prompt “Ask the other player a question about the existence of an object,'' and after some exploration, the setter could ask, ”Please tell me whether there is a blue duck in a room that does not also have any furniture.'' To ensure sufficient behavioural diversity, we also included free-form prompts, which granted setters free choice to improvise interactions (E.g. “Now take any object that you like and hit the tennis ball off the stool so that it rolls near the clock, or somewhere near it.''). In total, we collected 2.94 years of real-time human interactions in the Playhouse.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335136819da8ccaf83a63b_converted-unnamed%20(2).jpg)Example of two humans interacting in the Playhouse.Our training strategy is a combination of supervised prediction of human actions (behavioural cloning) and self-supervised learning. When predicting human actions, we found that using a hierarchical control strategy significantly improved agent performance. In this setting, the agent receives new observations roughly 4 times per second. For each observation, it produces a sequence of open-loop movement actions and optionally emits a sequence of language actions. In addition to behavioural cloning we use a form of self-supervised learning, which tasks agents with classifying whether certain vision and language inputs belong to the same or different episodes.

To evaluate agent performance, we asked human participants to interact with agents and provide binary feedback indicating whether the agent successfully carried out an instruction. MIA achieves over 70% success rate in human-rated online interactions, representing 75% of the success rate that humans themselves achieve when they play as solvers. To better understand the role of various components in MIA, we performed a series of ablations, removing, for example, visual or language inputs, the self-supervised loss, or the hierarchical control.

Contemporary machine learning research has uncovered remarkable regularities of performance with respect to different scale parameters; in particular, model performance scales as a power-law with dataset size, model size, and compute. These effects have been most crisply noted in the language domain, which is characterised by massive dataset sizes and highly evolved architectures and training protocols. In this work, however, we are in a decidedly different regime – with comparatively small datasets and multimodal, multi-task objective functions training heterogeneous architectures. Nevertheless, we demonstrate clear effects of scaling: as we increase dataset and model size, performance increases appreciably.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/6233525b4690fbf4c4ff6dbe_creative1.jpg)Scripted probe tasks performance and human evaluation for data and model scaling. In both cases performance improvements when increasing both dataset size and model size.

In an ideal case, training becomes more efficient given a reasonably large dataset, as knowledge is transferred between experiences. To investigate how ideal our circumstances are, we examined how much data is needed to learn to interact with a new, previously unseen object and to learn how to follow a new, previously unheard command / verb. We partitioned our data into background data and data involving a language instruction referring to the object or the verb. When we reintroduced the data referring to the new object, we found that fewer than 12 hours of human interaction was enough to acquire the ceiling performance. Analogously, when we introduced the new command or verb ‘to clear’ (i.e. to remove all objects from a surface), we found that only 1 hour of human demonstrations was enough to reach ceiling performance in tasks involving this word.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/62335317c41c26f04bb44896_creative2.jpg)When learning a new command or object, the agent’s performance quickly improves with mere hours of demonstration experience.MIA exhibits startlingly rich behaviour, including a diversity of behaviours that were not preconceived by researchers, including tidying a room, finding multiple specified objects, and asking clarifying questions when an instruction is ambiguous. These interactions continually inspire us. However, the open-endedness of MIA’s behaviour presents immense challenges for quantitative evaluation. Developing comprehensive methodologies to capture and analyse open-ended behaviour in human-agent interactions will be an important focus in our future work.

For a more detailed description of our work, see our [paper](https://arxiv.org/abs/2112.03763).

Josh Abramson
Arun Ahuja
Arthur Brussee
Federico Carnevale
Mary Cassin
Felix Fischer
Petko Georgiev
Alex Goldin
Tim Harley
Felix Hill
Peter C Humphreys
Alden Hung
Jessica Landon
Timothy Lillicrap
Hamza Merzic
Alistair Muldal
Adam Santoro
Guy Scully
Tamara von Glehn
Gregory Wayne
Nathaniel Wong
Chen Yan
Rui Zhu
Mary Cassin
Hamza Merzic


On the Expressivity of Markov Reward

Reward is the driving force for reinforcement learning (RL) agents. Given its central role in RL, reward is often assumed to be suitably general in its expressivity, as summarized by Sutton and Littman’s reward hypothesis:

> "...all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward)."
>
> **- SUTTON (2004), LITTMAN (2017)**

In our work, we take first steps toward a systematic study of this hypothesis. To do so, we consider the following thought experiment involving Alice, a designer, and Bob, a learning agent:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623357d30a672579f94903f6_markov%203.jpg)We suppose that Alice thinks of a task she might like Bob to learn to solve – this task could be in the form a a natural language description (“balance this pole”), an imagined state of affairs (“reach any of the winning configurations of a chess board”), or something more traditional like a reward or value function. Then, we imagine Alice translates her choice of task into some generator that will provide learning signal (such as reward) to Bob (a learning agent), who will learn from this signal throughout his lifetime. We then ground our study of the reward hypothesis by addressing the following question: given Alice’s choice of task, is there always a reward function that can convey this task to Bob?

#### What is a task?

To make our study of this question concrete, we first restrict focus to three kinds of task. In particular, we introduce three task types that we believe capture sensible kinds of tasks: 1) A set of acceptable policies (SOAP), 2) A policy order (PO), and 3) A trajectory order (TO). These three forms of tasks represent concrete instances of the kinds of task we might want an agent to learn to solve.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623357b62faa374cc0174395_markov%20(1).jpg)We then study whether reward is capable of capturing each of these task types in finite environments. Crucially, we only focus attention on Markov reward functions; for instance, given a state space that is sufficient to form a task such as (x,y) pairs in a grid world, is there a reward function that only depends on this same state space that can capture the task?

#### First Main Result

Our first main result shows that for each of the three task types, there are environment-task pairs for which there is no Markov reward function that can capture the task. One example of such a pair is the “go all the way around the grid clockwise or counterclockwise” task in a typical grid world:

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623357e8ab00de441f317409_markov%20(2).jpg)This task is naturally captured by a SOAP that consists of two acceptable policies: the “clockwise” policy (in blue) and the “counterclockwise” policy (in purple). For a Markov reward function to express this task, it would need to make these two policies strictly higher in value than all other deterministic policies. However, there is no such Markov reward function: the optimality of a single “move clockwise” action will depend on whether the agent was already moving in that direction in the past. Since the reward function must be Markov, it cannot convey this kind of information. Similar examples demonstrate that Markov reward cannot capture every policy order and trajectory order, too.

#### Second Main Result

Given that some tasks can be captured and some cannot, we next explore whether there is an efficient procedure for determining whether a given task can be captured by reward in a given environment. Further, if there is a reward function that captures the given task, we would ideally like to be able to output such a reward function. Our second result is a positive result which says that for any finite environment-task pair, there is a procedure that can 1) decide whether the task can be captured by Markov reward in the given environment, and 2) outputs the desired reward function that exactly conveys the task, when such a function exists.

This work establishes initial pathways toward understanding the scope of the reward hypothesis, but there is much still to be done to generalize these results beyond finite environments, Markov rewards, and simple notions of “task” and “expressivity”. We hope this work provides new conceptual perspectives on reward and its place in reinforcement learning.

David Abel
Doina Precup
Anna Harutyunyan
Mark Ho *
Michael Littman *
Will Dabney
Satinder Baveja


Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons

Our brain has an amazing ability to process visual information. We can take one glance at a complex scene, and within milliseconds be able to parse it into objects and their attributes, like colour or size, and use this information to describe the scene in simple language. Underlying this seemingly effortless ability is a complex computation performed by our visual cortex, which involves taking millions of neural impulses transmitted from the retina and transforming them into a more meaningful form that can be mapped to the simple language description. In order to fully understand how this process works in the brain, we need to figure out both how the semantically meaningful information is represented in the firing of neurons at the end of the visual processing hierarchy, and how such a representation may be learnt from largely untaught experience.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623358932faa37a01b17d6b6_unnamed.gif)Figure 1. Disentangling refers to the ability of neural networks to discover semantically meaningful attributes of images without being explicitly taught what these attributes are. These models learn by mapping images into a lower-dimensional representation through an inference neural network, and trying to reconstruct the image using a generation neural network. Each individual latent unit in a disentangled representation learns to encode a single interpretable attribute, like colour or size of an object. Manipulating such latents one at a time results in interpretable changes in the generated image reconstruction. Animation credit Chris Burgess.To answer these questions in the context of face perception, we joined forces with our collaborators at Caltech ([Doris Tsao](https://www.tsaolab.caltech.edu/)) and the Chinese Academy of Science ([Le Chang](http://english.cebsit.cas.cn/)). We chose faces because they are well studied in the neuroscience community and are often seen as a “[microcosm of object recognition](https://pubmed.ncbi.nlm.nih.gov/18558862/)”. In particular, we wanted to compare the responses of single cortical neurons in the face patches at the end of the visual processing hierarchy, recorded by our collaborators to a recently emerged class of so called  “disentangling” deep neural networks that, unlike the usual “black box” systems, explicitly aim to be interpretable to humans. A “disentangling” neural network learns to map complex images into a small number of internal neurons (called latent units), each one representing a single semantically meaningful attribute of the scene, like colour or size of an object (see Figure 1). Unlike the “black box” deep classifiers trained to recognise visual objects through a biologically unrealistic amount of external supervision, such disentangling models are trained without an external teaching signal using a self-supervised objective of reconstructing input images (generation in Figure 1) from their learnt latent representation (obtained through inference in Figure 1).

Disentangling was [hypothesised](https://arxiv.org/pdf/1305.0445v2.pdf) to be important in the machine learning community almost ten years ago as an integral component for building more [data-efficient](https://arxiv.org/abs/1911.10866), [transferable](https://arxiv.org/abs/1707.08475), [fair](https://arxiv.org/pdf/2002.02886.pdf), and [imaginative](https://deepmind.com/blog/article/imagine-creating-new-visual-concepts-recombining-familiar-ones) artificial intelligence systems. However, for years, building a model that can disentangle in practice has eluded the field. The first model able to do this successfully and robustly, called [β-VAE](https://openreview.net/references/pdf?id=Sy2fzU9gl), was developed by taking [inspiration from neuroscience](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3306444/#:~:text=Mounting%20evidence%20suggests%20that%20%E2%80%9Ccore,in%20the%20inferior%20temporal%20cortex.): β-VAE learns by [predicting its own inputs](https://www.nature.com/articles/s41593-018-0200-7); it requires similar visual experience for successful learning as [that encountered by babies](https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(18)30027-5); and its learnt latent representation mirrors the [properties known of the visual brain](https://www.nature.com/articles/s41593-019-0377-4).

In our [new paper](https://www.nature.com/articles/s41467-021-26751-5), we measured the extent to which the disentangled units discovered by a β-VAE trained on a dataset of face images are similar to the responses of single neurons at the end of the visual processing recorded in primates looking at the same faces. The neural data was collected by our collaborators under rigorous oversight from the [Caltech Institutional Animal Care and Use Committee](https://researchcompliance.caltech.edu/committees/institutional-animal-care-and-use-committee-iacuc). When we made the comparison, we found something surprising - it seemed like the handful of disentangled units discovered by β-VAE were behaving as if they were equivalent to a similarly sized subset of the real neurons. When we looked closer, we found a strong one-to-one mapping between the real neurons and the artificial ones (see Figure 2). This mapping was much stronger than that for alternative models, including the deep classifiers previously considered to be state of the art computational models of visual processing, or a hand-crafted model of face perception seen as the “gold standard” in the neuroscience community. Not only that, β-VAE units were encoding semantically meaningful information like age, gender, eye size, or the presence of a smile, enabling us to understand what attributes single neurons in the brain use to represent faces.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623358f6c03d631aa459a833_converted-unnamed%20(3).jpg)Figure 2. Single neurons in the primate face patches at the end of the visual processing hierarchy represent interpretable face attributes, like eye shape or the presence of a smile, and are equivalent to single artificial neurons in β-VAE discovered through disentangled representation learning. Image credit Marta Garnelo.If β-VAE was indeed able to automatically discover artificial latent units that are equivalent to the real neurons in terms of how they respond to face images, then it should be possible to translate the activity of real neurons into their matched artificial counterparts, and use the generator (see Figure 1) of the trained β-VAE to visualise what faces the real neurons are representing. To test this, we presented the primates with new face images that the model has never experienced, and checked if we could render them using the β-VAE generator (see Figure 3). We found that this was indeed possible. Using the activity of as few as 12 neurons, we were able to generate face images that were more accurate reconstructions of the originals and of better visual quality than those produced by the alternative deep generative models. This is despite the fact that the alternative models are known to be better image generators than β-VAE in general.

![](https://assets-global.website-files.com/621e749a546b7592125f38ed/623359325d31d8716bb08212_converted-unnamed%20(4).jpg)Figure 3. Face images were accurately reconstructed by the trained β-VAE generator from the activity of 12 one-to-one matched neurons in the primate visual cortex as the primates were viewing novel faces. Novel face images reproduced with permission from Ma et al. and Phillips et al.Our findings summarised in the [new paper](https://www.nature.com/articles/s41467-021-26751-5) suggest that the visual brain can be understood at a single-neuron level, even at the end of its processing hierarchy. This is contrary to the common belief that semantically meaningful information is [multiplexed between a large number of such neurons](https://www.sciencedirect.com/science/article/abs/pii/S0959438818300990), each one remaining largely uninterpretable individually, not unlike how information is encoded across full layers of artificial neurons in deep classifiers. Not only that, our findings suggest that it is possible that the brain learns to support our effortless ability to do visual perception by optimising the disentanglement objective. While β-VAE was originally developed with inspiration from [high-level neuroscience principles](https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(18)30027-5), the utility of disentangled representations for intelligent behaviour has so far been primarily demonstrated in the [machine-learning community](https://arxiv.org/pdf/2002.02886.pdf). In line with the rich history of mutually beneficial [interactions between neuroscience and machine learning](https://www.cell.com/neuron/pdf/S0896-6273(17)30509-3.pdf), we hope that the latest insights from machine learning may now feed back to the neuroscience community to investigate the merit of disentangled representations for supporting intelligence in biological systems, in particular as the basis for [abstract reasoning](https://www.science.org/doi/10.1126/science.aat6766), or generalisable and efficient [task learning](https://www.nature.com/articles/s41593-019-0470-8).

Irina Higgins
L Chang*
V Langston
Demis Hassabis
Christopher Summerfield
Doris Tsao*
Matt Botvinick