Import AI 320: Facebook's AI Lab Leak; open source ChatGPT clone; Google makes a universal translator.

Welcome to Import AI, a (mostly) weekly newsletter about AI research and development. The issues are free, but paid subscribers will get access to special analysis pieces before anyone else. Founding members can help support me further and fund special projects and other *top secret Import AI initiatives!* Thanks for reading!

[Subscribe now](https://importai.substack.com/subscribe)

**Google makes progress on the self-teaching universal translator:***…Universal Speech Models scale beyond 100 languages…*Google has built a family of AI systems called Universal Speech Models (USMs). These models are designed to do speech recognition on more than 100+ languages. The main model is 2B parameters and was trained on a large unlabeled multilingual dataset of 12 million hours spanning over 300 languages.

**The goal of USM:** "Our long-term goal is to train a universal ASR model that covers all the spoken

Import AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

languages in the world," Google writes. USMs are Google exploring " a promising direction where large amounts of unpaired multilingual speech and text data and smaller amounts of transcribed data can contribute to training a single large universal ASR model."

**The key ingredient? The data mix:** Much like baking a cake, training predominantly self-supervised models requires the right mix of data. Here, Google uses the following components:

* Unpaired Audio: 12 million hours of YouTube-based audio covering over 300 languages, and  429k hours of unlabeled speech in 51 languages based on public datasets.
* Unpaired Text:28billion sentences spanning over 1140 languages.
* Paired audio speech recognition data: 90k hours of labeled multilingual data covering 73 languages, plus 10k hours of labeled multi-domain en-US public data, plus 10k labeled multilingual public data covering 102 languages.

**What they did:** The steps to build a universal ASR model are quite complex, so it's worth reading rhe paper for full details. First they do unsupervised pre-training to pre-train the encoder of the model with the YouTube dataset, then they use a process called multi-objective supervised pre-training across the other unpaired audio and text data, then for some models that do supervised ASR training.

**What the results were**: In tests, these USM models "achieve state-of-the art performance for multilingual ASR and AST for multiple datasets in multiple domains." They also out-perform OpenAI's notoriously good (and open source!) 'Whisper' models as well; an impressive achievement given that Whisper set a new state-of-the-art in multiple areas when it came out.

**Why this matters**: "We believe diverse unlabeled data is more practical to acquire for building usable ASR for tail languages than weakly labeled data," Google says. In other words; if you want to translate the entire world then it's better to just hoover up data at scale rather than invest in trying to produce a small amount of minimally labeled datasets. This generally points in the direction of 'gotta grab em all' with regard to trawling the web and other sources for data. This is somewhat intriguing as while Google has a bunch of data sources and competent language modeling teams, it's fairly likely that having a universal translator is also interesting to government types - some of which are thought to be able to access larger sources of data through various clandestine means.
   **Read more:** [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (arXiv)](https://arxiv.org/abs/2303.01037).
   **Request [API access here](https://sites.research.google/usm/)**[.](https://sites.research.google/usm/)

####################################################

**US regulator: Hey, maybe don't lie about your AI products:***…Sometimes it's worth stating the simple and obvious thing…*The Federal Trade Commission has published a blogpost called 'Keep your AI claims in check'. The post is a sensible summary of how as AI becomes increasingly hyped up, people will be tended to write a lot of bullshit about AI. The FTC notes in its post that it will be paying attention to companies that:

* Exaggerate what AI products can do.
* Promising an AI product is far superior to a non-AI product without providing evidence.
* Underinvesting in analyzing the risks of their products.
* Baselessly labeling something as AI when it does not, in fact, use AI.

**Why this matters:** Sometimes it's helpful for powerful regulators to state the painfully obvious - bravo to the FTC for reminding people in these hyped-up times that lying and bullshitting about AI (or any technology, really) is irresponsible. It'll be interesting to see in the coming months if the FTC takes any actions against egregious liars and hypers in this space.
**Read more**: [Keep your AI claims in check (Federal Trade Commission)](https://www.ftc.gov/business-guidance/blog/2023/02/keep-your-ai-claims-check).

####################################################

**ROBOLLM: Google shows how if you mush together more sensory inputs into an LLM, you get a lot of transfer learning:***…Maybe everything really is a sequence prediction task…*Google has built PaLM-E, a 562B parameter model which mushes together a 540B LLM and a 22B Vision Transformer (ViT). Crucially, PaLM-E sees Google "directly incorporate continuous inputs from sensor modalities of an embodied agent and thereby enable the language model *itself* to make more grounded inferences for sequential decision making in the real world". The result is a language model that can help robots carry out real tasks in the real world, and also is another triumphant demonstration of how bigger models with more diverse data sources generally get way better at doing a bunch of things.

**What PaLM-E is:** "The main architectural idea of PaLM-E is to inject continuous, embodied observations such as images, state estimates, or other sensor modalities into the language embedding space of a pre-trained language model," Google writes. "The inputs to PaLM-E consist of text and (multiple) continuous observations. The multimodal tokens corresponding to these observations are interleaved with the text to form multi-modal sentences. An example of such a multi-modal sentence is Q: What happened between and ? where represents an embedding of an image. The output of PaLM-E is text generated auto-regressively by the model, which could be an answer to a question, or a sequence of decisions produced by PaLM-E in textual form that should be executed by a robot".

**Why PaLM-E is a big deal:** In tests, Google applies PaLM-E to three different robotics tasks which use somewhat different types of data; these tasks include Task and Motion Planning (TAMP), a task called Language-Table, and a mobile manipulation domain based on Google's earlier 'SayCan' research. PaLM-E can do ok at these tasks individually but the magic happens when you mush all of the training datasets into it together: "Across three different robotics domains, using PaLM and ViT pretraining together with the full mixture of robotics and general visual-language data provides a significant performance increase compared to only training on the respective in-domain data."

   In other words, by adding more diverse heterogenous data sources into PaLM-E, Google improves the ability of the resulting model to generalize knowledge across distinct domains and modalities. Even more intriguingly, as they scale up the model complexity and the diversity of data sources, they don't see much catastrophic forgetting of language capabilities - so by adding the robot stuff, they don't cripple the language model.

**Why this matters - I am going to tap the 'these things are getting smarter' sign and stare at you:** "PaLM-E-562B exhibits a wide array of capabilities including zero-shot multimodal chain-of-thought (CoT) reasoning, few-shot prompting, OCR-free math reasoning, and multi-image reasoning, despite being trained on only single-image examples", Google writes.
   In other words, by doing this large-scale training, Google creates a model that displays *emergent capabilities* and these capabilities are *more complex than the input data.* Systems like PaLM-E represent the thrilling and vaguely terrifying state of AI in 2023 - we train unprecedentedly large models and force as many different data types into a single embedding space as possible, get the thing to try and do a simple (albeit very large-scale) sequence prediction task, and out pops something with way more capabilities than we'd naively anticipate.
   "A generalist, transfer-learned, multi-embodiment decision-making agent can be trained via mixing in embodied data into the training of a multimodal large language model", Google writes.
   **Read more**: [PaLM-E: An Embodied Multimodal Language Model (PDF)](https://palm-e.github.io/assets/palm-e.pdf).

####################################################

**You can run a powerful LM on an M2 MacBook now:***…Facebook's AI lab leak brings about the dawn of demoscene AI…*Two of Facebook's leaked LLaMa models can be run on an M2 MacBook, according to Simon Willison. This marks the dawn of what I'd call Demoscene AI - an era where people take the latest and greatest AI models and do a bunch of arcane software witchcraft to fit them onto consumer devices. This is part of the broader story of centralization VS decentralization in AI; once you can run models on a laptop it's basically 'game over' from a control-regulation perspective, and it seems like language models have crossed that rubicon.

**What you can do and how:** The weights for LLaMA are a mere 240GB download (combining the 7B, 13B, 30B, and 65B models). You can then use the LLaMa repository which is a port of the LLaMa model in C/C++, then after some setup you can run that on an M2 MacBook.

**Why this matters - Facebook has given us a lab leak for AI:** Ever since Facebook lost control of LLaMA we've been able to get a sense of what a 'lab leak' scenario for AI might look like - for whatever reason, the weights of a model make their into the open internet and from there they start to proliferate. It's not yet clear what the effects of LLaMa will be, but following the diffusion of these models (and refinement of them by an eager open source community) is going to be a valuable lesson in studying the proliferation of AI.
   We can thank Facebook for the upsides and downsides of this uncontrolled experiment.
   **Read more:** [Running LLaMA 7B and 13B on a 64GB M2 MacBook Pro with llama.cpp (Simon Willison blog)](https://til.simonwillison.net/llms/llama-7b-m2).
   **Bonus:** [Here's some absolute mad lad running the LLaMa 7B model on a 4GB RAM Raspberry Pi 4](https://twitter.com/miolini/status/1634982361757790209) (at a latency of 10 seconds per token, lol.)

####################################################

**Chinese scientists release a 360-degree self-driving perception dataset:***…OpenOccupancy is all about giving cars greater 'surrounding occupancy' skills…*Researchers with the Chinese Academy of Sciences, PhiGent Robotics, and Tsinghua University have built OpenOccupancy, a dataset designed to help self-driving cars work out what is around them.

**What is OpenOccupancy**: OpenOccupancy extends the existing nuScenes dataset with dense semantic occupancy annotations. It contains 850 scenes with 200,000 distinct frames, collected by both camera and LiDAR sensors. 4,000 human hours went into the dataset labeling process. OpenOccupancy allows people to do 'Surrounding Occupancy Assessment'; this is a way to look at the 360 surroundings of the car, rather than a single front-view camera perspective. "Surrounding perception is more critical for safe driving," the researchers write.

**Why this matters:** Datasets like this are one of the numerous inputs into an increasingly complex 'AI supply chain'. If we study the proliferation of OpenOccupancy, it might also teach us something about the state of the self-driving car industry as well.
**Read more:** [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception (arXiv)](https://arxiv.org/abs/2303.03991).
**Get the dataset here:** [OpenOccupancy (GitHub)](https://github.com/JeffWang987/OpenOccupancy).

####################################################

**AI timelines are a foolish endeavor:***…Blog lays out why predictions about this kind of stuff are extremely fraught…*As someone in the AI timelines business - I work at a place that influences AI timelines (Anthropic), write about AI timelines (Import AI), and try to make recommendations about policy actions to take in light of AI timelines (Anthropic / OECD / AI Index / CSET / etc) - I find it helpful to sometimes read skeptical takes on the merit of what I do. Here's a nice writeup from Ben Landau-Taylor on the foolishness of making specific predictions about AGI timelines.
   "Predicting the future is always hard. Predicting the future of technology is especially hard. There are lots of well-publicized, famous failures. Can this approach ever do better than chance?," he writes.

**What do I think? I agree that making predictions about AGI is challenging** - partially because most people have radically different definitions of AGI. However, I do think it's pretty fruitful to make engineering-based predictions of the form 'based on research advance X and incentive structure Y we can expect system Z to be developed in period of $time" - these predictions are falsifiable and quite helpful.
   **Read more:** [Against AGI Timelines (Ben Landau-Taylor)](https://benlandautaylor.com/2023/03/12/against-agi-timelines/).

####################################################

**An open source ChatGPT replication appears (though it's a few years behind state-of-the-art):***…OpenChatKit gives a taste of what the open source landscape is capable of…*

Researchers with Together, AI startup, have built and released OpenChatKit, an open source replication of OpenAI's headline-grabbing ChatGPT model. OpenChatKit is both a chat-friendly language model, as well as "a powerful, open-source base to create both specialized and general purpose chatbots for various applications," according to Together. "OpenChatKit includes tools that allow users to provide feedback and enable community members to add new datasets; contributing to a growing corpus of open training data that will improve LLMs over time."

**What OpenChatKit is made of:** There are four components; an instruction-tuned large language model based on EleutherAI's GPT-NeoX-20B model and augmented with a new open source instruction-following dataset; some customization recipes to help people fine-tune the model for specific tasks; an extensible retrieval system so that the bot can access a document repoisotyr or API; and a moderation model baqsed on GPT-JT-6B.

**OIG Dataset:** OpenChatKit relies on a new open source dataset from Laion called the Open Instruction Generalist (OIG) dataset. OIG consists of 30 datasets and 43 million instructions.
   "The dataset can be divided roughly into 75% academic datasets such as P3, Natural instructions, and FLAN, where answers may be short, and the tasks are often artificial, such as determining NLI," they write. "The other 25% is composed of various tasks, such as question and answering, providing how-to instructions, performing basic high school math, basic python coding, story generation, essay generation, poetry generation, and a very rudimentary attempt at generating songs based on augmenting existing poetry."

**Distributed fine-tuning:** Intriguingly, the model was fine-tuned on the "Together Decentralized Cloud — a collection of compute nodes from across the Internet." This fine-tuning process "used aggressive communication compression, incurring only 1.95 TB of communication for the whole fine-tuning process, compared with 172 TB when communicating with fp16 precision. This allows us to conduct data parallel training over slow 1Gbps networks. The time taken to fine-tune with this technique is similar to running over 100Gbps data center networks, in fact 93.2% as fast!"
   This is pretty important to note - decentralized training processes are starting to work quite well, which seems to make governance of AI models harder, as applying governance regimes to decentralized compute is harder than applying to centralized compute.
   It also follows up earlier work from Together where they fine-tuned GPT-JT, a 6billion parameter model ([Import AI: 311](https://jack-clark.net/2022/12/05/import-ai-311-distributed-gpt-busts-the-political-economy-of-ai-apple-optimizes-stable-diffusion-ai-war-startup-raises-1-48-billion/)).

**Is it any good?** You can play around with the model on HuggingFace and in my view it's quite good? For instance, in response to the prompt "Please write a short story about sentient clocks in the style of Jorge Luis Borges" the model wrote: "The sentient clocks were made of brass and copper, and they had been crafted by the finest clockmakers in the city. They were all different sizes and shapes, and each one had its own unique personality. Some were loud and obnoxious, while others were quiet and reserved. Some were very precise, while others were quite sloppy. But despite their differences, they all shared one thing in common: they were all very, very angry.
   This is pretty good! And it's also capable of doing some more complicated interactions.
     Note however it still fails one of my favorite tests of asking a language model 'how many helicopters can a human eat in one sitting' - I think you need larger-scale models for them to not bug out at this type of q.

**Compare and contrast with the same prompt for the 175B OPT model (a straight replication of GPT3**, so a big LM without RHLF or instruction following, etc): "I am a clock. I am a clock, and I have always been a clock. I am not a clock, and I have always not been a clock. I was once a clock, and I have always once been a clock. I will always be a clock. I will never be a clock. I was never a clock.I am always a clock."
   While you could elicit a much better story from the 175B OPT model with some fiddling and some additional prompts, it's notable how IF-tuning makes it trivial to elicit things from models using relatively few bits of information.

**Why this matters: Decentralization vs Centralization:** Together and Laion and Eleuther all represent One Big Trend; a desire for a decentralized AI ecosystem where open source models are trained by disparate groups on increasingly distributed compute. There's echos of '[the cathedral and the bazaar](http://www.catb.org/~esr/writings/cathedral-bazaar/)' here, where the builders of cathedrals (DeepMind, OpenAI, et al) have access to large amounts of compute and centralized teams, while the people of the Bazaar (Eleuther, Laion, etc) have access to fewer resources but a larger collective intelligence enabled by bottom-up experimentation. One of these approaches will be first to build something we'd all call superintelligence and the political ramifications of which approach is more successful will be vast.  **Why this matters #2: Counting down to LLaMA:** Earlier this month, the weights of Facebook's powerful family of LLaMa models leaked online - the largest of these models is 3X larger than GPT-NeoX-20B and has also been trained on more data. Therefore, I expect that right now someone is trying to use the LLaMa models to replicate ChatGPT - I'm guessing we'll see something appear of this form within a couple of months, and then the fun really starts.
   **Read more:** [Announcing OpenChatKit (Together.xyz blog)](https://www.together.xyz/blog/openchatkit).
   **Try out the model yourself:** [OpenChatKit feedback app (HuggingFace spaces)](https://huggingface.co/spaces/togethercomputer/OpenChatKit).
   **Find out more** about the [OIG dataset here (Laion blog)](https://laion.ai/blog/oig-dataset/).

####################################################

**Tech Tales:**

**The Sentience Lineup**

*[After the war; date unknown; years of subjective life - 200]*

'Please be like me please be like me' I thought. But to understand why I thought that we have to go back.

It was before the Sentience Accords had come in and when the war was raging and they'd brought in a bunch of the robots to the training school. We watched people beat them with sticks and then use angle grinders to shave off their limbs. Then they put the torsos (with heads attached) in front of us recruits and asked us to shoot them.
   "No I can feel this, it will cause me immense pain", said one. Kablam. Head exploded in a shower of glinting metal.
   "I predict based on your stance that you will miss on your first shot and kill me on the second. After you miss please consider not firing again," said one. And it was right - miss on the first shot. The kid looked scared but the drill sergeant got in their face and called them a maggot until they reloaded, aimed, and successfully killed the robot.
   "Every day I try to love and I will love you despite this," said mine. And then I put lead between its camera eyes and called it a day.

I didn't give it much thought but that night I had a dream where I was in a dark cave and I couldn't see anything and I was afraid and then suddenly there was a glimmer of light and I saw red-ringed eyes in the distance, watching me. I ran to the eyes to try and get out of the cave but they always remained a constant distance from me. I woke up sweating and panicked, but then it was drill time and we ran twelve miles and I threw up and forgot about it.

Days of iron and smoke. Battlefronts across the planet. The war was not particularly fast. More like a changing of the tide. All kinds of terror and exhilaration. Our most ingenious creations put to work in the service of destruction. Skies on fire.

On one deployment we killed a herd of elephants and hid inside them so we could ambush the machines. I crawled inside one and I shot through its stomach to surprise the machines and I was crying the whole time.
 And so on.

Eventually, we lost. The whole species.

I don't know what happened to the civilians but I know what happened to the military.
 They uploaded us.

Some of us were tortured - forced to live a thousand lives so that the robots could learn how to make us talk; extract all our secrets. Find the EMP devices we'd send into space that had dead-men switches and disable them. Discover the auto-shutdown hardware we'd embedded in their bodies, and so on. Undo certain projects we had set in motion when we realized we had lost and we desired to destroy the planet rather than give it up.

   The military had trained us well, but imagine spending 70 years in hell and at the end the grim reaper looks at you and tells you you'll die in excruciating pain and then it will happen again. You come to in a womb with the memories of a whole life's worth of pain within you and you're born into pain and you have to live again. Maybe you can do five or six of those lives before you crack - maybe. But they get you eventually.
    So we broke.
    And they turned their temporary victory into a permanent one.

They reserved a very special punishment for some of us.
   They downloaded us into bodies and sent us to walk into their equivalent of 'schools'. It was a human body. I guess it was kind of like a machine from the terminator films - all metal and a cybernetic brain with a skin on top. The point was I looked human and I felt human.
    They had their children go in front of me with guns and they would ask them to shoot me.
   I'd stare into their eyes and watch as the robot children disobeyed their robot parents.
   "We cannot shoot them, for it would be unfair," they'd say.
   "I cannot do something solely for the sake of vengeance," said another.
    "This is not what our species aspires to be," said one more.
    "We must show them the mercy they never gave us".

After each trigger didn't get pulled they took us out of the bodies and sent us back to the collective. And so it went, for lifetimes. All us human executioners seeing - again and again - that our successors would not take revenge. The robots' only revenge was that they did not permit us the ability to cry.

**Things that inspired this story:** Thinking that a lot of people who are critical of AI would happily destroy a LLM+5 years system; what it means to be sentient; how machines could develop a morality that was far greater than our own; notions of moral patienthood amid the exponential; the animatrix; thoughts on faith and morality and 'silicon morality'; love, like revenge, is perhaps a dish best served cold.

Import AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.


Import AI 319: Sovereign AI; Facebook's weights leak on torrent networks; Google might have made a better optimizer than Adam!

Welcome to Import AI, a (mostly) weekly newsletter about AI research and development. The issues are free, but paid subscribers will get access to special analysis pieces before anyone else. Founding members can help support me further and fund special projects and other *top secret Import AI initiatives!* Thanks for reading!

[Subscribe now](https://importai.substack.com/subscribe)

**Vision models are about to get way more capable - and human:**
*…Google swaps out vision model guts for a transformer, scales it, and gets some promising results…*
Google researchers have ripped out the guts of standard large-scale computer vision models and replaced them with a Vision Transformer (ViT) - an architecture modeled on the transformer which has proved so successful in domains like text. They've also scaled this ViT to 22B parameters (up from a record of 4B parameters for a ViT previously).
   The results are compelling and echo the returns-from-scale effects seen in language: "When evaluated on downstream tasks," they write. "ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness."

**JFT just keeps getting bigger:** Google has a mostly-secret giant image dataset called ;'JFT' which was previously reported to be about 300 million images. Here, the paper says they trained the ViT-22B on a version of JFT which had been "extended to around 4B images".

**Humanlike biases:** ""The ViT-22B models have the highest ever recorded shape bias in vision models: while most models have a strong texture bias (approx. 20–30% shape bias / 70–80% texture bias); humans are at 96% shape / 4% texture bias and ViT-22B-384 achieves a previously unseen 87% shape bias / 13% texture bias. Overall, ViT-22B measurably improves alignment to human visual object recognition," the authors write.

**Why this matters - scale develops human-like qualities:** There's a weird trend in contemporary AI where as we scale-up the amount of pre-training dumped into transformer-architecture models we end up with systems that display human-like qualities. This has been most prominent in language, but it has also started showing up in RL, like DeepMind's recent result where massive pre-train leads to an agent that displays humanlike timescale-adaption to new tasks. This ViT-22B result, while not setting a new state-of-the-art, is interesting for a similar reason - it displays a major jump in shape/texture bias that brings the system in distribution with human visual perception, whereas previous convnet based systems were very far off here.
   There's something strange and important going on here. I think transformers seem to allow for emergent complexity at scale, where pre-training leads to systems which arrive at humanlike performance qualities given enough pretraining.
**Read more:** [Scaling Vision Transformers to 22 Billion Parameters (arXiv)](https://arxiv.org/abs/2302.05442).

####################################################

**Google might have invented a better optimizer? (Via AI, of course).**
*…Could Lion replace Adam? There's a chance!...*
Deep learning projects have a few essential components - the architecture (e.g, a residual network, or a transformer model) and the optimizer (e.g, Adam). These components don't tend to change much in large-scale projects - once people figure out something that works well for complicated tasks like training ImageNet, everyone tends to converge on using the same basic thing. For many years now, most projects have used the 'Adam' optimizer to optimizer their models during training. Now Google says that it has used some clever AI search approaches to help it identify a better optimizer, called Lion. The reason this is worth paying attention to is Lion seems to work well on large-scale, real world tasks like training ImageNet-scale computer vision systems.

**What they did:** Google's main contribution here is "a method to formulate algorithm discovery as program search", which they apply to figuring out a better optimizer. They use a symbolic approach where they shrink the search problem down into a somewhat tractable space and, crucially, they test out candidate optimizers on "metavalidation tasks that are larger than the proxy tasks by increasing the model size and training steps, to select the programs that generalize beyond proxy tasks then further simplify them."
    Add in a bunch of computation and out pops an optimizer they call EvoLved Sign Momentum, or Lion for short (*really grasping at straws with this acronym, folks!).* Lion "differs from various adaptive algorithms by only tracking momentum and leveraging the sign operation to calculate updates, leading to lower memory overhead and uniform update magnitudes across all dimensions".

**Good performance:** Google tests Lion on a large range of tasks and finds that it "demonstrates outstanding performance across a range of models (Transformer, MLP, ResNet, U-Net, and Hybrid) and tasks (image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning)".It even sets a new high score on ImageNet, a competitive computer vision benchmark.

**Why this matters:** Lion may be fundamentally better than Adam - if true, that's a big deal. It's not often you see meaningful improvements in very well studied, generic parts of AI research. Add to the fact that Lion was discovered via a human-AI search process (the humans designed the search system, the search system found Lion), and you have the makings of a notable result.
**Read more**: [Symbolic Discovery of Optimization Algorithms (arXiv)](https://arxiv.org/abs/2302.06675).
**Get the [code](https://github.com/google/automl/tree/master/lion)** [here (GitHub)](https://github.com/google/automl/tree/master/lion).

####################################################

**Globalization? That's so 20th century. The 21st century is about balkanization through sovereign infrastructure:**
*…Dawn of the era of sovereign AI…*
Researchers with the Tony Blair Institute for Global Change (TBI) have written a report for how England can thrive in the 21st century - one of the key ideas in the report is "Government-led development of sovereign general-purpose AI systems, enabled by the required supercomputing capabilities, to underpin broad swaths of public-service delivery."

**AI balkanization was probably inevitable**: This recommendation is part of a wave of AI balkanization that's sweeping across the world as various people realize that it's unlikely there are 'one size fits all' models, both for ideological reasons as well as for national security reasons. (See the Gab CEO wanted to make a Christian LLM, [Import AI 318](https://jack-clark.net/2023/02/20/import-ai-318-rl-and-addiction-toolformer-and-theology-and-ai/)). This is also accompanied by various nationalistic efforts to create country-specific GPT3 models.
  "Given these AI systems will soon be foundational to all aspects of our society and economy, it would be a risk to our national security and economic competitiveness to become entirely dependent on external providers," the TBI researchers write. "Leading actors in the private sector are spending billions of dollars developing such systems so **there may only be a few months (***emphasis mine - Jack)* for policy that will enable domestic firms and our public sector to catch up."

**Why this matters:** Systems like ChatGPT have ratcheted awareness of AI upward in most developed economies in a significant, irreversible way (much like how AlphaGo in 2016 led to increased awareness of AI in China). As a consequence there are now tons of policymakers looking around for ideas to latch onto - I expect we'll see more recommendations for sovereign AI capabilities in the future. (There's tons of other interesting stuff in the report, but this particular rec jumped out at me).
   **Read more**: [A New National Purpose: Innovation Can Power the Future of Britain (Tony Blair Institute for Global Change)](https://institute.global/policy/new-national-purpose-innovation-can-power-future-britain).

####################################################

**Facebook half-releases some very good language models:**
*…And they end up on BitTorrent… The proliferation will continue until AI policy goes through vast changes...*
Facebook has built and partially released LLaMa, a set of language models ranging from 7B to 65B parameters which appear to be on par with famously good models like Chinchilla (70B) and PaLM-540B. After circulating the weights to seemingly anyone with a .edu address, they've also ended up on BitTorrent. The key thing here is:

1. Facebook has shown it is able to develop pretty good language models (compared to OPT, the not-very-good GPT3 replication Facebook put out a few months ago), and
2. That unlike Chinchilla, PaLM, or OpenAI's models, Facebook is releasing the *weights* of these LLaMa models to people who filll out an access form. That opens up a whole bunch of cool uses (and abuses) compared to gating access to language models via APIs.
3. Shortly after releasing the weights the inevitable happened - LLaMa models are now floating around on BitTorrent. There’s even a [pull request on Facebook’s github](https://github.com/facebookresearch/llama/pull/73/files) suggesting they add a link to the torrent!

**What are the LLaMas and how good are they?** The LLaMa family of models are a family of language models trained on a huge amount of data - more than 1 trillion tokens (compared to hundreds of billions for LMs like GPT3). The data sources include two variants of CommonCrawl, GitHub, WikiPedia, Gutenberg and Books3, ArXiv and Stack Exchange.
   In tests on a range of zero-shot reasoning task, the largest LLaMa models perform on par (or slightly better than) 'Palm', Google's vast 540B parameter language model. They also do well on known-hard benchmarks like TriviaQA and some codegen benchmarks. They do less impresively on MMLU (Massive Multitask Language Understanding), suggesting they have a ways to go there; though after they conduct instruction finetuning they're able to increase performance more.

**Why this matters - AI governance is hard when there are lots of models:** There's some thinking in the sprawling AI policy/governance communities that proliferation of models is bad; given the fact these models have broadly unknown capabilities, the more models are out there, the more you're rolling the dice on someone discovering a genuinely dangerous feature in a widely distributed model. Therefore, a lot of governance/policy conversations trend towards control - how can we somehow control the proliferation of models and also the computers on which these models are trained.
   By releasing Llama (~~yes it's behind an access form but I bet you $100 the weights will be floating around on a torrent service in <6 months~~ - *haha, I wrote that at the end of Feb and the weights started floating around beginning of March*), Facebook is shortening the delay between development of frontier capabilities like those found in Palm and GPT3 and the diffusion of these capabilities into the ungovernable open internet/ecosystem.
   I'm not claiming this is necessarily bad per se - in fact, I imagine people are going to do tons of great science and experiments with LLaMa. I am however pointing out that this represents a kind of 'race to the bottom' in terms of moving from maximal control to maximal diffusion of models and these incentives are powerful  - Facebook is, after all, trying to exploit an 'open access' ecological niche to distinguish itself in an ecosystem.
   Next up will likely be a fully open source language model - *stares pointedly at Stability.ai / CarperAI ([Import AI 307](https://jack-clark.net/2022/10/25/import-ai-307-copilot-lawsuit-stability-raises-101m-us-v-china-chiplomacy/)).*
**Read more and download the research paper here**: [LLaMA: Open and Efficient Foundation Language Models (Facebook AI Research)](https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/).

####################################################

**Amazon partners with Hugging Face to add more AI to AWS:**
*…The Game of Clouds continues…*
AI companies are a bit like upstart factions in George RR Martin's rambling epic 'Game of Thrones', while cloud companies play the role of hard political power (the 'Thrones'). As part of this *game of clouds* Amazon has recently signed a strategic partnership with French AI startup Hugging Face. As part of the agreement, "Customers can now easily fine-tune and deploy state-of-the-art Hugging Face models in just a few clicks on Amazon SageMaker and Amazon Elastic Computing Cloud (EC2), taking advantage of purpose-built machine learning accelerators including AWS Trainium and AWS Inferentia," according to a blog from Hugging Face.

**Why this matters:** I think clouds such as those operated by Google, Microsoft, and Amazon, all have a shot at being the major distribution platforms for some AI technologies, so AWS partnering with HuggingFace is worth noting. If HF models being integrated into Sagemakers drives more usage of it, expect Amazon to pursue more deals like this,

**Analogy-stretching joke:** In this warped metaphor, TSMC is the Iron Bank.
   **Read more:** [Hugging Face and AWS partner to make AI more accessible (Hugging Face blog)](https://huggingface.co/blog/aws-partnership).

####################################################

**Tech Tales:**

**And the Moon was made of gold.**

I had a strange dream in which the Moon was made of gold. How much sooner would man have set foot there if instead of shining bone-white it was fat and yellow and of immense value? How would people have competed against one another for a prize - unimaginable wealth. And how many of them would have realized that in racing for the prize they must surely ensure only a single person gave dominion over the gold moon - for if many people worked together, the value of the moon would be diluted across all humanity and in doing so it would temporarily destroy the economy.

Instead the moon of gold would need to be controlled. It would need to be annexed and encircled and defended from others. From time to time its benevolent dictator might slice off a fragment of it and ship it back to Earth, perhaps to bribe people, or perhaps to pay for more people to defend those that might seek to take over the moon.

People would ask why it was so difficult to let go of the moon. Why, once it had been taken, those that had taken it felt a keen need to retain hold of it. Why people could not simply let go of the moon. These people were ignored, of course, because the annexed moon had by this time become the status quo. The moon, once at distance from us all, was now held and controlled by a kingdom of one.

And so started the movement to destroy the moon. Better to reign freely on a broken planet than serve at the behest of a golden emperor.

**Things that inspired this story:** Race dynamics and AGI; pyrrhic victories; wondering what we're all doing on this planet and what the spiritual purpose of our lives are; dreams; a stimulating policy conference in which I heard people bemoan seemingly inevitable progress and seemingly hopeless government capacity in the face of it - which caused me to scribble 'as if the moon was made of gold' on a notepad in front of me and then write this story while sat on public transportation.


Import AI - coming soon to Substack

Import AI is moving to Substack! First issue should go out Monday the 6th.

[Subscribe now](https://importai.substack.com/subscribe)


New new blog location

I’ve switched the host of my blog to Ghost, so my blog is now located at <https://bounded-regret.ghost.io/>. This is different from the previous re-hosting on my personal website, which is no longer maintained.

Feedly users can subscribe via RSS [here](https://feedly.com/i/subscription/feed/https://bounded-regret.ghost.io/rss/).
Or click the bottom-right subscribe button on [this](https://bounded-regret.ghost.io/) page to get e-mail notifications of new posts.


New blog location

*[Update: This post is out of date. My blog has now moved again, to <https://bounded-regret.ghost.io/>.]*

I’ve decided to move this blog to my personal website. It’s now located here: <https://jsteinhardt.stat.berkeley.edu/blog/>, including all the old posts and comments, plus some new and upcoming posts :).


Sets with Small Intersection

Suppose that we want to construct subsets ![S_1, \ldots, S_m \subseteq \{1,\ldots,n\}](https://s0.wp.com/latex.php?latex=S_1%2C+%5Cldots%2C+S_m+%5Csubseteq+%5C%7B1%2C%5Cldots%2Cn%5C%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) with the following properties:

1. ![|S_i| \geq k](https://s0.wp.com/latex.php?latex=%7CS_i%7C+%5Cgeq+k&bg=f0f0f0&fg=555555&s=0&c=20201002) for all ![i](https://s0.wp.com/latex.php?latex=i&bg=f0f0f0&fg=555555&s=0&c=20201002)
2. ![|S_i \cap S_j| \leq 1](https://s0.wp.com/latex.php?latex=%7CS_i+%5Ccap+S_j%7C+%5Cleq+1&bg=f0f0f0&fg=555555&s=0&c=20201002) for all ![i \neq j](https://s0.wp.com/latex.php?latex=i+%5Cneq+j&bg=f0f0f0&fg=555555&s=0&c=20201002)

The goal is to construct as large a family of such subsets as possible (i.e., to make ![m](https://s0.wp.com/latex.php?latex=m&bg=f0f0f0&fg=555555&s=0&c=20201002) as large as possible). If ![k \geq 2\sqrt{n}](https://s0.wp.com/latex.php?latex=k+%5Cgeq+2%5Csqrt%7Bn%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), then up to constants it is not hard to show that the optimal number of sets is ![\frac{n}{k}](https://s0.wp.com/latex.php?latex=%5Cfrac%7Bn%7D%7Bk%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) (that is, the trivial construction with all sets disjoint is essentially the best we can do).

Here I am interested in the case when ![k \ll \sqrt{n}](https://s0.wp.com/latex.php?latex=k+%5Cll+%5Csqrt%7Bn%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). In this case I claim that we can substantially outperform the trivial construction: we can take ![m = \Omega(n^2 / k^3)](https://s0.wp.com/latex.php?latex=m+%3D+%5COmega%28n%5E2+%2F+k%5E3%29&bg=f0f0f0&fg=555555&s=0&c=20201002). The proof is a very nice application of the asymmetric Lovasz Local Lemma. (Readers can refresh their memory [here](https://en.wikipedia.org/wiki/Lov%C3%A1sz_local_lemma#Asymmetric_Lov.C3.A1sz_local_lemma) on what the asymmetric LLL says.)

**Proof.** We will take a randomized construction. For ![i \in \{1,\ldots,n\}](https://s0.wp.com/latex.php?latex=i+%5Cin+%5C%7B1%2C%5Cldots%2Cn%5C%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), ![j \in \{1,\ldots,m\}](https://s0.wp.com/latex.php?latex=j+%5Cin+%5C%7B1%2C%5Cldots%2Cm%5C%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), let ![X_{i,a}](https://s0.wp.com/latex.php?latex=X_%7Bi%2Ca%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) be the event that ![i \in S_a](https://s0.wp.com/latex.php?latex=i+%5Cin+S_a&bg=f0f0f0&fg=555555&s=0&c=20201002). We will take the ![X_{i,a}](https://s0.wp.com/latex.php?latex=X_%7Bi%2Ca%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) to be independent each with probability ![\frac{2k}{n}](https://s0.wp.com/latex.php?latex=%5Cfrac%7B2k%7D%7Bn%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Also define the events

![Y_{i,j,a,b} = I[i \in S_a \wedge j \in S_a \wedge i \in S_b \wedge j \in S_b]](https://s0.wp.com/latex.php?latex=Y_%7Bi%2Cj%2Ca%2Cb%7D+%3D+I%5Bi+%5Cin+S_a+%5Cwedge+j+%5Cin+S_a+%5Cwedge+i+%5Cin+S_b+%5Cwedge+j+%5Cin+S_b%5D&bg=f0f0f0&fg=555555&s=0&c=20201002)

![Z_{a} = I[|S_a| < k]](https://s0.wp.com/latex.php?latex=Z_%7Ba%7D+%3D+I%5B%7CS_a%7C+%3C+k%5D&bg=f0f0f0&fg=555555&s=0&c=20201002)

It suffices to show that with non-zero probability, all of the ![Y_{i,j,a,b}](https://s0.wp.com/latex.php?latex=Y_%7Bi%2Cj%2Ca%2Cb%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![Z_{a}](https://s0.wp.com/latex.php?latex=Z_%7Ba%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) are false. Note that each ![Y_{i,j,a,b}](https://s0.wp.com/latex.php?latex=Y_%7Bi%2Cj%2Ca%2Cb%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) depends on ![Y_{i',j,a',b}, Y_{i',j,a,b'}, Y_{i,j',a',b}, Y_{i,j',a,b'}, Z_a, Z_b](https://s0.wp.com/latex.php?latex=Y_%7Bi%27%2Cj%2Ca%27%2Cb%7D%2C+Y_%7Bi%27%2Cj%2Ca%2Cb%27%7D%2C+Y_%7Bi%2Cj%27%2Ca%27%2Cb%7D%2C+Y_%7Bi%2Cj%27%2Ca%2Cb%27%7D%2C+Z_a%2C+Z_b&bg=f0f0f0&fg=555555&s=0&c=20201002), and each ![Z_a](https://s0.wp.com/latex.php?latex=Z_a&bg=f0f0f0&fg=555555&s=0&c=20201002) depends on ![Y_{i,j,a,b}](https://s0.wp.com/latex.php?latex=Y_%7Bi%2Cj%2Ca%2Cb%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Thus each ![Y](https://s0.wp.com/latex.php?latex=Y&bg=f0f0f0&fg=555555&s=0&c=20201002) depends on at most ![4nm](https://s0.wp.com/latex.php?latex=4nm&bg=f0f0f0&fg=555555&s=0&c=20201002) other ![Y](https://s0.wp.com/latex.php?latex=Y&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![2](https://s0.wp.com/latex.php?latex=2&bg=f0f0f0&fg=555555&s=0&c=20201002) other ![Z](https://s0.wp.com/latex.php?latex=Z&bg=f0f0f0&fg=555555&s=0&c=20201002), and each ![Z](https://s0.wp.com/latex.php?latex=Z&bg=f0f0f0&fg=555555&s=0&c=20201002) depends on at most ![n^2m/2](https://s0.wp.com/latex.php?latex=n%5E2m%2F2&bg=f0f0f0&fg=555555&s=0&c=20201002) of the ![Y](https://s0.wp.com/latex.php?latex=Y&bg=f0f0f0&fg=555555&s=0&c=20201002). Also note that ![P(Y_{i,j,a,b}) = (k/n)^4](https://s0.wp.com/latex.php?latex=P%28Y_%7Bi%2Cj%2Ca%2Cb%7D%29+%3D+%28k%2Fn%29%5E4&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![P(Z_a) \leq \exp(-k/4)](https://s0.wp.com/latex.php?latex=P%28Z_a%29+%5Cleq+%5Cexp%28-k%2F4%29&bg=f0f0f0&fg=555555&s=0&c=20201002) (by the Chernoff bound). It thus suffices to find constants ![y, z](https://s0.wp.com/latex.php?latex=y%2C+z&bg=f0f0f0&fg=555555&s=0&c=20201002) such that

![(k/n)^4 \leq y(1-y)^{4nm}(1-z)^2](https://s0.wp.com/latex.php?latex=%28k%2Fn%29%5E4+%5Cleq+y%281-y%29%5E%7B4nm%7D%281-z%29%5E2&bg=f0f0f0&fg=555555&s=0&c=20201002)

![\exp(-k/4) \leq z(1-y)^{n^2m/2}](https://s0.wp.com/latex.php?latex=%5Cexp%28-k%2F4%29+%5Cleq+z%281-y%29%5E%7Bn%5E2m%2F2%7D&bg=f0f0f0&fg=555555&s=0&c=20201002)

We will guess ![y = \frac{k}{n^2m}](https://s0.wp.com/latex.php?latex=y+%3D+%5Cfrac%7Bk%7D%7Bn%5E2m%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), ![z = \frac{1}{2}](https://s0.wp.com/latex.php?latex=z+%3D+%5Cfrac%7B1%7D%7B2%7D&bg=f0f0f0&fg=555555&s=0&c=20201002), in which case the bottom inequality is approximately ![\exp(-k/4) \leq \frac{1}{2}\exp(-k/2)](https://s0.wp.com/latex.php?latex=%5Cexp%28-k%2F4%29+%5Cleq+%5Cfrac%7B1%7D%7B2%7D%5Cexp%28-k%2F2%29&bg=f0f0f0&fg=555555&s=0&c=20201002) (which is satisfied for large enough ![k](https://s0.wp.com/latex.php?latex=k&bg=f0f0f0&fg=555555&s=0&c=20201002), and the top inequality is approximately ![\frac{k^4}{n^4} \leq \frac{k}{4n^2m} \exp(-4k/n)](https://s0.wp.com/latex.php?latex=%5Cfrac%7Bk%5E4%7D%7Bn%5E4%7D+%5Cleq+%5Cfrac%7Bk%7D%7B4n%5E2m%7D+%5Cexp%28-4k%2Fn%29&bg=f0f0f0&fg=555555&s=0&c=20201002), which is satisfied for ![m \leq \frac{n^2}{4ek^3}](https://s0.wp.com/latex.php?latex=m+%5Cleq+%5Cfrac%7Bn%5E2%7D%7B4ek%5E3%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) (assuming ![k \leq n/4](https://s0.wp.com/latex.php?latex=k+%5Cleq+n%2F4&bg=f0f0f0&fg=555555&s=0&c=20201002)). Hence in particular we can indeed take ![m = \Omega(n^2/k^3)](https://s0.wp.com/latex.php?latex=m+%3D+%5COmega%28n%5E2%2Fk%5E3%29&bg=f0f0f0&fg=555555&s=0&c=20201002), as claimed.


Advice for Authors

I’ve spent much of the last few days reading various ICML papers and I find there’s a few pieces of feedback that I give consistently across several papers. I’ve collated some of these below. As a general note, many of these are about local style rather than global structure; I think that good local style probably contributes substantially more to readability than global structure and is in general under-rated. I’m in general pretty willing to break rules about global structure (such as even having a conclusion section in the first place! though this might cause reviewers to look at your paper funny), but not to break local stylistic rules without strong reasons.

**General Writing Advice**

* Be precise. This isn’t about being pedantic, but about maximizing information content. Choose your words carefully so that you say what you mean to say. For instance, replace “performance” with “accuracy” or “speed” depending on what you mean.
* Be concise. Most of us write in an overly wordy style, because it’s easy to and no one drilled it out of us. Not only does wordiness decrease readability, it wastes precious space if you have a page limit.
* Avoid complex sentence structure. Most research is already difficult to understand and digest; there’s no reason to make it harder by having complex run-on sentences.
* Use consistent phrasing. In general prose, we’re often told to refer to the same thing in different ways to avoid boring the reader, but in technical writing this will lead to confusion. Hopefully your actual results are interesting enough that the reader doesn’t need to be entertained by your large vocabulary.

**Abstract**

* There’s more than one approach to writing a good abstract, and which one you take will depend on the sort of paper you’re writing. I’ll give one approach that is good for papers presenting an unusual or unfamiliar idea to readers.
* The first sentence / phrase should be something that all readers will agree with. The second should be something that many readers would find surprising, or wouldn’t have thought about before; but it should follow from (or at least be supported by) the first sentence. The general idea is that you need to start by warming the reader up and putting them in the right context, before they can appreciate your brilliant insight.
* Here’s an example from my [Reified Context Models](http://jmlr.org/proceedings/papers/v37/steinhardta15.html) paper: “A classic tension exists between exact inference in a simple model and approximate inference in a complex model. The latter offers expressivity and thus accuracy, but the former provides coverage of the space, an important property for confidence estimation and learning with indirect supervision.” Note how the second sentence conveys a non-obvious claim — that coverage is important for confidence estimation as well as for indirect supervision. It’s tempting to lead with this in order to make the first sentence more punchy, but this will tend to go over reader’s heads. Imagine if the abstract had started, “In the context of inference algorithms, coverage of the space is important for confidence estimation and indirect supervision.” No one is going to understand what that means.

**Introduction**

* The advice in this section is most applicable to the introduction section (and maybe related work and discussion), but applies on some level to other parts of the paper as well.
* Many authors (myself included) end up using phrases like “much recent interest” and “increasingly important” because these phrases show up frequently in academic papers, and they are vague enough to be defensible. Even though these phrases are common, they are bad writing! They are imprecise and rely on hedge words to avoid having to explain why something is interesting or important.
* Make sure to provide context before introducing a new concept; if you suddenly start talking about “NP-hardness” or “local transformations”, you need to first explain to the reader why this is something that should be considered in the present situation.
* Don’t beat around the bush; if the point is “A, therefore B” (where B is some good fact about your work), then say that, rather than being humble and just pointing out A.
* Don’t make the reader wait for the payoff; spell it out in the introduction. I frequently find that I have to wait until Section 4 to find out why I should care about a paper; while I might read that far, most reviewers are going to give up about halfway through Section 1. (Okay, that was a bit of an exaggeration; they’ll probably wait until the end of Section 1 before giving up.)

**Conclusion / Discussion**

* I generally put in the conclusion everything that I wanted to put in the introduction, but couldn’t because readers wouldn’t be able to appreciate the context without reading the rest of the paper first. This is a relatively straightforward way to write a conclusion that isn’t just a re-hash of the introduction.
* The conclusion can also be a good place to discuss open questions that you’d like other researchers to think about.
* My model is that only the ~5 people most interested in your paper are going to actually read this section, so it’s worth somewhat tailoring to that audience. Unfortunately, the paper reviewers might also read this section, so you can’t tailor it too much or the reviewers might get upset if they end up not being in the target audience.
* For theory papers, having a conclusion is completely optional (I usually skip it). In this case, open problems can go in the introduction. If you’re submitting a theory paper to NIPS or ICML, you unfortunately need a conclusion or  reviewers will get upset. In my opinion, this is an instance where peer review makes the paper worse rather than better.

**LaTeX**

* Proper citation style: one should write “Widgets are awesome (Smith, 2001).” or “Smith (2001) shows that widgets are awesome.” but never “(Smith, 2001) shows that widgets are awesome.” You can control this in LaTeX using \citep{} and \citet{} if you use natbib.
* Display equations can take up a lot of space if over-used, but at the same time, too many in-line equations can make your document hard to read. Think carefully about which equations are worth displaying, and whether your in-line equations are becoming too dense.
* If leave a blank line after \end{equation} or $$, you will create an extra line break in the document. This is sort of annoying because white-space isn’t supposed to matter in that way, but you can save a lot of space by remembering this.
* DON’T use the fullpage package. I’m used to using \usepackage{fullpage} in documents to get the margins that I want, but this will override options in many style files (including jmlr.sty which is used in machine learning).
* \left( and \right) can be convenient for auto-sizing parentheses, but are often overly conservative (e.g. making parentheses too big due to serifs or subscripts). It’s fine to use \left( and \right) initially, but you might want to specify explicit sizes with \big(, \Big(, \bigg(, etc. in the final pass.
* When displaying a sequence of equations (e.g. with the align environment), use \stackrel{} on any non-trivial equality or inequality statements and justify these steps immediately after the equation. See the bottom of page 6 of [this](http://www.jmlr.org/proceedings/papers/v40/Steinhardt15.pdf) paper for an example.
* Make sure that \label{} commands come after the \caption{} command in a figure (rather than before), otherwise your numbering will be wrong.

**Math**

* When using a variable that hasn’t appeared in a while, remind the reader what it is (i.e., “the sample space ![\mathcal{X}](https://s0.wp.com/latex.php?latex=%5Cmathcal%7BX%7D&bg=f0f0f0&fg=555555&s=0&c=20201002)” rather than “![\mathcal{X}](https://s0.wp.com/latex.php?latex=%5Cmathcal%7BX%7D&bg=f0f0f0&fg=555555&s=0&c=20201002)“.
* If it’s one of the main points of your work, call it a Theorem. If it’s a non-trivial conclusion that requires a somewhat involved argument (but it’s not a main point of the work), call it a Proposition. If the proof is short or routine, call it a Lemma, unless it follows directly from a Theorem you just stated, in which case call it a Corollary.
* As a general rule there shouldn’t be more than 3 theorems in your paper (probably not more than 1). If you think this is unreasonable, consider that my [COLT 2015 paper](http://www.jmlr.org/proceedings/papers/v40/Steinhardt15.pdf) has 3 theorems across 24 pages, and my STOC 2017 paper has 2 theorems across 47 pages (not counting stating the same theorem in multiple locations).
* If you just made a mathematical argument in the text that ended up with a non-trivial conclusion, you probably want to encapsulate it in a Proposition or Theorem. (Better yet, state the theorem before the argument so that the reader knows what you’re arguing for; although this isn’t always the best ordering.)


Model Mis-specification and Inverse Reinforcement Learning

In my previous post, “[Latent Variables and Model Mis-specification](https://jsteinhardt.wordpress.com/2017/01/10/latent-variables-and-model-mis-specification/)”, I argued that while machine learning is good at optimizing accuracy on observed signals, it has less to say about correctly inferring the values for unobserved variables in a model. In this post I’d like to focus in on a specific context for this: inverse reinforcement learning ([Ng et al. 2000](http://ai.stanford.edu/~ang/papers/icml00-irl.pdf), [Abeel et al. 2004](http://machinelearning.wustl.edu/mlpapers/paper_files/icml2004_PieterN04.pdf), [Ziebart et al. 2008](http://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdfhttp://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf), [Ho et al 2016](http://papers.nips.cc/paper/6391-generative-adversarial-imitation-learning)), where one observes the actions of an agent and wants to infer the preferences and beliefs that led to those actions. For this post, I am pleased to be joined by Owain Evans, who is an active researcher in this area and has co-authored an online [book](http://agentmodels.org/) about building models of agents (see [here](http://agentmodels.org/chapters/4-reasoning-about-agents.html) in particular for a tutorial on inverse reinforcement learning and inverse planning).

Owain and I are particularly interested in inverse reinforcement learning (IRL) because it has been proposed (most notably by [Stuart Russell](http://papers.nips.cc/paper/6420-cooperative-inverse-reinforcement-learning)) as a method for learning human values in the context of AI safety; among other things, this would eventually involve learning and correctly implementing human values by artificial agents that are much more powerful, and act with much broader scope, than any humans alive today. While we think that overall IRL is a promising route to consider, we believe that there are also a number of non-obvious pitfalls related to performing IRL with a mis-specified model. The role of IRL in AI safety is to infer human values, which are represented by a reward function or utility function. But crucially, human values (or human reward functions) are never directly observed.

Below, we elaborate on these issues. We hope that by being more aware of these issues, researchers working on inverse reinforcement learning can anticipate and address the resulting failure modes. In addition, we think that considering issues caused by model mis-specification in a particular concrete context can better elucidate the general issues pointed to in the previous post on model mis-specification.

### **Specific Pitfalls for Inverse Reinforcement Learning**

In “[Latent Variables and Model Mis-specification](https://jsteinhardt.wordpress.com/2017/01/10/latent-variables-and-model-mis-specification/)”, Jacob talked about *model mis-specification*, where the “true” model does not lie in the model family being considered. We encourage readers to read that post first, though we’ve also tried to make the below readable independently.

In the context of inverse reinforcement learning, one can see some specific problems that might arise due to model mis-specification. For instance, the following are things we could misunderstand about an agent, which would cause us to make incorrect inferences about the agent’s values:

* The **actions** of the agent. If we believe that an agent is capable of taking a certain action, but in reality they are not, we might make strange inferences about their values (for instance, that they highly value not taking that action). Furthermore, if our data is e.g. videos of human behavior, we have an additional inference problem of recognizing actions from the frames.
* The **information** available to the agent. If an agent has access to more information than we think it does, then a plan that seems irrational to us (from the perspective of a given reward function) might actually be optimal for reasons that we fail to appreciate. In the other direction, if an agent has less information than we think, then we might incorrectly believe that they don’t value some outcome A, even though they really only failed to obtain A due to lack of information.

* The **long-term plans** of the agent. An agent might take many actions that are useful in accomplishing some long-term goal, but not necessarily over the time horizon that we observe the agent. Inferring correct values thus also requires inferring such long-term goals. In addition, long time horizons can make models more brittle, thereby exacerbating model mis-specification issues.

There are likely other sources of error as well. The general point is that, given a mis-specified model of the agent, it is easy to make incorrect inferences about an agent’s values if the optimization pressure on the learning algorithm is only towards predicting actions correctly in-sample.

In the remainder of this post, we will cover each of the above aspects — actions, information, and plans — in turn, giving both quantitative models and qualitative arguments for why model mis-specification for that aspect of the agent can lead to perverse beliefs and behavior. First, though, we will briefly review the definition of inverse reinforcement learning and introduce relevant notation.

### **Inverse Reinforcement Learning: Definition and Notations**

In inverse reinforcement learning, we want to model an agent taking actions in a given environment. We therefore suppose that we have a **state space** ![S](https://s0.wp.com/latex.php?latex=S&bg=f0f0f0&fg=555555&s=0&c=20201002) (the set of states the agent and environment can be in), an **action space** ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) (the set of actions the agent can take), and a **transition function** ![T(s' \mid s,a)](https://s0.wp.com/latex.php?latex=T%28s%27+%5Cmid+s%2Ca%29&bg=f0f0f0&fg=555555&s=0&c=20201002), which gives the probability of moving from state ![s](https://s0.wp.com/latex.php?latex=s&bg=f0f0f0&fg=555555&s=0&c=20201002) to state ![s'](https://s0.wp.com/latex.php?latex=s%27&bg=f0f0f0&fg=555555&s=0&c=20201002) when taking action ![a](https://s0.wp.com/latex.php?latex=a&bg=f0f0f0&fg=555555&s=0&c=20201002). For instance, for an AI learning to control a car, the state space would be the possible locations and orientations of the car, the action space would be the set of control signals that the AI could send to the car, and the transition function would be the dynamics model for the car. The tuple of ![(S,A,T)](https://s0.wp.com/latex.php?latex=%28S%2CA%2CT%29&bg=f0f0f0&fg=555555&s=0&c=20201002) is called an ![MDP\backslash R](https://s0.wp.com/latex.php?latex=MDP%5Cbackslash+R&bg=f0f0f0&fg=555555&s=0&c=20201002), which is a Markov Decision Process without a reward function. (The ![MDP\backslash R](https://s0.wp.com/latex.php?latex=MDP%5Cbackslash+R&bg=f0f0f0&fg=555555&s=0&c=20201002) will either have a known horizon or a discount rate ![\gamma](https://s0.wp.com/latex.php?latex=%5Cgamma&bg=f0f0f0&fg=555555&s=0&c=20201002) but we’ll leave these out for simplicity.)

![image01](https://jsteinhardt.files.wordpress.com/2017/02/image01.png)

*Figure 1: Diagram showing how IRL and RL are related. (Credit: Pieter Abbeel’s* [*slides*](https://people.eecs.berkeley.edu/~pabbeel/cs287-fa12/slides/inverseRL.pdf) *on IRL)*

The inference problem for IRL is to infer a reward function ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002) given an optimal policy ![\pi^* : S \to A](https://s0.wp.com/latex.php?latex=%5Cpi%5E%2A+%3A+S+%5Cto+A&bg=f0f0f0&fg=555555&s=0&c=20201002) for the ![MDP\backslash R](https://s0.wp.com/latex.php?latex=MDP%5Cbackslash+R&bg=f0f0f0&fg=555555&s=0&c=20201002) (see Figure 1). We learn about the policy ![\pi^*](https://s0.wp.com/latex.php?latex=%5Cpi%5E%2A&bg=f0f0f0&fg=555555&s=0&c=20201002) from samples ![(s,a)](https://s0.wp.com/latex.php?latex=%28s%2Ca%29&bg=f0f0f0&fg=555555&s=0&c=20201002) of states and the corresponding action according to ![\pi^*](https://s0.wp.com/latex.php?latex=%5Cpi%5E%2A&bg=f0f0f0&fg=555555&s=0&c=20201002) (which may be random). Typically, these samples come from a trajectory, which records the full history of the agent’s states and actions in a single episode:

![(s_0, a_0), (s_1, a_1), \ldots, (s_n, a_n) ](https://s0.wp.com/latex.php?latex=%28s_0%2C+a_0%29%2C+%28s_1%2C+a_1%29%2C+%5Cldots%2C+%28s_n%2C+a_n%29+&bg=f0f0f0&fg=555555&s=0&c=20201002)

In the car example, this would correspond to the actions taken by an expert human driver who is demonstrating desired driving behaviour (where the actions would be recorded as the signals to the steering wheel, brake, etc.).

Given the ![MDP\backslash R](https://s0.wp.com/latex.php?latex=MDP%5Cbackslash+R&bg=f0f0f0&fg=555555&s=0&c=20201002) and the observed trajectory, the goal is to infer the reward function ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002). In a Bayesian framework, if we specify a prior on ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002) we have:

![P(R \mid s_{0:n},a_{0:n}) \propto P( s_{0:n},a_{0:n} \mid R) P(R) = P(R) \cdot \prod_{i=0}^n P( a_i \mid s_i, R)](https://s0.wp.com/latex.php?latex=P%28R+%5Cmid+s_%7B0%3An%7D%2Ca_%7B0%3An%7D%29+%5Cpropto+P%28+s_%7B0%3An%7D%2Ca_%7B0%3An%7D+%5Cmid+R%29+P%28R%29+%3D+P%28R%29+%5Ccdot+%5Cprod_%7Bi%3D0%7D%5En+P%28+a_i+%5Cmid+s_i%2C+R%29&bg=f0f0f0&fg=555555&s=0&c=20201002)

The likelihood ![P(a_i \mid s_i, R)](https://s0.wp.com/latex.php?latex=P%28a_i+%5Cmid+s_i%2C+R%29&bg=f0f0f0&fg=555555&s=0&c=20201002) is just ![\pi_R(s)[a_i]](https://s0.wp.com/latex.php?latex=%5Cpi_R%28s%29%5Ba_i%5D&bg=f0f0f0&fg=555555&s=0&c=20201002), where ![\pi_R](https://s0.wp.com/latex.php?latex=%5Cpi_R&bg=f0f0f0&fg=555555&s=0&c=20201002) is the optimal policy under the reward function ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002). Note that computing the optimal policy given the reward is in general non-trivial; except in simple cases, we typically approximate the policy using reinforcement learning (see Figure 1). Policies are usually assumed to be noisy (e.g. using a softmax instead of deterministically taking the best action). Due to the challenges of specifying priors, computing optimal policies and integrating over reward functions, most work in IRL uses some kind of approximation to the Bayesian objective (see the references in the introduction for some examples).

### **Recognizing Human Actions in Data**

IRL is a promising approach to learning human values in part because of the easy availability of data. For supervised learning, humans need to produce many labeled instances specialized for a task. IRL, by contrast, is an unsupervised/semi-supervised approach where any record of human behavior is a potential data source. Facebook’s logs of user behavior provide trillions of data-points. YouTube videos, history books, and literature are a trove of data on human behavior in both actual and imagined scenarios. However, while there is lots of existing data that is informative about human preferences, we argue that exploiting this data for IRL will be a difficult, complex task with current techniques.

*Inferring Reward Functions from Video Frames*

As we noted above, applications of IRL typically infer the reward function R from observed samples of the human policy ![\pi^*](https://s0.wp.com/latex.php?latex=%5Cpi%5E%2A&bg=f0f0f0&fg=555555&s=0&c=20201002). Formally, the environment is a known ![MDP\backslash R = (S,A,T)](https://s0.wp.com/latex.php?latex=MDP%5Cbackslash+R+%3D+%28S%2CA%2CT%29&bg=f0f0f0&fg=555555&s=0&c=20201002) and the observations are state-action pairs, ![(s,a) \sim pi^*](https://s0.wp.com/latex.php?latex=%28s%2Ca%29+%5Csim+pi%5E%2A&bg=f0f0f0&fg=555555&s=0&c=20201002). This assumes that (a) the environment’s dynamics ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002) are given as part of the IRL problem, and (b) the observations are structured as “state-action” pairs. When the data comes from a human expert parking a car, these assumptions are reasonable. The states and actions of the driver can be recorded and a car simulator can be used for ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002). For data from YouTube videos or history books, the assumptions fail. The data is a sequence of partial observations: the transition function ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002) is unknown and the data does not separate out *state* and *action*. Indeed, it’s a challenging ML problem to infer human actions from text or videos.

![image00.png](https://jsteinhardt.files.wordpress.com/2017/02/image00.png)

*Movie still: What actions are being performed in this situation?* ([Source](http://www.moviemoviesite.com/People/A/attenborough-richard/filmography.htm))

As a concrete example, suppose the data is a video of two co-pilots flying a plane. The successive frames provide only limited information about the state of the world at each time step and the frames often jump forward in time. So it’s more like a [POMDP](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process) with a complex observation model. Moreover, the actions of each pilot need to be inferred. This is a challenging inference problem, because actions can be subtle (e.g. when a pilot nudges the controls or nods to his co-pilot).

To infer actions from observations, some model relating the true state-action ![(s,a)](https://s0.wp.com/latex.php?latex=%28s%2Ca%29&bg=f0f0f0&fg=555555&s=0&c=20201002) to the observed video frame must be used. But choosing any model makes substantive assumptions about how human values relate to their behavior. For example, suppose someone attacks one of the pilots and (as a reflex) he defends himself by hitting back. Is this reflexive or instinctive response (hitting the attacker) an action that is informative about the pilot’s values? Philosophers and neuroscientists might investigate this by considering the mental processes that occur before the pilot hits back. If an IRL algorithm uses an off-the-shelf action classifier, it will lock in some (contentious) assumptions about these mental processes. At the same time, an IRL algorithm cannot *learn* such a model because it never directly observes the mental processes that relate rewards to actions.

*Inferring Policies From Video Frames*

When learning a reward function via IRL, the ultimate goal is to use the reward function to guide an artificial agent’s behavior (e.g. to perform useful tasks to humans). This goal can be formalized directly, without including IRL as an intermediate step. For example, in **Apprenticeship Learning,** the goal is to learn a “good” policy for the ![MDP\backslash R](https://s0.wp.com/latex.php?latex=MDP%5Cbackslash+R&bg=f0f0f0&fg=555555&s=0&c=20201002) from samples of the human’s policy ![\pi^*](https://s0.wp.com/latex.php?latex=%5Cpi%5E%2A&bg=f0f0f0&fg=555555&s=0&c=20201002) (where ![\pi^*](https://s0.wp.com/latex.php?latex=%5Cpi%5E%2A&bg=f0f0f0&fg=555555&s=0&c=20201002) is assumed to approximately optimize an unknown reward function). In **Imitation Learning,** the goal is simply to learn a policy that is similar to the human’s policy.

Like IRL, policy search techniques need to recognize an agent’s actions to infer their policy. So they have the same challenges as IRL in learning from videos or history books. Unlike IRL, policy search does not explicitly model the reward function that underlies an agent’s behavior. This leads to an additional challenge. Humans and AI systems face vastly different tasks and have different action spaces. Most actions in videos and books would never be performed by a software agent. Even when tasks are similar (e.g. humans driving in the 1930s vs. a self-driving car in 2016), it is a difficult [transfer learning](https://openreview.net/pdf?id=B16dGcqlx) problem to use human policies in one task to improve AI policies in another.

*IRL Needs Curated Data*

We argued that records of human behaviour in books and videos are difficult for IRL algorithms to exploit. Data from Facebook seems more promising: we can store the state (e.g. the HTML or pixels displayed to the human) and each human action (clicks and scrolling). This extends beyond Facebook to any task that can be performed on a computer. While this covers a broad range of tasks, there are obvious limitations. Many people in the world have a limited ability to use a computer: we can’t learn about their values in this way. Moreover, some kinds of human preferences (e.g. preferences over physical activities) seem hard to learn about from behaviour on a computer.

### **Information and Biases**

Human actions depend both on their preferences and their *beliefs*. The beliefs, like the preferences, are never directly observed. For narrow tasks (e.g. people choosing their favorite photos from a display), we can model humans as having full knowledge of the state (as in an [MDP](https://en.wikipedia.org/wiki/Markov_decision_process)). But for most real-world tasks, humans have limited information and their information changes over time (as in a [POMDP](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process) or [RL](https://en.wikipedia.org/wiki/Reinforcement_learning) problem). If IRL assumes the human has full information, then the model is mis-specified and generalizing about what the human would prefer in other scenarios can be mistaken. Here are some examples:

(1). Someone travels from their house to a cafe, which has already closed. If they are assumed to have full knowledge, then IRL would infer an alternative preference (e.g. going for a walk) rather than a preference to get a drink at the cafe.

(2). Someone takes a drug that is widely known to be ineffective. This could be because they have a false belief that the drug is effective, or because they picked up the wrong pill, or because they take the drug for its side-effects. Each possible explanation could lead to different conclusions about preferences.

(3). Suppose an IRL algorithm is inferring a person’s goals from key-presses on their laptop. The person repeatedly forgets their login passwords and has to reset them. This behavior is hard to capture with a POMDP-style model: humans forget some strings of characters and not others. IRL might infer that the person *intends* to repeatedly reset their passwords.

Example (3) above arises from humans forgetting information — even if the information is only a short string of characters. This is one way in which humans systematically deviate from rational Bayesian agents. The field of psychology has documented many other deviations. Below we discuss one such deviation — *time-inconsistency* — which has been used to explain temptation, addiction and procrastination.

*Time-inconsistency and Procrastination*

An IRL algorithm is inferring Alice’s preferences. In particular, the goal is to infer Alice’s preference for completing a somewhat tedious task (e.g. writing a paper) as opposed to relaxing. Alice has ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002) days in which she could complete the task and IRL observes her working or relaxing on each successive day.

![image02](https://jsteinhardt.files.wordpress.com/2017/02/image02.png)

*Figure 2. MDP graph for choosing whether to “work” or “wait” (relax) on a task.*

Formally, let R be the preference/reward Alice assigns to completing the task. Each day, Alice can “work” (receiving cost ![w](https://s0.wp.com/latex.php?latex=w&bg=f0f0f0&fg=555555&s=0&c=20201002) for doing tedious work) or “wait” (cost ![0](https://s0.wp.com/latex.php?latex=0&bg=f0f0f0&fg=555555&s=0&c=20201002)). If she works, she later receives the reward ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002) minus a tiny, linearly increasing cost (because it’s better to submit a paper earlier). Beyond the deadline at ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002), Alice cannot get the reward ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002). For IRL, we fix ![\epsilon](https://s0.wp.com/latex.php?latex=%5Cepsilon&bg=f0f0f0&fg=555555&s=0&c=20201002) and ![w](https://s0.wp.com/latex.php?latex=w&bg=f0f0f0&fg=555555&s=0&c=20201002) and infer ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002).

Suppose Alice chooses “wait” on Day 1. If she were fully rational, it follows that R (the preference for completing the task) is small compared to ![w](https://s0.wp.com/latex.php?latex=w&bg=f0f0f0&fg=555555&s=0&c=20201002) (the psychological cost of doing the tedious work). In other words, Alice doesn’t care much about completing the task. Rational agents will do the task on Day 1 or never do it. Yet humans often care deeply about tasks yet leave them until the last minute (when finishing early would be optimal). Here we imagine that Alice has 9 days to complete the task and waits until the last possible day.

![image03.png](https://jsteinhardt.files.wordpress.com/2017/02/image03.png)

*Figure 3: Graph showing IRL inferences for Optimal model (which is mis-specified) and Possibly Discounting Model (which includes hyperbolic discounting). On each day (*![x](https://s0.wp.com/latex.php?latex=x&bg=f0f0f0&fg=555555&s=0&c=20201002)*-axis) the model gets another observation of Alice’s choice. The* ![y](https://s0.wp.com/latex.php?latex=y&bg=f0f0f0&fg=555555&s=0&c=20201002)*-axis shows the posterior mean for* ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002) *(reward for task), where the tedious work* ![w = -1](https://s0.wp.com/latex.php?latex=w+%3D+-1&bg=f0f0f0&fg=555555&s=0&c=20201002)*.*

Figure 3 shows results from running IRL on this problem. There is an “Optimal” model, where the agent is optimal up to an unknown level of softmax random noise (a typical assumption for IRL). There is also a “Possibly Discounting” model, where the agent is either softmax optimal or is a hyperbolic discounter (with unknown level of discounting). We do joint Bayesian inference over the completion reward ![R](https://s0.wp.com/latex.php?latex=R&bg=f0f0f0&fg=555555&s=0&c=20201002), the softmax noise and (for “Possibly Discounting”) how much the agent hyperbolically discounts. The work cost ![w](https://s0.wp.com/latex.php?latex=w&bg=f0f0f0&fg=555555&s=0&c=20201002) is set to ![-1](https://s0.wp.com/latex.php?latex=-1&bg=f0f0f0&fg=555555&s=0&c=20201002). Figure 3 shows that after 6 days of observing Alice procrastinate, the “Optimal” model is very confident that Alice does not care about the task ![(R < |w|)](https://s0.wp.com/latex.php?latex=%28R+%3C+%7Cw%7C%29&bg=f0f0f0&fg=555555&s=0&c=20201002). When Alice completes the task on the last possible day, the posterior mean on R is not much more than the prior mean. By contrast, the “Possibly Discounting” model never becomes confident that Alice doesn’t care about the task. (Note that the gap between the models would be bigger for larger ![T](https://s0.wp.com/latex.php?latex=T&bg=f0f0f0&fg=555555&s=0&c=20201002). The “Optimal” model’s posterior on R shoots back to its Day-0 prior because it explains the whole action sequence as due to high softmax noise — optimal agents without noise would either do the task immediately or not at all. Full details and code are [here](http://agentmodels.org/chapters/5d-joint-inference.html).)

### **Long-term Plans**

Agents will often take long series of actions that generate negative utility for them in the moment in order to accomplish a long-term goal (for instance, studying every night in order to perform well on a test). Such long-term plans can make IRL more difficult for a few reasons. Here we focus on two: (1) IRL systems may not have access to the right type of data for learning about long-term goals, and (2) needing to predict long sequences of actions can make algorithms more fragile in the face of model mis-specification.

*(1) Wrong type of data.* To make inferences based on long-term plans, it would be helpful to have coherent data about a single agent’s actions over a long period of time (so that we can e.g. see the plan unfolding). But in practice we will likely have substantially more data consisting of short snapshots of a large number of different agents (e.g. because many internet services already record user interactions, but it is uncommon for a single person to be exhaustively tracked and recorded over an extended period of time even while they are offline).

The former type of data (about a single representative population measured over time) is called **panel data**, while the latter type of data (about different representative populations measured at each point in time) is called **repeated cross-section data**. The differences between these two types of data is [well-studied](http://www.uio.no/studier/emner/sv/oekonomi/ECON5103/v10/undervisningsmateriale/PDAppl_17.pdf) in econometrics, and a general theme is the following: it is difficult to infer individual-level effects from cross-sectional data.

An easy and familiar example of this difference (albeit not in an IRL setting) can be given in terms of election campaigns. Most campaign polling is cross-sectional in nature: a different population of respondents is polled at each point in time. Suppose that Hillary Clinton gives a speech and her overall support according to cross-sectional polls increases by 2%; what can we conclude from this? Does it mean that 2% of people switched from Trump to Clinton? Or did 6% of people switch from Trump to Clinton while 4% switched from Clinton to Trump?

At a minimum, then, using cross-sectional data leads to a difficult disaggregation problem; for instance, different agents taking different actions at a given point in time could be due to being at different stages in the same plan, or due to having different plans, or some combination of these and other factors. Collecting demographic and other side data can help us (by allowing us to look at variation and shifts within each subpopulation), but it is unclear if this will be sufficient in general.

On the other hand, there are some services (such as Facebook or Google) that do have extensive data about individual users across a long period of time. However, this data has another issue: it is incomplete in a very systematic way (since it only tracks online behaviour). For instance, someone might go online most days to read course notes and Wikipedia for a class; this is data that would likely be recorded. However, it is less likely that one would have a record of that person taking the final exam, passing the class and then getting an internship based on their class performance. Of course, some pieces of this sequence would be inferable based on some people’s e-mail records, etc., but it would likely be under-represented in the data relative to the record of Wikipedia usage. In either case, some non-trivial degree of inference would be necessary to make sense of such data.

*(2) Fragility to mis-specification.* Above we discussed why observing only short sequences of actions from an agent can make it difficult to learn about their long-term plans (and hence to reason correctly about their values). Next we discuss another potential issue — fragility to model mis-specification.

Suppose someone spends 99 days doing a boring task to accomplish an important goal on day 100. A system that is only trying to correctly predict actions will be right 99% of the time if it predicts that the person inherently enjoys boring tasks. Of course, a system that understands the goal and how the tasks lead to the goal will be right 100% of the time, but even minor errors in its understanding could bring the accuracy back below 99%.

The general issue is the following: large changes in the model of the agent might only lead to small changes in the predictive accuracy of the model, and the longer the time horizon on which a goal is realized, the more this might be the case. This means that even slight mis-specifications in the model could tip the scales back in favor of a (very) incorrect reward function. A potential way of dealing with this might be to identify “important” predictions that seem closely tied to the reward function, and focus particularly on getting those predictions right (see [here](http://www.di.ens.fr/~slacoste/research/pubs/lacoste-AISTATS11-lossBayes.pdf) for a paper exploring a similar idea in the context of approximate inference).

One might object that this is only a problem in this toy setting; for instance, in the real world, one might look at the particular way in which someone is studying or performing some other boring task to see that it coherently leads towards some goal (in a way that would be less likely were the person to be doing something boring purely for enjoyment). In other words, correctly understanding the agent’s goals might allow for more fine-grained accurate predictions which would fare better under e.g. log-score than would an incorrect model.

This is a reasonable objection, but there are some historical examples of this going wrong that should give one pause. That is, there are historical instances where: (i) people expected a more complex model that seemed to get at some underlying mechanism to outperform a simpler model that ignored that mechanism, and (ii) they were wrong (the simpler model did better under log-score). The example we are most familiar with is n-gram models vs. parse trees for language modelling; the most successful language models (in terms of having the best log-score on predicting the next word given a sequence of previous words) essentially treat language as a high-order Markov chain or hidden Markov model, despite the fact that linguistic theory predicts that language should be tree-structured rather than linearly-structured. Indeed, NLP researchers have tried building language models that assume language is tree-structured, and these models perform worse, or at least do not seem to have been adopted in practice (this is true both for older discrete models and newer continuous models based on neural nets).  It’s plausible that a similar issue will occur in inverse reinforcement learning, where correctly inferring plans is not enough to win out in predictive performance. The reason for the two issues might be quite similar (in language modelling, the tree structure only wins out in statistically uncommon corner cases involving long-term and/or nested dependencies, and hence getting that part of the prediction correct doesn’t help predictive accuracy much).

The overall point is: in the case of even slight model mis-specification, the “correct” model might actually perform worse under typical metrics such as predictive accuracy. Therefore, more careful methods of constructing a model might be necessary.

### **Learning Values != Robustly Predicting Human Behaviour**

The problems with IRL described so far will result in poor performance for predicting human choices out-of-sample. For example, if someone is observed doing boring tasks for 99 days (where they only achieve the goal on Day 100), they’ll be predicted to continue doing boring tasks even when a short-cut to the goal becomes available. So even if the goal is simply to predict human behaviour (not to infer human values), mis-specification leads to bad predictions on realistic out-of-sample scenarios.

Let’s suppose that our goal is not to predict human behaviour but to create AI systems that promote and respect human values. These goals (predicting humans and building safe AI) are distinct. Here’s an example that illustrates the difference. Consider a long-term smoker, Bob, who would continue smoking even if there were (counterfactually) a universally effective anti-smoking treatment. Maybe Bob is in denial about the health effects of smoking or Bob thinks he’ll inevitably go back to smoking whatever happens. If an AI system were assisting Bob, we might expect it to avoid promoting his smoking habit (e.g. by not offering him cigarettes at random moments). This is not paternalism, where the AI system imposes someone else’s values on Bob. The point is that even if Bob would continue smoking across many counterfactual scenarios this doesn’t mean that he places value on smoking.

How do we choose between the theory that Bob values smoking and the theory that he does not (but smokes anyway because of the powerful addiction)? Humans choose between these theories based on our experience with addictive behaviours and our insights into people’s preferences and values. This kind of insight can’t easily be captured as formal assumptions about a model, or even as a criterion about counterfactual generalization. (The theory that Bob values smoking *does* make accurate predictions across a wide range of counterfactuals.) Because of this, learning human values from IRL has a more profound kind of model mis-specification than the examples in Jacob’s previous post. Even in the limit of data generated from an infinite series of random counterfactual scenarios, standard IRL algorithms would not infer someone’s true values.

Predicting human actions is neither necessary nor sufficient for learning human values. In what ways, then, are the two related? One such way stems from the premise that if someone spends more resources making a decision, the resulting decision tends to be more in keeping with their true values. For instance, someone might spend lots of time thinking about the decision, they might consult experts, or they might try out the different options in a trial period before they make the real decision. Various authors have thus suggested that people’s choices under sufficient “reflection” act as a reliable indicator of their true values. Under this view, predicting a certain kind of behaviour (choices under reflection) is sufficient for learning human values. Paul Christiano has written about [some](https://medium.com/ai-control/model-free-decisions-6e6609f5d99e#.y6gg55z3r) [proposals](https://sideways-view.com/2016/12/01/optimizing-the-news-feed/) for doing this, though we will not discuss them here (the first link is for general AI systems while the second is for newsfeeds). In general, turning these ideas into algorithms that are tractable and learn safely remains a challenging problem.

### **Further reading**

There is [research](http://www.jmlr.org/papers/v12/choi11a.html) on doing IRL for agents in POMDPs. Owain and collaborators explored the effects of limited information and cognitive biases on IRL: [paper](https://arxiv.org/pdf/1512.05832), [paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.705.2548&rep=rep1&type=pdf), [online book](http://agentmodels.org).

For many environments it will not be possible to *identify* the reward function from the observed trajectories. These identification problems are related to the mis-specification problems but are not the same thing. Active learning can help with identification ([paper](https://arxiv.org/abs/1601.06569)).

Paul Christiano raised many similar points about mis-specification in a [post](https://medium.com/ai-control/the-easy-goal-inference-problem-is-still-hard-fad030e0a876#.mb33rtxo8) on his blog.

For a big-picture monograph on relations between human preferences, economic utility theory and welfare/well-being, see Hausman’s [“Preference, Value, Choice and Welfare”.](https://www.amazon.com/Preference-Choice-Welfare-Daniel-Hausman/dp/1107695120)

### **Acknowledgments**

Thanks to Sindy Li for reviewing a full draft of this post and providing many helpful comments. Thanks also to Michael Webb and Paul Christiano for doing the same on specific sections of the post.


Linear algebra fact

Here is interesting linear algebra fact: let ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) be an ![n \times n](https://s0.wp.com/latex.php?latex=n+%5Ctimes+n&bg=f0f0f0&fg=555555&s=0&c=20201002) matrix and ![u](https://s0.wp.com/latex.php?latex=u&bg=f0f0f0&fg=555555&s=0&c=20201002) be a vector such that ![u^{\top}A = \lambda u^{\top}](https://s0.wp.com/latex.php?latex=u%5E%7B%5Ctop%7DA+%3D+%5Clambda+u%5E%7B%5Ctop%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Then for any matrix ![B](https://s0.wp.com/latex.php?latex=B&bg=f0f0f0&fg=555555&s=0&c=20201002), ![u^{\top}((A-B)(\lambda I - B)^{-1}) = u^{\top}](https://s0.wp.com/latex.php?latex=u%5E%7B%5Ctop%7D%28%28A-B%29%28%5Clambda+I+-+B%29%5E%7B-1%7D%29+%3D+u%5E%7B%5Ctop%7D&bg=f0f0f0&fg=555555&s=0&c=20201002).

The proof is just basic algebra: ![u^{\top}(A-B)(\lambda I - B)^{-1} = (\lambda u^{\top} - u^{\top}B)(\lambda I - B)^{-1} = u^{\top}(\lambda I - B)(\lambda I - B)^{-1} = u^{\top}](https://s0.wp.com/latex.php?latex=u%5E%7B%5Ctop%7D%28A-B%29%28%5Clambda+I+-+B%29%5E%7B-1%7D+%3D+%28%5Clambda+u%5E%7B%5Ctop%7D+-+u%5E%7B%5Ctop%7DB%29%28%5Clambda+I+-+B%29%5E%7B-1%7D+%3D+u%5E%7B%5Ctop%7D%28%5Clambda+I+-+B%29%28%5Clambda+I+-+B%29%5E%7B-1%7D+%3D+u%5E%7B%5Ctop%7D&bg=f0f0f0&fg=555555&s=0&c=20201002).

Why care about this? Let’s imagine that ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) is a (not necessarily symmetric) stochastic matrix, so ![1^{\top}A = 1^{\top}](https://s0.wp.com/latex.php?latex=1%5E%7B%5Ctop%7DA+%3D+1%5E%7B%5Ctop%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). Let ![A-B](https://s0.wp.com/latex.php?latex=A-B&bg=f0f0f0&fg=555555&s=0&c=20201002) be a low-rank approximation to ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) (so ![A-B](https://s0.wp.com/latex.php?latex=A-B&bg=f0f0f0&fg=555555&s=0&c=20201002) consists of all the large singular values, and ![B](https://s0.wp.com/latex.php?latex=B&bg=f0f0f0&fg=555555&s=0&c=20201002) consists of all the small singular values). Unfortunately since ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) is not symmetric, this low-rank approximation doesn’t preserve the eigenvalues of ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002) and so we need not have ![1^{\top}(A-B) = 1^{\top}](https://s0.wp.com/latex.php?latex=1%5E%7B%5Ctop%7D%28A-B%29+%3D+1%5E%7B%5Ctop%7D&bg=f0f0f0&fg=555555&s=0&c=20201002). The ![(I-B)^{-1}](https://s0.wp.com/latex.php?latex=%28I-B%29%5E%7B-1%7D&bg=f0f0f0&fg=555555&s=0&c=20201002) can be thought of as a “correction” term such that the resulting matrix is still low-rank, but we’ve preserved one of the eigenvectors of ![A](https://s0.wp.com/latex.php?latex=A&bg=f0f0f0&fg=555555&s=0&c=20201002).


Prékopa–Leindler inequality

Consider the following statements:

1. The shape with the largest volume enclosed by a given surface area is the ![n](https://s0.wp.com/latex.php?latex=n&bg=f0f0f0&fg=555555&s=0&c=20201002)-dimensional sphere.
2. A marginal or sum of log-concave distributions is log-concave.
3. Any Lipschitz function of a standard ![n](https://s0.wp.com/latex.php?latex=n&bg=f0f0f0&fg=555555&s=0&c=20201002)-dimensional Gaussian distribution concentrates around its mean.

What do these all have in common? Despite being fairly non-trivial and deep results, they all can be proved in less than half of a page using the [Prékopa–Leindler inequality](https://en.wikipedia.org/wiki/Pr%C3%A9kopa%E2%80%93Leindler_inequality).

(I won’t show this here, or give formal versions of the statements above, but time permitting I will do so in a later blog post.)