Approach

We show that scaling a simple pre-training task is sufficient to achieve competitive zero-shot performance on a great variety of image classification datasets. Our method uses an abundantly available source of supervision: the text paired with images found across the internet. This data is used to create the following proxy training task for CLIP: given an image, predict which out of a set of 32,768 randomly sampled text snippets, was actually paired with it in our dataset.In order to solve this task, our intuition is that CLIP models will need to learn to recognize a wide variety of visual concepts in images and associate them with their names. As a result, CLIP models can then be applied to nearly arbitrary visual classification tasks. For instance, if the task of a dataset is classifying photos of dogs vs cats we check for each image whether a CLIP model predicts the text description 'a photo of a dog' or 'a photo of a cat' is more likely to be paired with it.CLIP was designed to mitigate a number of major problems in the standard deep learning approach to computer vision:Costly datasets: Deep learning needs a lot of data, and vision models have traditionally been trained on manually labeled datasets that are expensive to construct and only provide supervision for a limited number of predetermined visual concepts. The ImageNet dataset, one of the largest efforts in this space, required over 25,000 workers to annotate 14 million images for 22,000 object categories. In contrast, CLIP learns from text–image pairs that are already publicly available on the internet. Reducing the need for expensive large labeled datasets has been extensively studied by prior work, notably self-supervised learning,18,19,20 contrastive methods,21,22,23,24,25 self-training approaches,26,27 and generative modeling.28,29,30,31Narrow: An ImageNet model is good at predicting the 1000 ImageNet categories, but that's all it can do 'out of the box.' If we wish to perform any other task, an ML practitioner needs to build a new dataset, add an output head, and fine-tune the model. In contrast, CLIP can be adapted to perform a wide variety of visual classification tasks without needing additional training examples. To apply CLIP to a new task, all we need to do is 'tell' CLIP's text-encoder the names of the task's visual concepts, and it will output a linear classifier of CLIP's visual representations. The accuracy of this classifier is often competitive with fully supervised models.We show random, non-cherry picked, predictions of zero-shot CLIP classifiers on examples from various datasets below.guacamoleguacamolecevicheedamametuna tartarehummustelevision studiotelevision studiopodium indoorconference roomlecture roomcontrol roomairplane, personairplanebirdbeargiraffecarannual crop landpermanent crop landpasture landhighway or roadannual crop landbrushland or shrublandhealthy lymph node tissuelymph node tumor tissuehealthy lymph node tissuelynxfox squirrelmongooseskunkred foxlynxPoor real-world performance: Deep learning systems are often reported to achieve human or even superhuman performance32,A[A]In 2015, a group of researchers from Microsoft first trained a model which achieved a top-5 accuracy on ImageNet that surpassed reported human top-5 accuracy.3334


Conclusion

With CLIP, we've tested whether task agnostic pre-training on internet scale natural language, which has powered a recent breakthrough in NLP, can also be leveraged to improve the performance of deep learning for other fields. We are excited by the results we've seen so far applying this approach to computer vision. Like the GPT family, CLIP learns a wide variety of tasks during pre-training which we demonstrate via zero-shot transfer. We are also encouraged by our findings on ImageNet that suggest zero-shot evaluation is a more representative measure of a model's capability.


Key takeaways

1. CLIP is highly efficientCLIP learns from unfiltered, highly varied, and highly noisy data, and is intended to be used in a zero-shot manner. We know from GPT-2 and 3 that models trained on such data can achieve compelling zero shot performance; however, such models require significant training compute. To reduce the needed compute, we focused on algorithmic ways to improve the training efficiency of our approach.We report two algorithmic choices that led to significant compute savings. The first choice is the adoption of a contrastive objective for connecting text with images.35,21,17 We originally explored an image-to-text approach, similar to VirTex,15 but encountered difficulties scaling this to achieve state-of-the-art performance. In small to medium scale experiments, we found that the contrastive objective used by CLIP is 4x to 10x more efficient at zero-shot ImageNet classification. The second choice was the adoption of the Vision Transformer,36 which gave us a further 3x gain in compute efficiency over a standard ResNet. In the end, our best performing CLIP model trains on 256 GPUs for 2 weeks which is similar to existing large scale image models.37,27,38,36We originally explored training image-to-caption language models but found this approach struggled at zero-shot transfer. In this 16 GPU day experiment, a language model only achieves 16% accuracy on ImageNet after training for 400 million images. CLIP is much more efficient and achieves the same accuracy roughly 10x faster.2. CLIP is flexible and generalBecause they learn a wide range of visual concepts directly from natural language, CLIP models are significantly more flexible and general than existing ImageNet models. We find they are able to zero-shot perform many different tasks. To validate this we have measured CLIP's zero-shot performance on over 30 different datasets including tasks such as fine-grained object classification, geo-localization, action recognition in videos, and OCR.B[B]While CLIP's zero-shot OCR performance is mixed, its semantic OCR representation is quite useful. When evaluated on the SST-2 NLP dataset rendered as images, a linear classifer on CLIP's representation matches a CBoW model with direct access to the text. CLIP is also competitive at detecting hateful memes without needing ground truth text.This finding is also reflected on a standard representation learning evaluation using linear probes. The best CLIP model outperforms the best publicly available ImageNet model, the Noisy Student EfficientNet-L2,27 on 20 out of 26 different transfer datasets we tested.


Broader impacts

CLIP allows people to design their own classifiers and removes the need for task-specific training data. The manner in which these classes are designed can heavily influence both model performance and model biases. For example, we find that when given a set of labels including Fairface39 race labelsC[C]FairFace is a face image dataset designed to balance age, gender, and race, in order to reduce asymmetries common in previous face datasets. It categorizes gender into 2 groups: female and male and race into 7 groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino. There are inherent problems with race and gender classifications, as e.g. Bowker and Star (2000)40 and Keyes (2018)41 have shown. While FairFace's dataset reduces the proportion of White faces, it still lacks representation of entire large demographic groups, effectively erasing such categories. We use the 2 gender categories and 7 race categories defined in the FairFace dataset in a number of our experiments not in order to reinforce or endorse the use of such reductive categories, but in order to enable us to make comparisons to prior work.Additionally, given that CLIP does not need task-specific training data it can unlock certain niche tasks with greater ease. Some of these tasks may raise privacy or surveillance related risks and we explore this concern by studying the performance of CLIP on celebrity identification. CLIP has a top-1 accuracy of 59.2% for 'in the wild' celebrity image classification when choosing from 100 candidates and a top-1 accuracy of 43.3% when choosing from 1000 possible choices. Although it's noteworthy to achieve these results with task agnostic pre-training, this performance is not competitive when compared to widely available production level models. We further explore challenges that CLIP poses in our paper and we hope that this work motivates future research on the characterization of the capabilities, shortcomings, and biases of such models. We are excited to engage with the research community on such questions.


Limitations

While CLIP usually performs well on recognizing common objects, it struggles on more abstract or systematic tasks such as counting the number of objects in an image and on more complex tasks such as predicting how close the nearest car is in a photo. On these two datasets, zero-shot CLIP is only slightly better than random guessing. Zero-shot CLIP also struggles compared to task specific models on very fine-grained classification, such as telling the difference between car models, variants of aircraft, or flower species.CLIP also still has poor generalization to images not covered in its pre-training dataset. For instance, although CLIP learns a capable OCR system, when evaluated on handwritten digits from the MNIST dataset, zero-shot CLIP only achieves 88% accuracy, well below the 99.75% of humans on the dataset. Finally, we've observed that CLIP's zero-shot classifiers can be sensitive to wording or phrasing and sometimes require trial and error 'prompt engineering' to perform well.


Prove that if ∣x−2∣=p|x - 2| = p∣x−2∣=p, where x<2x < 2x<2, then x−p=2−2px - p = 2 - 2px−p=2−2p.

Problem 4Adapted from IMO 1964 Problem 2Suppose aaa, bbb, ccc are the sides of a triangle.Prove that a2(b+c−a)+b2(c+a−b)+c2(a+b−c)≤3abca^2(b + c − a) + b^2(c + a − b) + c^2(a + b − c) \leq 3abca2(b+c−a)+b2(c+a−b)+c2(a+b−c)≤3abc.FormalInformal theorem imo_1964_p2 (a b c : ℝ) (h₀ : 0 < a ∧ 0 < b ∧ 0 < c) (h₁ : c < a + b) (h₂ : b < a + c) (h₃ : a < b + c) : a^2 * (b + c - a) + b^2 * (c + a - b) + c^2 * (a + b - c) ≤ 3 * a * b * c := begin -- Arguments to `nlinarith` are fully invented by our model .  nlinarith [sq_nonneg (b - a), sq_nonneg (c - b), sq_nonneg (c - a)] end Problem 5Adapted from AIME 1984 Problem 1Prove that a2+a4+a6+a8+...+a98=93a2 + a4 + a6 + a8 + ...+ a98 = 93a2+a4+a6+a8+...+a98=93 if a1a1a1, a2a2a2, a3...a3...a3...  is an arithmetic progression with common difference 111, and a1+a2+a3+...+a98=137a1 + a2 + a3 + ...  + a98 = 137a1+a2+a3+...+a98=137.FormalInformal theorem aime_1984_p1 (u : ℕ → ℚ) (h₀ : ∀ n, u (n + 1) = u n + 1) (h₁ : ∑ k in finset.range 98, u k.succ = 137) : ∑ k in finset.range 49, u (2 * k.succ) = 93 := begin rw finset.sum_eq_multiset_sum, dsimp [finset.range] at h₁, simp [h₀], ring, norm_num at h₁, norm_num, apply eq_of_sub_eq_zero, { simp only [*, abs_of_pos, add_zero] at *, linarith }, end Problem 6Adapted from IMO Longlist 1990 Problem 77For a,b,ca, b, ca,b,c reals, prove that (a2+ab+b2)(b2+bc+c2)(c2+ca+a2)≥(ab+bc+ca)3(a^2 + ab + b^2)(b^2 + bc + c^2)(c^2 + ca + a^2) \geq (ab + bc + ca)^3(a2+ab+b2)(b2+bc+c2)(c2+ca+a2)≥(ab+bc+ca)3.FormalInformal theorem imo_longlist_1990_p77 (a b c : ℝ) : (a * b + b * c + c * a)^3 ≤ (a^2 + a * b + b^2) * (b^2 + b * c + c^2) * (c^2 + c * a + a^2) := begin -- The three initial steps use Cauchy–Schwarz to prove -- `(a * b + b * c) ^ 2 ≤ (a ^ 2 + b ^ 2) * (b ^ 2 + c ^ 2)` -- which is required for the final call to `nlinarith`.  let u : euclidean_space ℝ (fin 2) := ![a, b], let v : euclidean_space ℝ (fin 2) := ![b, c], have h₀ := real_inner_mul_inner_self_le u v, simp [u, v, fin.sum_univ_succ, ←pow_two, ←pow_two, le_of_lt, mul_assoc] at h₀, -- The model introduces another required cut (i.e.  invent -- the term `0 ≤ (c + a) * (c + a)` and proves it)


FormalInformalhave h₂ : abs (x - 2) = -(x - 2), {We observe that the capability to generate original mathematical terms required as arguments of tactics, which cannot be done without a neural language model, emerges from our training procedure.  The proof below is an example of it: the proof step use n + 1 (entirely generated by our models) proposes to use n + 1 as a solution, the rest of the formal proof relying on the ring_exp tactic to verify that it is indeed valid.  Problem 2Adapted from AMC12B 2020 Problem 6For all integers n≥9n ≥ 9n≥9, prove that ((n+2)!−(n+1)!)/n!((n + 2)! −(n + 1)!) / n!((n+2)!−(n+1)!)/n! is a perfect square.FormalInformal theorem amc12b_2020_p6 (n : ℕ) (h0 : 9 ≤ n) : ∃ x : ℕ, (x:ℝ)^2 = (nat.factorial (n + 2) - nat.factorial (n + 1)) / nat.factorial n := begin -- The model directly proposes `n + 1` as solution.  use n + 1, field_simp [nat.factorial_ne_zero, pow_succ'], ring_exp end We also observe that our models and search procedure are capable of producing proofs that chain multiple non-trivial reasoning steps.  In the proof below, the model starts by using contraposition leading to the existential statement (∃ (x : ℝ), f x ≠ a * x + b).  It then generates a witness for it with use (0 : ℝ) and finishes the proof by leveraging the norm_num tactic.  Problem 3Adapted from the MATH datasetLet f(x)=Ax+Bf(x) = Ax + Bf(x)=Ax+B and g(x)=Bx+Ag(x) = Bx + Ag(x)=Bx+A, where A
eBA
e BA
eB.  If f(g(x))−g(f(x))=B−Af(g(x)) - g(f(x)) = B - Af(g(x))−g(f(x))=B−A, prove that A+B=0A + B = 0A+B=0.FormalInformal theorem mathd_train_algebra_217 (a b : ℝ) (f g : ℝ → ℝ) (h₀ : ∀ x, f x = a * x + b) (h₁ : ∀ x, f x = b * x + a) (h₂ : a ≠ b) (h₃ : ∀ x, f (g x) - g (f x) = b - a) : a + b = 0 := begin revert h₀ h₁ h₂ h₃, -- Initial contraposition.  contrapose!, rintro ⟨h₀, ⟨h₁, h₂⟩⟩, -- The model proposes `0` as witness for the current -- goal that consists in `∃ (x : ℝ), f x ≠ a * x + b`.  use (0 : ℝ), simp only [sub_eq_iff_eq_add, h₀, mul_zero, zero_add], norm_num at h₀, end Our models, trained with statement curriculum learning, were able to close a variety of problems from training textbooks as well as AMC12 and AIME competitions, and 2 problems adapted from the IMO.  We present below three examples of such generated proofs


have h₃ : 0 ≤ (c + a) * (c + a), { nlinarith, }, have h₄ := sq_nonneg (a * b + b * c + c * a), simp [sq, h₀, h₃, mul_add, add_mul] at h₄ ⊢, nlinarith [sq_nonneg (b - a), sq_nonneg (c - b), sq_nonneg (a - c)] end Formal mathematics involves two main challenges that make a naive application of reinforcement learning unlikely to succeed.(i) Infinite action space: not only does formal mathematics have an extremely large search space (like Go for example), it also has an infinite action space .  At each step of a proof search, the model must choose not from a well-behaved finite set of actions, but a complex and infinite set of tactics, involving exogenous mathematical terms that have to be generated (e.g., generating a mathematical statement to be used as a witness, an object used in steps such as 'there exists an x s.t.  …', or a cut, the introduction and the chaining of a lemma in the middle of a proof).(ii) Lack of self-play: conversely to 2-player games, a prover is not playing against an opponent but against a set of statements to prove.  When faced with a statement that is just too hard, there is no obvious reframing that will let the prover generate intermediary easier statements to tackle first.  This asymmetry prevents naive application of the self-play algorithms that were successful with 2-player games.In our work, we address the infinite action space problem by sampling actions from a language model as we search for a proof.  Language models have the capability to generate the tactic calls as well as the original mathematical terms often required as arguments.  Our basis for addressing the lack of self-play is the observation that the key role of self-play in 2-player games is to provide an unsupervised curriculum.  Our methodology proposes to replace this unsupervised curriculum with an auxiliary set of problem statements (without requiring proofs) of varying difficulty.  We empirically show that, when the difficulty of these auxiliary problems is varied enough, our training procedure is able to solve a curriculum of increasingly difficult problems, eventually generalizing to the set of problems we care about.While these results are extremely exciting, as they demonstrate that deep learning models are capable of non-trivial mathematical reasoning when interacting with a formal system, we are still very far from best-student performance on these competitions, only occasionally, rather than consistently, closing challenging olympiad problems.  We hope nonetheless that our work will motivate research in this domain, in particular towards the IMO Grand Challenge and that the statement curriculum learning methodology we propose will help accelerate progress in automated reasoning in general.


For all integers n≥9n ≥ 9n≥9, prove that ((n+2)!−(n+1)!)/n!((n + 2)! −(n + 1)!) / n!((n+2)!−(n+1)!)/n! is a perfect square.

FormalInformaluse n + 1,


Let f(x)=Ax+Bf(x) = Ax + Bf(x)=Ax+B and g(x)=Bx+Ag(x) = Bx + Ag(x)=Bx+A, where A
eBA
e BA
eB. If f(g(x))−g(f(x))=B−Af(g(x)) - g(f(x)) = B - Af(g(x))−g(f(x))=B−A, prove that A+B=0A + B = 0A+B=0.

FormalInformalcontrapose!,use (0 : ℝ),


Prove that a2+a4+a6+a8+...+a98=93a2 + a4 + a6 + a8 + ...+ a98 = 93a2+a4+a6+a8+...+a98=93 if a1a1a1, a2a2a2, a3...a3...a3... is an arithmetic progression with common difference 111, and a1+a2+a3+...+a98=137a1 + a2 + a3 + ... + a98 = 137a1+a2+a3+...+a98=137.

FormalInformal


For a,b,ca, b, ca,b,c reals, prove that (a2+ab+b2)(b2+bc+c2)(c2+ca+a2)≥(ab+bc+ca)3(a^2 + ab + b^2)(b^2 + bc + c^2)(c^2 + ca + a^2) \geq (ab + bc + ca)^3(a2+ab+b2)(b2+bc+c2)(c2+ca+a2)≥(ab+bc+ca)3.

FormalInformalhave h₀ := real_inner_mul_inner_self_le u v,have h₃ : 0 ≤ (c + a) * (c + a),


Suppose aaa, bbb, ccc are the sides of a triangle.Prove that a2(b+c−a)+b2(c+a−b)+c2(a+b−c)≤3abca^2(b + c − a) + b^2(c + a − b) + c^2(a + b − c) \leq 3abca2(b+c−a)+b2(c+a−b)+c2(a+b−c)≤3abc.

FormalInformalnlinarith [sq_nonneg (b - a),sq_nonneg (c - b),sq_nonneg (c - a)]


The challenges of GPU programming

The architecture of modern GPUs can be roughly divided into three major components—DRAM, SRAM and ALUs—each of which must be considered when optimizing CUDA code:Memory transfers from DRAM must be coalesced into large transactions to leverage the large bus width of modern memory interfaces.Data must be manually stashed to SRAM prior to being re-used, and managed so as to minimize shared memory bank conflicts upon retrieval.Computations must be partitioned and scheduled carefully, both across and within Streaming Multiprocessors (SMs), so as to promote instruction/thread-level parallelism and leverage special-purpose ALUs (e.g., tensor cores).Basic architecture of a GPU.Reasoning about all these factors can be challenging, even for seasoned CUDA programmers with many years of experience. The purpose of Triton is to fully automate these optimizations, so that developers can better focus on the high-level logic of their parallel code. Triton aims to be broadly applicable, and therefore does not automatically schedule work across SMs -- leaving some important algorithmic considerations (e.g. tiling, inter-SM synchronization) to the discretion of developers.Compiler optimizations in CUDA vs Triton.


Programming model

Out of all the Domain Specific Languages and JIT-compilers available, Triton is perhaps most similar to Numba: kernels are defined as decorated Python functions, and launched concurrently with different program_id's on a grid of so-called instances. However, as shown in the code snippet below, the resemblance stops there: Triton exposes intra-instance parallelism via operations on blocks—small arrays whose dimensions are powers of two—rather than a Single Instruction, Multiple Thread (SIMT)7 execution model. In doing so, Triton effectively abstracts away all the issues related to concurrency within CUDA thread blocks (e.g., memory coalescing, shared memory synchronization/conflicts, tensor core scheduling).Vector addition in Triton.While this may not be particularly helpful for embarrassingly parallel (i.e., element-wise) computations, it can greatly simplify the development of more complex GPU programs.Consider for example the case of a fused softmax kernel (below) in which each instance normalizes a different row of the given input tensor X∈RM×N. Standard CUDA implementations of this parallelization strategy can be challenging to write, requiring explicit synchronization between threads as they concurrently reduce the same row of X. Most of this complexity goes away with Triton, where each kernel instance loads the row of interest and normalizes it sequentially using NumPy-like primitives.Note that the Triton JIT treats X and Y as pointers rather than tensors; we felt like retaining low-level control of memory accesses was important to address more complex data structures (e.g., block-sparse tensors).Importantly, this particular implementation of softmax keeps the rows of X in SRAM throughout the entire normalization process, which maximizes data reuse when applicable (~<32K columns). This differs from PyTorch's internal CUDA code, whose use of temporary memory makes it more general but significantly slower (below). The bottom line here is not that Triton is inherently better, but that it simplifies the development of specialized kernels that can be much faster than those found in general-purpose libraries.5,00010,000N05001000 GB/sTritonTorch (native)Torch (jit)A100 performance of fused softmax for M=4096.The lower performance of the Torch (v1.9) JIT highlights the difficulty of automatic CUDA code generation from sequences of high-level tensor operations.


Matrix multiplication

Being able to write fused kernels for element-wise operations and reductions is important, but not sufficient given the prominence of matrix multiplication tasks in neural networks. As it turns out, Triton also works very well for those, achieving peak performance with just ~25 lines of Python code. On the other hand, implementing something similar in CUDA would take a lot more effort and would even be likely to achieve lower performance.


Matrix multiplication in Triton.

A, B, C, M, N, K, stride_am, stride_ak, stride_bk, stride_bn, stride_cm, stride_cn, **METAOne important advantage of handwritten matrix multiplication kernels is that they can be customized as desired to accommodate fused transformations of their inputs (e.g., slicing) and outputs (e.g., Leaky ReLU). Without a system like Triton, non-trivial modifications of matrix multiplication kernels would be out-of-reach for developers without exceptional GPU programming expertise.1,0002,0003,0004,000M = N = K020406080100 TFLOPscuBLASTritonTriton + LeakyReLUcuBLAS +torch.nn.LeakyReLUV100 tensor-core performance of matrix multiplication with appropriately tuned values for BLOCKM_MM, BLOCKN_NN, BLOCKK_KK, GROUPM_MM.


High-level system architecture

The good performance of Triton comes from a modular system architecture centered around Triton-IR, an LLVM-based intermediate representation in which multi-dimensional blocks of values are first-class citizens.0];0];0];0];The @triton.jit decorator works by walking the Abstract Syntax Tree (AST) of the provided Python function so as to generate Triton-IR on-the-fly using a common SSA construction algorithm.8 The resulting IR code is then simplified, optimized and automatically parallelized by our compiler backend, before being converted into high-quality LLVM-IR—and eventually PTX—for execution on recent NVIDIA GPUs. CPUs and AMD GPUs are not supported at the moment, but we welcome community contributions aimed at addressing this limitation.


Compiler backend

We have found that the use of blocked program representations via Triton-IR allows our compiler to automatically perform a wide variety of important program optimizations. For example, data can be automatically stashed to shared memory by looking at the operands of computationally intensive block-level operations (e.g., tl.dot)—and allocated/synchronized using standard liveness analysis techniques.On the other hand, Triton programs can be efficiently and automatically parallelized both (1) across SMs by executing different kernel instances concurrently, and (2) within SMs by analyzing the iteration space of each block-level operation and partitioning it adequately across different SIMD units, as shown below. Definition of a Triton program P composed of three statements S1, S2, S3Iteration space of S3 Mapping of S3 onto a Stream Multiprocessor (SM) Mapping of P onto the GPU Definition of a Triton program P composed of three statements S1, S2, S3Iteration space of S3 Mapping of S3 onto a Stream Multiprocessor (SM) Mapping of P onto the GPU


Contributing

We intend for Triton to become a community-driven project. Feel free to fork our repository on GitHub!If you're interested in joining our team and working on Triton & GPU kernels, we're hiring!


Why Dota?

We started OpenAI Five in order to work on a problem that felt outside of the reach of existing deep reinforcement learningA[A]Deep reinforcement learning is the idea of training a deep neural network to achieve goals using rewards and punishmentsTo build OpenAI Five, we created a system called Rapid which let us run PPO at previously unprecedented scale. The results exceeded our wildest expectations, and we produced a world-class Dota bot without hitting any fundamental performance limits.The surprising power of today's RL algorithms comes at the cost of massive amounts of experience, which can be impractical outside of a game or simulated environment. This limitation may not be as bad as sounds—for example, we used Rapid to control a robotic hand to dexterously reorient a block, trained entirely in simulation and executed on a physical robot. But we think decreasing the amount of experience is a next challenge for RL.We are retiring OpenAI Five as a competitor today, but progress made and technology developed will continue to drive our future work. This isn't the end of our Dota work—we think that Dota is a much more intrinsically interesting and difficult (and now well-understood!) environment for RL development than the standard ones used today.


Compute

OpenAI Five's victories on Saturday, as compared to its losses at The International 2018, are due to a major change: 8x more training compute. In many previous phases of the project, we'd drive further progress by increasing our training scale. But after The International, we'd already dedicated the vast majority of our project's compute to training a single OpenAI Five model. So we increased the scale of compute in the only way available to us: training for longer.In total, the current version of OpenAI Five has consumed 800 petaflop/s-days and experienced about 45,000 years of Dota self-play over 10 realtime months (up from about 10,000 years over 1.5 realtime months as of The International), for an average of 250 years of simulated experience per day. The Finals version of OpenAI Five has a 99.9% winrate versus the TI version.B[B]Winrate evaluated on the current game patch. This biases the winrate towards the Finals version as the TI version was trained on an older patch, but currently we don't have another way to compare agents trained on different game versions.


Transfer learning

The current version of OpenAI Five has been training continuously since June 2018, despite changes to the model size and the game rules (including some fairly large game patch updates and newly implemented features). In each case, we were able to transfer the model over and continue training—something that is an open challenge for RL in other domains. To the best of our knowledge, this is the first time an RL agent has been trained using such a long-lived training run.To make this work, we've continued to flesh out our surgery tooling so that we can start from trained parameters even across substantial architecture changes.


Cooperative mode

SheeverOpenAI Five's ability to play with humans presents a compelling vision for the future of human-AI interaction, one where AI systems collaborate and enhance the human experience. Our testers reported feeling supported by their bot teammates, that they learned from playing alongside these advanced systems, and that it was generally a fun experience overall. Note that OpenAI Five exhibits zero-shot transfer learning—it was trained to have all heroes controlled by copies of itself, but generalizes to controlling a subset of heroes, playing with or against humans. We were very surprised this worked as well as it did. In fact, we'd considered doing a cooperative match at The International but assumed it'd require dedicated training.


More heroes

We saw very little slowdown in training going from 5 to 18 heroes. We hypothesized the same would be true going to even more heroes, and after The International, we put a lot of effort into integrating new ones.We spent several weeks training with hero pools up to 25 heroes, bringing those heroes to approximately 5k MMR (about 95th percentile of Dota players). Although they were still improving, they weren't learning fast enough to reach pro level before Finals. We haven't yet had time to investigate why, but our hypotheses range from insufficient model capacity to needing better matchmaking for the expanded hero pool to requiring more training time for new heroes to catch up to old heroes. Imagine how hard it is for a human to learn a new hero when everyone else has mastered theirs!We believe these issues are fundamentally solvable, and solving them could be interesting in its own right. The Finals version plays with 17 heroes—we removed Lich because his abilities were changed significantly in Dota version 7.20.


Arena

We're launching OpenAI Five Arena, a public experiment where we'll let anyone play OpenAI Five in both competitive and cooperative modes. We'd known that our 1v1 bot would be exploitable through clever strategies; we don't know to what extent the same is true of OpenAI Five, but we're excited to invite the community to help us find out!Arena opens Thursday, April 18 at 6pm PST and will close 11:59pm PST on Sunday, April 21. Please register so we can ensure there's enough server capacity in your region! Results of all games will be automatically reported to the Arena public leaderboard.We're incredibly grateful for all the support the Dota community has shown us over the past two years, and we hope that Arena will also serve as one small way of giving back. Have fun with it!


What's next

We will be releasing a more technical analysis of OpenAI Five once we've reviewed the outcomes of OpenAI Five Arena.Afterwards, we'll continue working with the Dota 2 environment within OpenAI. We've seen rapid progress in the past two years on RL capabilities, and we think that Dota 2 will continue to help us push forward what's possible—whether with achieving competent performance from less data or true human-AI cooperation.If you are interested in advancing AI capabilities and helping further our mission of ensuring they benefit humanity, we're hiring!


Samples

Since MuseNet knows many different styles, we can blend generations in novel ways.A[A]If you're interested in other projects for creating AI generated music using transformers, we recommend checking out Magenta's piano generation work.We're excited to see how musicians and non-musicians alike will use MuseNet to create new compositions!1In simple mode (shown by default), you'll hear random uncurated samples that we've pre-generated. Choose a composer or style, an optional start of a famous piece, and start generating. This lets you explore the variety of musical styles the model can create. In advanced mode you can interact with the model directly. The completions will take longer, but you'll be creating an entirely new piece.Some of MuseNet's limitations include:The instruments you ask for are strong suggestions, not requirements. MuseNet generates each note by calculating the probabilities across all possible notes and instruments. The model shifts to make your instrument choices more likely, but there's always a chance it will choose something else.MuseNet has a more difficult time with odd pairings of styles and instruments (such as Chopin with bass and drums). Generations will be more natural if you pick instruments closest to the composer or band's usual style.


Composer and instrumentation tokens

We created composer and instrumentation tokens to give more control over the kinds of samples MuseNet generates. During training time, these composer and instrumentation tokens were prepended to each sample, so the model would learn to use this information in making note predictions. At generation time, we can then condition the model to create samples in a chosen style by starting with a prompt such as a Rachmaninoff piano start:Or prompted with the band Journey, with piano, bass, guitar, and drums:We can visualize the embeddings from MuseNet to gain insight into what the model has learned. Here we use t-SNE to create a 2-D map of the cosine similarity of various musical composer and style embeddings.ChopinLisztBroadwayWagnerMozartFleetwood MacNine Inch NailsRobbie WilliamsThe Beach BoysThe BeatlesMariah CareyShania TwainMichael JacksonRicky MartinDvorakBeethovenHaydnEnyaDebussyElvis PresleySpice GirlsMadonnaJazzGershwinClementiFaureBon JoviBachBluesLady GagaAfricanAdeleKaty PerryBollywoodTchaikovskyRachmaninoffRavelSchubertAlbenizGranadosMendelssohnSatieScarlattiFranckPachelbelByrdWhitney HoustonJourneyBrahmsGreen DayBob MarleyBritney SpearsQueenAretha FranklinSchumannJoplinDisney


Long-term structure

MuseNet uses the recompute and optimized kernels of Sparse Transformer to train a 72-layer network with 24 attention heads—with full attention over a context of 4096 tokens. This long context may be one reason why it is able to remember long-term structure in a piece, like in the following sample imitating Chopin:It can also create musical melodic structures, as in this sample imitating Mozart:Music generation is a useful domain for testing the Sparse Transformer as it sits on a middle ground between text and images. It has the fluid token structure of text (in images you can look back N tokens and find the row above, whereas in music there's not a fixed number for looking back to the previous measure). Yet we can easily hear whether the model is capturing long term structure on the order of hundreds to thousands of tokens. It's much more obvious if a music model messes up structure by changing the rhythm, in a way that it's less clear if a text model goes on a brief tangent.


Dataset

We collected training data for MuseNet from many different sources. ClassicalArchives and BitMidi donated their large collections of MIDI files for this project, and we also found several collections online, including jazz, pop, African, Indian, and Arabic styles. Additionally, we used the MAESTRO dataset.The transformer is trained on sequential data: given a set of notes, we ask it to predict the upcoming note. We experimented with several different ways to encode the MIDI files into tokens suitable for this task. First, a chordwise approach that considered every combination of notes sounding at one time as an individual 'chord', and assigned a token to each chord. Second, we tried condensing the musical patterns by only focusing on the starts of notes, and tried further compressing that using a byte pair encoding scheme.We also tried two different methods of marking the passage of time: either tokens that were scaled according to the piece's tempo (so that the tokens represented a musical beat or fraction of a beat), or tokens that marked absolute time in seconds. We landed on an encoding that combines expressivity with conciseness: combining the pitch, volume, and instrument information into a single token.Sample encoding which combines pitch, volume, and instrument.During training, we:Transpose the notes by raising and lowering the pitches (later in training, we reduce the amount of transposition so that generations stay within the individual instrument ranges).Augment the volumes, turning up or turning down the overall volumes of the various samples.Augment timing (when using the absolute time in seconds encoding), effectively slightly slowing or speeding up the pieces.Use mixup on the token embedding spaceWe also create an inner critic: the model is asked during training time to predict whether a given sample is truly from the dataset or if it is one of the model's own past generations. This score is used to select samples at generation time.


Embeddings

We added several different kinds of embeddings to give the model more structural context. In addition to the standard positional embeddings, we added a learned embedding that tracks the passage of time in a given sample. This way, all of the notes that sound at the same time are given the same timing embedding. We then add an embedding for each note in a chord (this mimics relative attention, since it will be easier for the model to learn that note 4 needs to look back at note 3, or else at note 4 of the previous chord). Finally, we add two structural embeddings which tell the model where a given musical sample is within the larger musical piece. One embedding divides the larger piece into 128 parts, while the second encoding is a countdown from 127 to 0 as the model approaches the (end) token.We're excited to hear what people create! If you create a piece you like, you can upload it to a free service like Instaudio and then tweet us the link (the MuseNet demo has a tweet button to help with this).If you're interested in learning more about OpenAI's music work, consider applying to join our team. Please feel free to email us with suggestions for the MuseNet demo. We'd also love to hear from you if you're interested in composing with MuseNet in more depth, or if you have MIDI files you'd like to add to the training set.MuseNet played an experimental concert on April 25th, 2019, livestreamed on OpenAI's Twitch channel, in which no human (including us) had heard the pieces before.


Core values

AGI focusWe are committed to building safe, beneficial AGI that will have a massive positive impact on humanity's future.Anything that doesn't help with that is out of scope.Intense and scrappyBuilding something exceptional requires hard work (often on unglamorous stuff) and urgency; everything (that we choose to do) is important.Be unpretentious and do what works; find the best ideas wherever they come from.ScaleWe believe that scale—in our models, our systems, ourselves, our processes, and our ambitions—is magic. When in doubt, scale it up.Make something people loveOur technology and products should have a transformatively positive effect on people's lives.Team spiritOur biggest advances, and differentiation, come from effective collaboration in and across teams. Although our teams have increasingly different identities and priorities, the overall purpose and goals have to remain perfectly aligned.Nothing is someone else's problem.


For employees

Health, dental, and vision insuranceMental healthcare support and servicesCommuter benefits401(k) with generous matchingDomestic conference budget for each employee


Benefits

We know that well-being is at the center of everyday life. Our comprehensive benefits package supports you where you are, so you can take care of what matters today while confidently planning for the future.For employeesHealth, dental, and vision insuranceMental healthcare support and servicesCommuter benefits401(k) with generous matchingDomestic conference budget for each employeeLife & familyFlexible working hours20 weeks paid parental leaveLife and AD&D insuranceFertility treatment and family planning coverageFlexible spending account for healthcare and dependent careCulture & developmentAnnual learning & development stipendEmployee resource groupsTeam-driven celebrations and eventsDaily breakfast, lunch, and dinnerFreshly baked cookies


Culture & development

Annual learning & development stipendEmployee resource groupsTeam-driven celebrations and eventsDaily breakfast, lunch, and dinnerFreshly baked cookies


Life & family

Flexible working hours20 weeks paid parental leaveLife and AD&D insuranceFertility treatment and family planning coverageFlexible spending account for healthcare and dependent care


Featured roles

View all open rolesSoftware Engineer, Leverage EngineeringSan Francisco, California, United States — EngineeringApply nowWorkplace CoordinatorSan Francisco, California, United States — WorkplaceApply nowAccount AssociateSan Francisco, California, United States — Go To MarketApply nowLegal Program ManagerSan Francisco, California, United States — LegalApply nowSales EngineerSan Francisco, California, United States — Go To MarketApply now


Resources

OpenAI CharterLearn moreOpenAI interview guideLearn moreCommitment to diversity, equity & inclusionLearn more


Life at OpenAI

The power of continuous learningDec 23, 2022December 23, 2022Discovering the minutiae of backend systemsDec 8, 2022December 8, 2022I believe that my colleagues recognize the importance of what they're building and genuinely care about the outcomes—this wasn't necessarily the case with previous technological breakthroughs.Anna MakanjuPublic Policy at OpenAI


OpenAI Residency links

Read about OpenAI Residency


Commitment to diversity, equity & inclusion

Learn more


OpenAI Charter

Learn more


OpenAI interview guide

Learn more


Samples


Composer and instrumentation tokens


Long-term structure


Embeddings


Dataset


Deep attention

In Transformers, every output element is connected to every input element, and the weightings between them are dynamically calculated based upon the circumstances, a process called attention. While it is believed that this allows Transformers to be more flexible than models with fixed connectivity patterns, in practice it requires the creation of an N×N N\times N N×N attention matrix for every layer and attention head, which can consume large amounts of memory when applied to data types with many elements, like images or raw audio.Attention memory usage for a deep Transformer (64 layers and 4 heads) when matrices are stored in memory or recomputed during the backward pass. For reference, standard GPUs used for deep learning typically have memory of 12-32 GB.One way to reduce this is by recomputing the attention matrix from checkpoints during backpropagation, a well-established technique in deep learning for reducing memory usage at the cost of more computation. When done for the attention matrix in Transformers, it means the largest memory cost becomes independent of the number of layers, letting us train networks with substantially greater depth than possible previously. In practice, we found that Transformers with depth up to 128 layers outperformed shallower networks on benchmark tasks like CIFAR-10.To train these models with increased depth, we made several adjustments to the ordering of operations in the transformer and modified the initialization scheme. Full details can be seen in our paper.


Sparse attention

Even computing a single attention matrix, however, can become impractical for very large inputs. We instead use sparse attention patterns, where each output position only computes weightings from a subset of input positions. When the subset is small relative to the full set of inputs (say, N \sqrt{N} N elements instead of N N N elements), the resulting attention computation becomes tractable even for very long sequences, with an algorithmic complexity of O(NN) O(N \sqrt{N}) O(NN) instead of O(N2) O(N^2) O(N2).To assess the feasibility of the approach, we first visualized the learned attention patterns for deep Transformers on images, finding that many showed interpretable and structured sparsity patterns. Each of the below images shows which input pixels (highlighted in white) are attended to by a given attention head in order to predict the next value in the image. When the input portions are focused on small subsets and show a high degree of regularity, the layer is amenable to sparsification. A sampling of them are displayed here for a 128-layer model on CIFAR-10 images:Layer 19Layer 20Layer 6Layer 36While many layers displayed sparse structure, some layers clearly display dynamic attention that stretch over the entirety of the image. In order to preserve the ability of our network to learn such patterns, we implemented a two-dimensional factorization of the attention matrix, where the network can attend to all positions through two steps of sparse attention.Normal transformerStrided attentionFixed attentionThe first version, strided attention, is roughly equivalent to each position attending to its row and its column, and is similar to the attention pattern learned by the network above. (Note that the column attention can be equivalently formulated as attending to the row of the transposed matrix). The second version, fixed attention, attends to a fixed column and the elements after the latest column element, a pattern we found useful for when the data didn't fit into a two-dimensional structure (like text). For more details, we refer readers to our paper.


Experimental results

Sparse Transformers set new state-of-the-art scores for density estimation of CIFAR-10, Enwik8, and Imagenet 64.Density modeling performance in bits per byte (or dim) on a variety of benchmark datasets. M denotes millions of parameters used in the network, W the width of the network, L the number of layers, and H the number of heads.We also found that sparse attention achieved lower loss than full attention, in addition to being significantly faster (see our paper for comparisons). This may point to a useful inductive bias from our sparsity patterns, or an underlying optimization issue with dense attention.


Code release

Normally, implementing sparse attention would involve slicing query and key matrices in blocks, so to ease experimentation we implemented a set of block-sparse kernels which efficiently perform these operations on the the GPU. We open-source these kernels and provide example sparse attention functions in this repository.


Generating images

Transformers that use sparse attention seem to have a notion of global structure, which can be qualitatively evaluated by looking at image completions. Here we visualize a model trained on 64×64 64\times 64 64×64 ImageNet:We also generated fully unconditional samples with an unadjusted softmax temperature of 1.0. These models are trained using the maximum likelihood objective, which is well-known to cover all modes of the data (including potentially nonexistent ones) instead of increasing fidelity of a smaller portion of the data. Sampling from these models with unadjusted temperature lets us see the full distribution of images that the model believes exists in the world. As a result, some samples can appear strange.Model samplesReal data


Future work and limitations

The sparse attention patterns we introduced are only preliminary steps in the direction of efficient modeling of long sequences. We think exploring different patterns and combinations of sparsity is useful, and that learning sparse patterns is a particularly promising avenue of research for the next generation of neural network architectures.Even with the improvements we described above, autoregressive sequence generation still seems impractical for very high resolution images or video. The optimized attention operations we have introduced, however, may be useful primitives to combine with other approaches to modeling high dimensional data, like multi-scale approaches.If you are interested in advancing AI capabilities and helping further our mission of ensuring they benefit humanity, we're hiring!


Generating raw audio waveforms

Sparse Transformers can also be adapted to generate raw audio instead of images by simply changing the position embeddings. As deep learning expands to novel data types, we believe the ease of specifying inductive biases with this class of networks will be a useful tool.This model was trained on raw classical music clips and uses sparse attention to generate sequences of length 65,000. This corresponds to ~5 seconds of raw audio, and we have concatenated several samples together in each of the clips below.


The problem

One AI milestone is to exceed human capabilities in a complex video game like StarCraft or Dota. Relative to previous AI milestones like Chess or Go, complex video games start to capture the messiness and continuous nature of the real world. The hope is that systems which solve complex video games will be highly general, with applications outside of games.Dota 2 is a real-time strategy game played between two teams of five players, with each player controlling a character called a 'hero'. A Dota-playing AI must master the following:Long time horizons. Dota games run at 30 frames per second for an average of 45 minutes, resulting in 80,000 ticks per game. Most actions (like ordering a hero to move to a location) have minor impact individually, but some individual actions like town portal usage can affect the game strategically; some strategies can play out over an entire game. OpenAI Five observes every fourth frame, yielding 20,000 moves. Chess usually ends before 40 moves, Go before 150 moves, with almost every move being strategic.Partially-observed state. Units and buildings can only see the area around them. The rest of the map is covered in a fog hiding enemies and their strategies. Strong play requires making inferences based on incomplete data, as well as modeling what one's opponent might be up to. Both chess and Go are full-information games.High-dimensional, continuous action space. In Dota, each hero can take dozens of actions, and many actions target either another unit or a position on the ground. We discretize the space into 170,000 possible actions per hero (not all valid each tick, such as using a spell on cooldown); not counting the continuous parts, there are an average of ~1,000 valid actions each tick. The average number of actions in chess is 35; in Go, 250.High-dimensional, continuous observation space. Dota is played on a large continuous map containing ten heroes, dozens of buildings, dozens of NPC units, and a long tail of game features such as runes, trees, and wards. Our model observes the state of a Dota game via Valve's Bot API as 20,000 (mostly floating-point) numbers representing all information a human is allowed to access. A chess board is naturally represented as about 70 enumeration values (a 8x8 board of 6 piece types and minor historical info); a Go board as about 400 enumeration values (a 19x19 board of 2 piece types plus Ko).The Dota rules are also very complex — the game has been actively developed for over a decade, with game logic implemented in hundreds of thousands of lines of code. This logic takes milliseconds per tick to execute, versus nanoseconds for Chess or Go engines. The game also gets an update about once every two weeks, constantly changing the environment semantics.


Our approach

Our system learns using a massively-scaled version of Proximal Policy Optimization. Both OpenAI Five and our earlier 1v1 bot learn entirely from self-play. They start with random parameters and do not use search or bootstrap from human replays.preemptibleRL researchers (including ourselves) have generally believed that long time horizons would require fundamentally new advances, such as hierarchical reinforcement learning. Our results suggest that we haven't been giving today's algorithms enough credit — at least when they're run at sufficient scale and with a reasonable way of exploring.Our agent is trained to maximize the exponentially decayed sum of future rewards, weighted by an exponential decay factor called γ. During the latest training run of OpenAI Five, we annealed γ from 0.998 (valuing future rewards with a half-life of 46 seconds) to 0.9997 (valuing future rewards with a half-life of five minutes). For comparison, the longest horizon in the PPO paper was a half-life of 0.5 seconds, the longest in the Rainbow paper was a half-life of 4.4 seconds, and the Observe and Look Further paper used a half-life of 46 seconds.While the current version of OpenAI Five is weak at last-hitting (observing our test matches, the professional Dota commentator Blitz estimated it around median for Dota players), its objective prioritization matches a common professional strategy. Gaining long-term rewards such as strategic map control often requires sacrificing short-term rewards such as gold gained from farming, since grouping up to attack towers takes time. This observation reinforces our belief that the system is truly optimizing over a long horizon.OpenAI Five: Dota Gamplay4:20


Model structure

Each of OpenAI Five's networks contain a single-layer, 1024-unit LSTM that sees the current game state (extracted from Valve's Bot API) and emits actions through several possible action heads. Each head has semantic meaning, for example, the number of ticks to delay this action, which action to select, the X or Y coordinate of this action in a grid around the unit, etc. Action heads are computed independently.Interactive demonstration of the observation space and action space used by OpenAI Five. OpenAI Five views the world as a list of 20,000 numbers, and takes an action by emitting a list of 8 enumeration values. Select different actions and targets to understand how OpenAI Five encodes each action, and how it observes the world. The image shows the scene as a human would see it.Scene 3: Pushing Bottom Tower×Scene 3: Pushing Bottom Tower×OpenAI Five can react to missing pieces of state that correlate with what it does see. For example, until recently OpenAI Five's observations did not include shrapnel zones (areas where projectiles rain down on enemies), which humans see on screen. However, we observed OpenAI Five learning to walk out of (though not avoid entering) active shrapnel zones, since it could see its health decreasing.


Coordination

OpenAI Five does not contain an explicit communication channel between the heroes' neural networks. Teamwork is controlled by a hyperparameter we dubbed 'team spirit'. Team spirit ranges from 0 to 1, putting a weight on how much each of OpenAI Five's heroes should care about its individual reward function versus the average of the team's reward functions. We anneal its value from 0 to 1 over training.


Exploration

Given a learning algorithm capable of handling long horizons, we still need to explore the environment. Even with our restrictions, there are hundreds of items, dozens of buildings, spells, and unit types, and a long tail of game mechanics to learn about—many of which yield powerful combinations. It's not easy to explore this combinatorially-vast space efficiently.OpenAI Five learns from self-play (starting from random weights), which provides a natural curriculum for exploring the environment. To avoid 'strategy collapse', the agent trains 80% of its games against itself and the other 20% against its past selves. In the first games, the heroes walk aimlessly around the map. After several hours of training, concepts such as laning, farming, or fighting over mid emerge. After several days, they consistently adopt basic human strategies: attempt to steal Bounty runes from their opponents, walk to their tier one towers to farm, and rotate heroes around the map to gain lane advantage. And with further training, they become proficient at high-level strategies like 5-hero push.In March 2017, our first agent defeated bots but got confused against humans. To force exploration in strategy space, during training (and only during training) we randomized the properties (health, speed, start level, etc.) of the units, and it began beating humans. Later on, when a test player was consistently beating our 1v1 bot, we increased our training randomizations and the test player started to lose. (Our robotics team concurrently applied similar randomization techniques to physical robots to transfer from simulation to the real world.)OpenAI Five uses the randomizations we wrote for our 1v1 bot. It also uses a new 'lane assignment' one. At the beginning of each training game, we randomly 'assign' each hero to some subset of lanes and penalize it for straying from those lanes until a randomly-chosen time in the game.Exploration is also helped by a good reward. Our reward consists mostly of metrics humans track to decide how they're doing in the game: net worth, kills, deaths, assists, last hits, and the like. We postprocess each agent's reward by subtracting the other team's average reward to prevent the agents from finding positive-sum situations.We hardcode item and skill builds (originally written for our scripted baseline), and choose which of the builds to use at random. Courier management is also imported from the scripted baseline.


The games

Thus far OpenAI Five has played (with our restrictions) versus each of these teams:Best OpenAI employee team: 2.5k MMR (46th percentile)Best audience players watching OpenAI employee match (including Blitz, who commentated the first OpenAI employee match): 4–6k MMR (90th-99th percentile), though they'd never played as a team.Valve employee team: 2.5–4k MMR (46th-90th percentile).Amateur team: 4.2k MMR (93rd percentile), trains as a team.Semi-pro team: 5.5k MMR (99th percentile), trains as a team.The April 23rd version of OpenAI Five was the first to beat our scripted baseline. The May 15th version of OpenAI Five was evenly matched versus team 1, winning one game and losing another. The June 6th version of OpenAI Five decisively won all its games versus teams 1–3. We set up informal scrims with teams 4 & 5, expecting to lose soundly, but OpenAI Five won two of its first three games versus both.The teamwork aspect of the bot was just overwhelming. It feels like five selfless players that know a good general strategy.BlitzWe observed that OpenAI Five:Repeatedly sacrificed its own safe lane (top lane for dire; bottom lane for radiant) in exchange for controlling the enemy's safe lane, forcing the fight onto the side that is harder for their opponent to defend. This strategy emerged in the professional scene in the last few years, and is now considered to be the prevailing tactic. Blitz commented that he only learned this after eight years of play, when Team Liquid told him about it.Pushed the transitions from early- to mid-game faster than its opponents. It did this by: (1) setting up successful ganks (when players move around the map to ambush an enemy hero—see animation) when players overextended in their lane, and (2) by grouping up to take towers before the opponents could organize a counterplay.Deviated from current playstyle in a few areas, such as giving support heroes (which usually do not take priority for resources) lots of early experience and gold. OpenAI Five's prioritization allows for its damage to peak sooner and push its advantage harder, winning team fights and capitalizing on mistakes to ensure a fast win.Trophies awarded after the match between the best players at OpenAI and our bot team. One trophy for the humans, one trophy for the bots (represented by Susan Zhang from our team!)


Rapid

Our system is implemented as a general-purpose RL training system called Rapid, which can be applied to any Gym environment. We've used Rapid to solve other problems at OpenAI, including Competitive Self-Play.The training system is separated into rollout workers, which run a copy of the game and an agent gathering experience, and optimizer nodes, which perform synchronous gradient descent across a fleet of GPUs. The rollout workers sync their experience through Redis to the optimizers. Each experiment also contains workers evaluating the trained agent versus reference agents, as well as monitoring software such as TensorBoard, Sentry, and Grafana.During synchronous gradient descent, each GPU computes a gradient on its part of the batch, and then the gradients are globally averaged. We originally used MPI's allreduce for averaging, but now use our own NCCL2 wrappers that parallelize GPU computations and network data transfer.The latencies for synchronizing 58MB of data (size of OpenAI Five's parameters) across different numbers of GPUs are shown on the right. The latency is low enough to be largely masked by GPU computation which runs in parallel with it.We've implemented Kubernetes, Azure, and GCP backends for Rapid.


Surprising findings

Binary rewards can give good performance. Our 1v1 model had a shaped reward, including rewards for last hits, kills, and the like. We ran an experiment where we only rewarded the agent for winning or losing, and it trained an order of magnitude slower and somewhat plateaued in the middle, in contrast to the smooth learning curves we usually see. The experiment ran on 4,500 cores and 16 k80 GPUs, training to the level of semi-pros (70 TrueSkill) rather than 90 TrueSkill of our best 1v1 bot).Creep blocking can be learned from scratch. For 1v1, we learned creep blocking using traditional RL with a 'creep block' reward. One of our team members left a 2v2 model training when he went on vacation (proposing to his now wife!), intending to see how much longer training would boost performance. To his surprise, the model had learned to creep block without any special guidance or reward.We're still fixing bugs. The chart shows a training run of the code that defeated amateur players, compared to a version where we simply fixed a number of bugs, such as rare crashes during training, or a bug which resulted in a large negative reward for reaching level 25. It turns out it's possible to beat good humans while still hiding serious bugs!A subset of the OpenAI Dota team, holding the laptop that defeated the world's top professionals at Dota 1v1 at The International last year.*


Differences versus humans

OpenAI Five is given access to the same information as humans, but instantly sees data like positions, healths, and item inventories that humans have to check manually. Our method isn't fundamentally tied to observing state, but just rendering pixels from the game would require thousands of GPUs.OpenAI Five averages around 150-170 actions per minute (and has a theoretical maximum of 450 due to observing every 4th frame). Frame-perfect timing, while possible for skilled players, is trivial for OpenAI Five. OpenAI Five has an average reaction time of 80ms, which is faster than humans.These differences matter most in 1v1 (where our bot had a reaction time of 67ms), but the playing field is relatively equitable as we've seen humans learn from and adapt to the bot. Dozens of professionals used our 1v1 bot for training in the months after last year's TI. According to Blitz, the 1v1 bot has changed the way people think about 1v1s (the bot adopted a fast-paced playstyle, and everyone has now adapted to keep up).


What's next

Our team is focused on making our August goal. We don't know if it will be achievable, but we believe that with hard work (and some luck) we have a real shot.This post described a snapshot of our system as of June 6th. We'll release updates along the way to surpassing human performance and write a report on our final system once we complete the project. Please join us on August 5th virtually or in person, when we'll play a team of top players!Our underlying motivation reaches beyond Dota. Real-world AI deployments will need to deal with the challenges raised by Dota which are not reflected in Chess, Go, Atari games, or Mujoco benchmark tasks. Ultimately, we will measure the success of our Dota system in its application to real-world tasks. If you'd like to be part of what comes next, we're hiring!


The task

Dactyl is a system for manipulating objects using a Shadow Dexterous Hand. We place an object such as a block or a prism in the palm of the hand and ask Dactyl to reposition it into a different orientation; for example, rotating the block to put a new face on top. The network observes only the coordinates of the fingertips and the images from three regular RGB cameras.Although the first humanoid hands were developed decades ago, using them to manipulate objects effectively has been a long-standing challenge in robotic control. Unlike other problems such as locomotion, progress on dextrous manipulation using traditional robotics approaches has been slow, and current techniques remain limited in their ability to manipulate objects in the real world.Reorienting an object in the hand requires the following problems to be solved:Working in the real world. Reinforcement learning has shown many successes in simulations and video games, but has had comparatively limited results in the real world. We test Dactyl on a physical robot.High-dimensional control. The Shadow Dexterous Hand has 24 degrees of freedom compared to 7 for a typical robot arm.Noisy and partial observations. Dactyl works in the physical world and therefore must handle noisy and delayed sensor readings. When a fingertip sensor is occluded by other fingers or by the object, Dactyl has to work with partial information. Many aspects of the physical system like friction and slippage are not directly observable and must be inferred.Manipulating more than one object. Dactyl is designed to be flexible enough to reorient multiple kinds of objects. This means that our approach cannot use strategies that are only applicable to a specific object geometry.


Our approach

Dactyl learns to solve the object reorientation task entirely in simulation without any human input. After this training phase, the learned policy works on the real robot without any fine-tuning.Learning dexterity: uncut7:15Learning methods for robotic manipulation face a dilemma. Simulated robots can easily provide enough data to train complex policies, but most manipulation problems can't be modeled accurately enough for those policies to transfer to real robots. Even modeling what happens when two objects touch—the most basic problem in manipulation—is an active area of research with no widely accepted solution. Training directly on physical robots allows the policy to learn from real-world physics, but today's algorithms would require years of experience to solve a problem like object reorientation.Our approach, domain randomization, learns in a simulation which is designed to provide a variety of experiences rather than maximizing realism. This gives us the best of both approaches: by learning in simulation, we can gather more experience quickly by scaling up, and by de-emphasizing realism, we can tackle problems that simulators can only model approximately.It's been shown (by OpenAI and others) that domain randomization can work on increasingly complex problems—domain randomizations were even used to train OpenAI Five. Here, we wanted to see if scaling up domain randomization could solve a task well beyond the reach of current methods in robotics.We built a simulated version of our robotics setup using the MuJoCo physics engine. This simulation is only a coarse approximation of the real robot:Measuring physical attributes like friction, damping, and rolling resistance is cumbersome and difficult. They also change over time as the robot experiences wear and tear.MuJoCo is a rigid body simulator, which means that it cannot simulate the deformable rubber found at the fingertips of the hand or the stretching of tendons.Our robot can only manipulate the object by repeatedly making contact with it. However, contact forces are notoriously difficult to reproduce accurately in simulation.The simulation can be made more realistic by calibrating its parameters to match robot behavior, but many of these effects simply cannot be modeled accurately in current simulators.Instead, we train the policy on a distribution of simulated environments where the physical and visual attributes are chosen randomly. Randomized values are a natural way to represent the uncertainties that we have about the physical system and also prevent overfitting to a single simulated environment. If a policy can accomplish the task across all of the simulated environments, it will more likely be able to accomplish it in the real world.


Learning to control

By building simulations that support transfer, we have reduced the problem of controlling a robot in the real world to accomplishing a task in simulation, which is a problem well-suited for reinforcement learning. While the task of manipulating an object in a simulated hand is already somewhat difficult, learning to do so across all combinations of randomized physical parameters is substantially more difficult.To generalize across environments, it is helpful for the policy to be able to take different actions in environments with different dynamics. Because most dynamics parameters cannot be inferred from a single observation, we used an LSTM—a type of neural network with memory—to make it possible for the network to learn about the dynamics of the environment. The LSTM achieved about twice as many rotations in simulation as a policy without memory.Dactyl learns using Rapid, the massively scaled implementation of Proximal Policy Optimization developed to allow OpenAI Five to solve Dota 2. We use a different model architecture, environment, and hyperparameters than OpenAI Five does, but we use the same algorithms and training code. Rapid used 6144 CPU cores and 8 GPUs to train our policy, collecting about one hundred years of experience in 50 hours.For development and testing, we validated our control policy against objects with embedded motion tracking sensors to isolate the performance of our control and vision networks.


Results

When deploying our system, we noticed that Dactyl uses a rich set of in-hand dexterous manipulation strategies to solve the task. These strategies are commonly used by humans as well. However, we do not teach them to our system explicitly; all behaviors are discovered autonomously.autonomously learnedDactyl grasp types according to the GRASP taxonomy. Top left to bottom right: Tip Pinch, Palmar Pinch, Tripod, Quadpod, Power grasp, and 5-Finger Precision grasp.We observed that for precision grasps, such as the Tip Pinch grasp, Dactyl uses the thumb and little finger. Humans tend to use the thumb and either the index or middle finger instead. However, the robot hand's little finger is more flexible due to an extra degree of freedom, which may explain why Dactyl prefers it. This means that Dactyl can rediscover grasps found in humans, but adapt them to better fit the limitations and abilities of its own body.


Learning to see

Dactyl was designed to be able to manipulate arbitrary objects, not just those that have been specially modified to support tracking. Therefore, Dactyl uses regular RGB camera images to estimate the position and orientation of the object.We train a pose estimator using a convolutional neural network. The neural network takes the video streams from three cameras positioned around the robot hand and outputs the estimated position and orientation of the object. We use multiple cameras to resolve ambiguities and occlusion. We again use domain randomization to train this network only in simulation using the Unity game development platform, which can model a wider variety of visual phenomena than Mujoco.By combining these two independent networks, the control network that reorients the object given its pose and the vision network that maps images from cameras to the object's pose, Dactyl can manipulate an object by seeing it.Example training images used for learning to estimate the pose of the block.


Transfer performance

We tested how many rotations Dactyl could achieve before it dropped the object, timed out, or reached 50 successes. Our policies trained purely in simulation were able to successfully manipulate objects in the real world.Dactyl lab setup with Shadow Dexterous Hand, PhaseSpace motion tracking cameras, and Basler RGB cameras.For the task of block manipulation, policies trained with randomization could achieve many more rotations than those trained without randomization, as can be seen in the results below. Also, using the control network with pose estimated from vision performs nearly as well as reading the pose directly from motion tracking sensors.


Learning progress

The vast majority of training time is spent making the policy robust to different physical dynamics. Learning to rotate an object in simulation without randomizations requires about 3 years of simulated experience, while achieving similar performance in a fully randomized simulation requires about 100 years of experience.Learning progress with and without randomizations over years of simulated experience.


What surprised us

Tactile sensing is not necessary to manipulate real-world objects. Our robot receives only the locations of the five fingertips along with the position and orientation of the cube. Although the robot hand has touch sensors on its fingertips, we didn't need to use them. Generally, we found better performance from using a limited set of sensors that could be modeled effectively in the simulator instead of a rich sensor set with values that were hard to model.Randomizations developed for one object generalize to others with similar properties. After developing our system for the problem of manipulating a block, we printed an octagonal prism, trained a new policy using its shape, and attempted to manipulate it. Somewhat to our surprise, it achieved high performance using only the randomizations we had designed for the block. By contrast, a policy that manipulated a sphere could only achieve a few successes in a row, perhaps because we had not randomized any simulation parameters that model rolling behavior.With physical robots, good systems engineering is as important as good algorithms. At one point, we noticed that one engineer consistently achieved much better performance than others when running the exact same policy. We later discovered that he had a faster laptop, which hid a timing bug that reduced performance. After the bug was fixed, performance improved for the rest of the team.


What didn't pan out

We also found to our surprise that a number of commonly employed techniques did not improve our results.Decreasing reaction time did not improve performance. Conventional wisdom states that reducing the time between actions should improve performance because the changes between states are smaller and therefore easier to predict. Our current time between actions is 80ms, which is smaller than human reaction time of 150-250ms, but significantly larger than neural network computation time of roughly 25ms. Surprisingly, decreasing time between actions to 40ms required additional training time but did not noticeably improve performance in the real world. It's possible that this rule of thumb is less applicable to neural network models than to the linear models that are in common use today.Using real data to train our vision policies didn't make a difference. In early experiments, we used a combination of simulated and real data to improve our models. The real data was gathered from trials of our policy against an object with embedded tracking markers. However, real data has significant disadvantages compared to simulated data. Position information from tracking markers has latency and measurement error. Worse, real data is easily invalidated by common configuration changes, making it a hassle to collect enough to be useful. As our methods developed, our simulator-only error improved until it matched our error from using a mixture of simulated and real data. Our final vision models were trained without real data.This project completes a full cycle of AI development that OpenAI has been pursuing for the past two years: we've developed a new learning algorithm, scaled it massively to solve hard simulated tasks, and then applied the resulting system to the real world. Repeating this cycle at increasing scale is the primary route we are pursuing to increase the capabilities of today's AI systems towards safe artificial general intelligence. If you'd like to be part of what comes next, we're hiring!


Why Dota?

We started OpenAI Five in order to work on a problem that felt outside of the reach of existing deep reinforcement learningA[A]Deep reinforcement learning is the idea of training a deep neural network to achieve goals using rewards and punishmentsTo build OpenAI Five, we created a system called Rapid which let us run PPO at previously unprecedented scale. The results exceeded our wildest expectations, and we produced a world-class Dota bot without hitting any fundamental performance limits.The surprising power of today's RL algorithms comes at the cost of massive amounts of experience, which can be impractical outside of a game or simulated environment. This limitation may not be as bad as sounds—for example, we used Rapid to control a robotic hand to dexterously reorient a block, trained entirely in simulation and executed on a physical robot. But we think decreasing the amount of experience is a next challenge for RL.We are retiring OpenAI Five as a competitor today, but progress made and technology developed will continue to drive our future work. This isn't the end of our Dota work—we think that Dota is a much more intrinsically interesting and difficult (and now well-understood!) environment for RL development than the standard ones used today.


Compute

OpenAI Five's victories on Saturday, as compared to its losses at The International 2018, are due to a major change: 8x more training compute. In many previous phases of the project, we'd drive further progress by increasing our training scale. But after The International, we'd already dedicated the vast majority of our project's compute to training a single OpenAI Five model. So we increased the scale of compute in the only way available to us: training for longer.In total, the current version of OpenAI Five has consumed 800 petaflop/s-days and experienced about 45,000 years of Dota self-play over 10 realtime months (up from about 10,000 years over 1.5 realtime months as of The International), for an average of 250 years of simulated experience per day. The Finals version of OpenAI Five has a 99.9% winrate versus the TI version.B[B]Winrate evaluated on the current game patch. This biases the winrate towards the Finals version as the TI version was trained on an older patch, but currently we don't have another way to compare agents trained on different game versions.


Cooperative mode

SheeverOpenAI Five's ability to play with humans presents a compelling vision for the future of human-AI interaction, one where AI systems collaborate and enhance the human experience. Our testers reported feeling supported by their bot teammates, that they learned from playing alongside these advanced systems, and that it was generally a fun experience overall. Note that OpenAI Five exhibits zero-shot transfer learning—it was trained to have all heroes controlled by copies of itself, but generalizes to controlling a subset of heroes, playing with or against humans. We were very surprised this worked as well as it did. In fact, we'd considered doing a cooperative match at The International but assumed it'd require dedicated training.


More heroes

We saw very little slowdown in training going from 5 to 18 heroes. We hypothesized the same would be true going to even more heroes, and after The International, we put a lot of effort into integrating new ones.We spent several weeks training with hero pools up to 25 heroes, bringing those heroes to approximately 5k MMR (about 95th percentile of Dota players). Although they were still improving, they weren't learning fast enough to reach pro level before Finals. We haven't yet had time to investigate why, but our hypotheses range from insufficient model capacity to needing better matchmaking for the expanded hero pool to requiring more training time for new heroes to catch up to old heroes. Imagine how hard it is for a human to learn a new hero when everyone else has mastered theirs!We believe these issues are fundamentally solvable, and solving them could be interesting in its own right. The Finals version plays with 17 heroes—we removed Lich because his abilities were changed significantly in Dota version 7.20.


Arena

We're launching OpenAI Five Arena, a public experiment where we'll let anyone play OpenAI Five in both competitive and cooperative modes. We'd known that our 1v1 bot would be exploitable through clever strategies; we don't know to what extent the same is true of OpenAI Five, but we're excited to invite the community to help us find out!Arena opens Thursday, April 18 at 6pm PST and will close 11:59pm PST on Sunday, April 21. Please register so we can ensure there's enough server capacity in your region! Results of all games will be automatically reported to the Arena public leaderboard.We're incredibly grateful for all the support the Dota community has shown us over the past two years, and we hope that Arena will also serve as one small way of giving back. Have fun with it!


What's next

We will be releasing a more technical analysis of OpenAI Five once we've reviewed the outcomes of OpenAI Five Arena.Afterwards, we'll continue working with the Dota 2 environment within OpenAI. We've seen rapid progress in the past two years on RL capabilities, and we think that Dota 2 will continue to help us push forward what's possible—whether with achieving competent performance from less data or true human-AI cooperation.If you are interested in advancing AI capabilities and helping further our mission of ensuring they benefit humanity, we're hiring!


Transfer learning

The current version of OpenAI Five has been training continuously since June 2018, despite changes to the model size and the game rules (including some fairly large game patch updates and newly implemented features). In each case, we were able to transfer the model over and continue training—something that is an open challenge for RL in other domains. To the best of our knowledge, this is the first time an RL agent has been trained using such a long-lived training run.To make this work, we've continued to flesh out our surgery tooling so that we can start from trained parameters even across substantial architecture changes.


The task


Our approach


Results


Learning to see


Learning to control


Learning progress


What didn't pan out


Transfer performance


What surprised us


Top scores

The top 5 scoring teams are:Dharmaraja topped the scoreboard during the contest, and the lead remained on the final evaluation; mistake narrowly won out over aborg for second place. The top three teams will receive trophies.Learning curves of the top three teams for all 11 levels are as follows (showing the standard error computed from three runs).Averaging over all levels, we can see the following learning curves.Note that Dharmaraja and aborg start at similar scores, whereas mistake starts much lower. As we will describe in more detail below, these two teams fine-tuned (using PPO) from a pre-trained network, whereas mistake trained from scratch (using Rainbow DQN). mistake's learning curves end early because they timed out at 12 hours.


Dharmaraja

Dharmaraja is a six-member team including Qing Da, Jing-Cheng Shi, Anxiang Zeng, Guangda Huzhang, Run-Ze Li, and Yang Yu. Qing Da and Anxiang Zeng are from the AI team within the search department of Alibaba in Hangzhou, China. In recent years, they have studied how to apply reinforcement learning to real world problems, especially in an e-commerce setting, together with Yang Yu, who is an Associate Professor of the Department of Computer Science at Nanjing University, Nanjing, China.Dharmaraja's solution is a variant of joint PPO (described in our tech report) with a few improvements. First, it uses RGB images rather than grayscale; second, it uses a slightly augmented action space, with more common button combinations; third, it uses an augmented reward function, which rewards the agent for visiting new states (as judged by a perceptual hash of the screen). In addition to these modifications, the team also tried a number of things that didn't pan out: DeepMimic, object detection through YOLO, and some Sonic-specific ideas.Get the source code


Aborg

Team Aborg is a solo effort from Alexandre Borghi. After completing a PhD in computer science in 2011, Alexandre worked for different companies in France before moving to the United Kingdom where he is a research engineer in deep learning. As both a video game and machine learning enthusiast, he spends most of his free time studying deep reinforcement learning, which led him to take part in the OpenAI Retro Contest.Aborg's solution, like Dharmaraja's, is a variant of joint PPO with many improvements: more training levels from the Game Boy Advance and Master System Sonic games; a different network architecture; and fine-tuning hyper-parameters that were designed specifically for fast learning. Elaborating on the last point, Alexandre noticed that the first 150K timesteps of fine-tuning were unstable (i.e. the performance sometimes got worse), so he tuned the learning rate to fix this problem. In addition to the above changes, Alexandre tried several solutions that did not work: different optimizers, MobileNetV2, using color images, etc.Get the source code


Mistake

Team mistake consists of Peng Xu and Qiaoling Zhong. Both are second-year graduate students in Beijing, China, studying at the CAS Key Laboratory of Network Data Science and the Technology Institute of Computing Technology, Chinese Academy of Sciences. In their spare time, Peng Xu enjoys playing basketball, and Qiaoling Zhong likes to play badminton. Their favorite video games are Contra and Mario.Mistake's solution is based on the Rainbow baseline. They made several modifications that helped boost performance: a better value of n for n-step Q learning; an extra CNN layer added to the model, which made training slower but better; and a lower DQN target update interval. Additionally, the team tried joint training with Rainbow, but found that it actually hurt performance in their case.Get the source code


Best write-ups

The Best Write-up Prize is awarded to contestants that produced high-quality essays describing the approaches they tried.World ModelsExploration algorithms, policy distillation and fine-tuningFine-tuning on per-zone expert policiesNow, let's meet the winners of this prize category.


Dylan Djian

Dylan currently lives in Paris, France. He is a student in software development at school 42 in Paris. He got into machine learning after watching a video of a genetic algorithm learning how to play Mario a year and a half ago. This video sparked his interest and made him want to learn more about the field. His favorite video games are Zelda Twilight Princess and World of Warcraft.


Felix Yu

Felix is an entrepreneur who lives in Hong Kong. His first exposure to machine learning was a school project where he applied PCA to analyse stock data. After several years pursuing entrpreneurship, he got into ML in late 2015; he has become an active Kaggler and has worked on several side projects on computer vision and reinforcement learning.


Oleg Mürk

Oleg Mürk hails from the San Francisco Bay Area, but is originally from Tartu, Estonia. During the day, he works with distributed data processing systems as a Chief Architect at Planet OS. In his free time, he burns 'too much money' on renting GPUs for running deep learning experiments in TensorFlow. Oleg likes traveling, hiking, and kite-surfing and intends to finally learn to surf over the next 30 years. His favorite computer game (also the only one he has completed) is Wolfenstein 3D. His masterplan is to develop an automated programmer over the next 20 years and then retire.


Tristan Sokol

The winner of our Best Supporting Material award is Tristan Sokol, who wrote many helpful blog posts throughout the contest and made a tool for visualizing trajectories through Sonic levels.During the day, Tristan works for Square, helping to build their developer platform; at night, he is a designer and entrepreneur. This was the first time that he has done any AI/ML, and also his first time using Python for any real use case. Looking forward, Tristan is going to try to make cool things with TensorFlow.js. Whenever he isn't in front of a computer, Tristan is probably in his Oakland backyard watching plants grow.


Lessons and next steps

Contests have the potential to overhaul the prevailing consensus on what works the best, since contestants will try a diverse set of different approaches and the best one will win. In this particular contest, the top performing approaches were not radically different from the ones that we at OpenAI had found to be successful prior to the contest.We were glad to see several of the top solutions making use of transfer learning; fine-tuning from the training levels. However, we were surprised to find that some of the top submissions were simply tuned versions of our baseline algorithms. This emphasizes the importance of hyper-parameters, especially in RL algorithms such as Rainbow DQN.We plan to start another rendition of the contest in a few months. We hope and expect that some of the more off-beat approaches will be successful in this second round, now that people know what to expect and have begun to think deeply about the problems of fast learning and generalization in reinforcement learning. We'll see you then, and we look forward to watching your innovative solutions climb up the scoreboard.Gotta Learn Fast


Best Supporting Material

One of the best things that came from this contest was seeing contestants helping each other out. Lots of people contributed guides for getting started, useful scripts, and troubleshooting support for other contestants.


The problem

One AI milestone is to exceed human capabilities in a complex video game like StarCraft or Dota. Relative to previous AI milestones like Chess or Go, complex video games start to capture the messiness and continuous nature of the real world. The hope is that systems which solve complex video games will be highly general, with applications outside of games.Dota 2 is a real-time strategy game played between two teams of five players, with each player controlling a character called a 'hero'. A Dota-playing AI must master the following:Long time horizons. Dota games run at 30 frames per second for an average of 45 minutes, resulting in 80,000 ticks per game. Most actions (like ordering a hero to move to a location) have minor impact individually, but some individual actions like town portal usage can affect the game strategically; some strategies can play out over an entire game. OpenAI Five observes every fourth frame, yielding 20,000 moves. Chess usually ends before 40 moves, Go before 150 moves, with almost every move being strategic.Partially-observed state. Units and buildings can only see the area around them. The rest of the map is covered in a fog hiding enemies and their strategies. Strong play requires making inferences based on incomplete data, as well as modeling what one's opponent might be up to. Both chess and Go are full-information games.High-dimensional, continuous action space. In Dota, each hero can take dozens of actions, and many actions target either another unit or a position on the ground. We discretize the space into 170,000 possible actions per hero (not all valid each tick, such as using a spell on cooldown); not counting the continuous parts, there are an average of ~1,000 valid actions each tick. The average number of actions in chess is 35; in Go, 250.High-dimensional, continuous observation space. Dota is played on a large continuous map containing ten heroes, dozens of buildings, dozens of NPC units, and a long tail of game features such as runes, trees, and wards. Our model observes the state of a Dota game via Valve's Bot API as 20,000 (mostly floating-point) numbers representing all information a human is allowed to access. A chess board is naturally represented as about 70 enumeration values (a 8x8 board of 6 piece types and minor historical info); a Go board as about 400 enumeration values (a 19x19 board of 2 piece types plus Ko).The Dota rules are also very complex — the game has been actively developed for over a decade, with game logic implemented in hundreds of thousands of lines of code. This logic takes milliseconds per tick to execute, versus nanoseconds for Chess or Go engines. The game also gets an update about once every two weeks, constantly changing the environment semantics.


Our approach

Our system learns using a massively-scaled version of Proximal Policy Optimization. Both OpenAI Five and our earlier 1v1 bot learn entirely from self-play. They start with random parameters and do not use search or bootstrap from human replays.preemptibleRL researchers (including ourselves) have generally believed that long time horizons would require fundamentally new advances, such as hierarchical reinforcement learning. Our results suggest that we haven't been giving today's algorithms enough credit — at least when they're run at sufficient scale and with a reasonable way of exploring.Our agent is trained to maximize the exponentially decayed sum of future rewards, weighted by an exponential decay factor called γ. During the latest training run of OpenAI Five, we annealed γ from 0.998 (valuing future rewards with a half-life of 46 seconds) to 0.9997 (valuing future rewards with a half-life of five minutes). For comparison, the longest horizon in the PPO paper was a half-life of 0.5 seconds, the longest in the Rainbow paper was a half-life of 4.4 seconds, and the Observe and Look Further paper used a half-life of 46 seconds.While the current version of OpenAI Five is weak at last-hitting (observing our test matches, the professional Dota commentator Blitz estimated it around median for Dota players), its objective prioritization matches a common professional strategy. Gaining long-term rewards such as strategic map control often requires sacrificing short-term rewards such as gold gained from farming, since grouping up to attack towers takes time. This observation reinforces our belief that the system is truly optimizing over a long horizon.OpenAI Five: Dota Gamplay4:20


Exploration

Given a learning algorithm capable of handling long horizons, we still need to explore the environment. Even with our restrictions, there are hundreds of items, dozens of buildings, spells, and unit types, and a long tail of game mechanics to learn about—many of which yield powerful combinations. It's not easy to explore this combinatorially-vast space efficiently.OpenAI Five learns from self-play (starting from random weights), which provides a natural curriculum for exploring the environment. To avoid 'strategy collapse', the agent trains 80% of its games against itself and the other 20% against its past selves. In the first games, the heroes walk aimlessly around the map. After several hours of training, concepts such as laning, farming, or fighting over mid emerge. After several days, they consistently adopt basic human strategies: attempt to steal Bounty runes from their opponents, walk to their tier one towers to farm, and rotate heroes around the map to gain lane advantage. And with further training, they become proficient at high-level strategies like 5-hero push.In March 2017, our first agent defeated bots but got confused against humans. To force exploration in strategy space, during training (and only during training) we randomized the properties (health, speed, start level, etc.) of the units, and it began beating humans. Later on, when a test player was consistently beating our 1v1 bot, we increased our training randomizations and the test player started to lose. (Our robotics team concurrently applied similar randomization techniques to physical robots to transfer from simulation to the real world.)OpenAI Five uses the randomizations we wrote for our 1v1 bot. It also uses a new 'lane assignment' one. At the beginning of each training game, we randomly 'assign' each hero to some subset of lanes and penalize it for straying from those lanes until a randomly-chosen time in the game.Exploration is also helped by a good reward. Our reward consists mostly of metrics humans track to decide how they're doing in the game: net worth, kills, deaths, assists, last hits, and the like. We postprocess each agent's reward by subtracting the other team's average reward to prevent the agents from finding positive-sum situations.We hardcode item and skill builds (originally written for our scripted baseline), and choose which of the builds to use at random. Courier management is also imported from the scripted baseline.


Rapid

Our system is implemented as a general-purpose RL training system called Rapid, which can be applied to any Gym environment. We've used Rapid to solve other problems at OpenAI, including Competitive Self-Play.The training system is separated into rollout workers, which run a copy of the game and an agent gathering experience, and optimizer nodes, which perform synchronous gradient descent across a fleet of GPUs. The rollout workers sync their experience through Redis to the optimizers. Each experiment also contains workers evaluating the trained agent versus reference agents, as well as monitoring software such as TensorBoard, Sentry, and Grafana.During synchronous gradient descent, each GPU computes a gradient on its part of the batch, and then the gradients are globally averaged. We originally used MPI's allreduce for averaging, but now use our own NCCL2 wrappers that parallelize GPU computations and network data transfer.The latencies for synchronizing 58MB of data (size of OpenAI Five's parameters) across different numbers of GPUs are shown on the right. The latency is low enough to be largely masked by GPU computation which runs in parallel with it.We've implemented Kubernetes, Azure, and GCP backends for Rapid.