Regression using the embedding features

Embeddings present an elegant way of predicting a numerical value.  In this example we predict the reviewer's star rating, based on the text of their review.  Because the semantic information contained within embeddings is high, the prediction is decent even with very few reviews.We assume the score is a continuous variable between 1 and 5, and allow the algorithm to predict any floating point value.  The ML algorithm minimizes the distance of the predicted value to the true score, and achieves a mean absolute error of 0.39, which means that on average the prediction is off by less than half a star.1 2 3 4 5 from sklearn.ensemble import RandomForestRegressor rfr = RandomForestRegressor(n_estimators=100) rfr.fit(X_train, y_train) CollapseClassification_using_embeddings.ipynbThis time, instead of having the algorithm predict a value anywhere between 1 and 5, we will attempt to classify the exact number of stars for a review into 5 buckets, ranging from 1 to 5 stars.After the training, the model learns to predict 1 and 5-star reviews much better than the more nuanced reviews (2-4 stars), likely due to more extreme sentiment expression.1 2 3 4 5 6 from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import classification_report, accuracy_score clf = RandomForestClassifier(n_estimators=100) clf.fit(X_train, y_train) CollapseZero-shot_classification_with_embeddings.ipynbWe can use embeddings for zero shot classification without any labeled training data.  For each class, we embed the class name or a short description of the class.  To classify some new text in a zero-shot manner, we compare its embedding to all class embeddings and predict the class with the highest similarity.1 2 3 4 5 6 7 8 9 10 11 12 from openai.embeddings_utils import cosine_similarity, get_embedding df= df[df.Score!=3] df['sentiment'] = df.Score.replace({1:'negative', 2:'negative', 4:'positive', 5:'positive'}) labels = ['negative', 'positive'] label_embeddings = [get_embedding(label, model=model) for label in labels] def label_score(review_embedding, label_embeddings): return cosine_similarity(review_embedding, label_embeddings[1]) - cosine_similarity(review_embedding, label_embeddings[0]) prediction = 'positive' if label_score('Sample Review', label_embeddings) > 0 else 'negative'CollapseUser_and_product_embeddings.ipynbWe can obtain a user embedding by averaging over all of their reviews.  Similarly, we can obtain a product embedding by averaging over all the reviews about that product.  In order to showcase the usefulness of this approach we use a subset of 50k reviews to cover more reviews per user and per product.We evaluate the usefulness of these embeddings on a separate test set, where we plot similarity of the user and product embedding as a function of the rating.  Interestingly, based on this approach, even before the user receives the product we can predict better than random whether they would like the product.user_embeddings = df.groupby('UserId').ada_embedding.apply(np.mean) prod_embeddings = df.groupby('ProductId').ada_embedding.apply(np.mean)CollapseClustering.ipynbClustering is one way of making sense of a large volume of textual data.  Embeddings are useful for this task, as they provide semantically meaningful vector representations of each text


Regression using the embedding features

Thus, in an unsupervised way, clustering will uncover hidden groupings in our dataset.In this example, we discover four distinct clusters: one focusing on dog food, one on negative reviews, and two on positive reviews.1 2 3 4 5 6 7 8 9 import numpy as np from sklearn.cluster import KMeans matrix = np.vstack(df.ada_embedding.values) n_clusters = 4 kmeans = KMeans(n_clusters = n_clusters, init='k-means++', random_state=42) kmeans.fit(matrix) df['Cluster'] = kmeans.labels_CollapseSemantic_text_search_using_embeddings.ipynbTo retrieve the most relevant documents we use the cosine similarity between the embedding vectors of the query and each document, and return the highest scored documents.1 2 3 4 5 6 7 8 9 from openai.embeddings_utils import get_embedding, cosine_similarity def search_reviews(df, product_description, n=3, pprint=True): embedding = get_embedding(product_description, model='text-embedding-ada-002') df['similarities'] = df.ada_embedding.apply(lambda x: cosine_similarity(x, embedding)) res = df.sort_values('similarities', ascending=False).head(n) return res res = search_reviews(df, 'delicious beans', n=3)CollapseCode_search.ipynbCode search works similarly to embedding-based text search .  We provide a method to extract Python functions from all the Python files in a given repository.  Each function is then indexed by the text-embedding-ada-002 model.To perform a code search, we embed the query in natural language using the same model.  Then we calculate cosine similarity between the resulting query embedding and each of the function embeddings.  The highest cosine similarity results are most relevant.1 2 3 4 5 6 7 8 9 10 11 from openai.embeddings_utils import get_embedding, cosine_similarity df['code_embedding'] = df['code'].apply(lambda x: get_embedding(x, model='text-embedding-ada-002')) def search_functions(df, code_query, n=3, pprint=True, n_lines=7): embedding = get_embedding(code_query, model='text-embedding-ada-002') df['similarities'] = df.code_embedding.apply(lambda x: cosine_similarity(x, embedding)) res = df.sort_values('similarities', ascending=False).head(n) return res res = search_functions(df, 'Completions API tests', n=3)CollapseRecommendation_using_embeddings.ipynbBecause shorter distances between embedding vectors represent greater similarity, embeddings can be useful for recommendation.Below, we illustrate a basic recommender.  It takes in a list of strings and one 'source' string, computes their embeddings, and then returns a ranking of the strings, ranked from most similar to least similar.  As a concrete example, the linked notebook below applies a version of this function to the AG news dataset (sampled down to 2,000 news article descriptions) to return the top 5 most similar articles to any given source article.1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 def recommendations_from_strings( strings: List[str], index_of_source_string: int, model='text-embedding-ada-002', ) -> List[int]: '''Return nearest neighbors of a given string.''' # get embeddings for all strings embeddings = [embedding_from_string(string, model=model) for string in strings] # get the embedding of the source string query_embedding = embeddings[index_of_source_string] # get distances between the source embedding and other embeddings (function from embeddings_utils.py) distances = distances_from_embeddings(query_embedding, embeddings, distance_metric='cosine') # get indices of nearest neighbors (function from embeddings_utils.py) indices_of_nearest_neighbors = indices_of_nearest_neighbors_from_distances(distances) return indices_of_nearest_neighborsCollapse


Code search embeddings

Similarly to search embeddings, there are two types: one for embedding natural language search queries and one for embedding code snippets to be retrieved.With the -001 text embeddings (not -002, and not code embeddings), we suggest replacing newlines (\n) in your input with a single space, as we have seen worse results when newlines are present.Collapse


Social bias

Limitation: The models encode social biases, e.g. via stereotypes or negative sentiment towards certain groups.We found evidence of bias in our models via running the SEAT (May et al, 2019) and the Winogender (Rudinger et al, 2018) benchmarks. Together, these benchmarks consist of 7 tests that measure whether models contain implicit biases when applied to gendered names, regional names, and some stereotypes.For example, we found that our models more strongly associate (a) European American names with positive sentiment, when compared to African American names, and (b) negative stereotypes with black women.These benchmarks are limited in several ways: (a) they may not generalize to your particular use case, and (b) they only test for a very small slice of possible social bias.These tests are preliminary, and we recommend running tests for your specific use cases. These results should be taken as evidence of the existence of the phenomenon, not a definitive characterization of it for your use case. Please see our usage policies for more details and guidance.Please contact our support team via chat if you have any questions; we are happy to advise on this.


Limitations & risks

Our embedding models may be unreliable or pose social risks in certain cases, and may cause harm in the absence of mitigations.


How can I tell how many tokens a string has before I embed it?

In Python, you can split a string into tokens with OpenAI's tokenizer tiktoken.Example code:1 2 3 4 5 6 7 8 9 import tiktoken def num_tokens_from_string(string: str, encoding_name: str) -> int: '''Returns the number of tokens in a text string.''' encoding = tiktoken.get_encoding(encoding_name) num_tokens = len(encoding.encode(string)) return num_tokens num_tokens_from_string('tiktoken is great!', 'cl100k_base')For second-generation embedding models like text-embedding-ada-002, use the cl100k_base encoding.More details and example code are in the OpenAI Cookbook guide how to count tokens with tiktoken.


Similarity embeddings

Similarity models are best at capturing semantic similarity between pieces of text.


Blindness to recent events

Limitation: Models lack knowledge of events that occurred after August 2020.Our models are trained on datasets that contain some information about real world events up until 8/2020. If you rely on the models representing recent events, then they may not perform well.


How can I retrieve K nearest embedding vectors quickly?

For searching over many vectors quickly, we recommend using a vector database. You can find examples of working with vector databases and the OpenAI API in our Cookbook on GitHub.Vector database options include:Chroma, an open-source embeddings storeElasticsearch, a popular search/analytics engine and vector databaseMilvus, a vector database built for scalable similarity searchPinecone, a fully managed vector databaseQdrant, a vector search engineRedis as a vector databaseTypesense, fast open source vector searchWeaviate, an open-source vector search engineZilliz, data infrastructure, powered by Milvus


Can I share my embeddings online?

Customers own their input and output from our models, including in the case of embeddings. You are responsible for ensuring that the content you input to our API does not violate any applicable law or our Terms of Use.


Which distance function should I use?

We recommend cosine similarity. The choice of distance function typically doesn't matter much.OpenAI embeddings are normalized to length 1, which means that:Cosine similarity can be computed slightly faster using just a dot productCosine similarity and Euclidean distance will result in the identical rankings


Text similarity models

Text similarity models provide embeddings that capture the semantic similarity of pieces of text. These models are useful for many tasks including clustering, data visualization, and classification.The following interactive visualization shows embeddings of text samples from the DBpedia dataset: Drag to pan, scroll or pinch to zoom animalathletefilmtransportationvillage Embeddings from the text-similarity-babbage-001 model, applied to the DBpedia dataset. We randomly selected 100 samples from the dataset covering 5 categories, and computed the embeddings via the /embeddings endpoint. The different categories show up as 5 clear clusters in the embedding space. To visualize the embedding space, we reduced the embedding dimensionality from 2048 to 3 using PCA. The code for how to visualize embedding space in 3D dimension is available here. To compare the similarity of two pieces of text, you simply use the dot product on the text embeddings. The result is a 'similarity score', sometimes called 'cosine similarity,' between –1 and 1, where a higher number means more similarity. In most applications, the embeddings can be pre-computed, and then the dot product comparison is extremely fast to carry out.One popular use of embeddings is to use them as features in machine learning tasks, such as classification. In machine learning literature, when using a linear classifier, this classification task is called a 'linear probe.' Our text similarity models achieve new state-of-the-art results on linear probe classification in SentEval (Conneau et al., 2018), a commonly used benchmark for evaluating embedding quality.Linear probe classification over 7 datasetsPrevious SOTA (Gao et al. 2021)90.2%text-similarity-davinci-00192.2% Show more


Linear probe classification over 7 datasets

Gao et al. 2021


Text search models

Text search models provide embeddings that enable large-scale search tasks, like finding a relevant document among a collection of documents given a text query. Embedding for the documents and query are produced separately, and then cosine similarity is used to compare the similarity between the query and each document.Embedding-based search can generalize better than word overlap techniques used in classical keyword search, because it captures the semantic meaning of text and is less sensitive to exact phrases or words. We evaluate the text search model's performance on the BEIR (Thakur, et al. 2021) search evaluation suite and obtain better search performance than previous methods. Our text search guide provides more details on using embeddings for search tasks.Average accuracy over 11 search tasks in BEIRPrevious SOTA (Izacard, et al. 2021)50.2%text-search-davinci-{doc, query}-00152.8% Show more


Code search models

Code search models provide code and text embeddings for code search tasks. Given a collection of code blocks, the task is to find the relevant code block for a natural language query. We evaluate the code search models on the CodeSearchNet (Husain et al., 2019) evaluation suite where our embeddings achieve significantly better results than prior methods. Check out the code search guide to use embeddings for code search.Average accuracy over 6 programming languagesPrevious SOTA (Guo, et al. 2021)77.4%code-search-babbage-{doc, query}-00193.5% Show more


Average accuracy over 11 search tasks in BEIR

Izacard, et al. 2021


Average accuracy over 6 programming languages

Guo, et al. 2021


JetBrains Research

JetBrains Research's Astroparticle Physics Lab analyzes data like The Astronomer's Telegram and NASA's GCN Circulars, which are reports that contain astronomical events that can't be parsed by traditional algorithms.Powered by OpenAI's embeddings of these astronomical reports, researchers are now able to search for events like 'crab pulsar bursts' across multiple databases and publications. Embeddings also achieved 99.85% accuracy on data source classification through k-means clustering.


Fabius

Fabius helps companies turn customer conversations into structured insights that inform planning and prioritization. OpenAI's embeddings allow companies to more easily find and tag customer call transcripts with feature requests.For instance, customers might use words like 'automated' or 'easy to use' to ask for a better self-service platform. Previously, Fabius was using fuzzy keyword search to attempt to tag those transcripts with the self-service platform label. With OpenAI's embeddings, they're now able to find 2x more examples in general, and 6x–10x more examples for features with abstract use cases that don't have a clear keyword customers might use.All API customers can get started with the embeddings documentation for using embeddings in their applications.Read documentation


FineTune Learning

FineTune Learning is a company building hybrid human-AI solutions for learning, like adaptive learning loops that help students reach academic standards.OpenAI's embeddings significantly improved the task of finding textbook content based on learning objectives. Achieving a top-5 accuracy of 89.1%, OpenAI's text-search-curie embeddings model outperformed previous approaches like Sentence-BERT (64.5%). While human experts are still better, the FineTune team is now able to label entire textbooks in a matter of seconds, in contrast to the hours that it took the experts.


Text similarity models


Gao et al. 2021


Izacard, et al. 2021


Text search models


Guo, et al. 2021


JetBrains Research


Fabius


Code search models


FineTune Learning


Model improvements

Stronger performance. text-embedding-ada-002 outperforms all the old embedding models on text search, code search, and sentence similarity tasks and gets comparable performance on text classification. For each task category, we evaluate the models on the datasets used in old embeddings.Text searchCode searchSentence similarityText classificationUnification of capabilities. We have significantly simplified the interface of the /embeddings endpoint by merging the five separate models shown above (text-similarity, text-search-query, text-search-doc, code-search-text and code-search-code) into a single new model. This single representation performs better than our previous embedding models across a diverse set of text search, sentence similarity, and code search benchmarks.Longer context. The context length of the new model is increased by a factor of four, from 2048 to 8192, making it more convenient to work with long documents.Smaller embedding size. The new embeddings have only 1536 dimensions, one-eighth the size of davinci-001 embeddings, making the new embeddings more cost effective in working with vector databases.Reduced price. We have reduced the price of new embedding models by 90% compared to old models of the same size. The new model achieves better or similar performance as the old Davinci models at a 99.8% lower price.Overall, the new embedding model is a much more powerful tool for natural language processing and code tasks. We are excited to see how our customers will use it to create even more capable applications in their respective fields.


Limitations

The new text-embedding-ada-002 model is not outperforming text-similarity-davinci-001 on the SentEval linear probing classification benchmark. For tasks that require training a light-weighted linear layer on top of embedding vectors for classification prediction, we suggest comparing the new model to text-similarity-davinci-001 and choosing whichever model gives optimal performance.Check the Limitations & Risks section in the embeddings documentation for general limitations of our embedding models.


Examples of the embeddings API in action

Kalendar AI is a sales outreach product that uses embeddings to match the right sales pitch to the right customers out of a dataset containing 340M profiles. This automation relies on similarity between embeddings of customer profiles and sale pitches to rank up most suitable matches, eliminating 40–56% of unwanted targeting compared to their old approach.Notion, the online workspace company, will use OpenAI's new embeddings to improve Notion search beyond today's keyword matching systems.Read documentation


Apply

Responses will be accepted on a rolling basis through December 31, 2023. We will offer $25,000 each in API credits to up to 10 top submissions, publish novel ideas and entries, and look for candidates for Preparedness from among the top contenders in this challenge. First name *Last name *Email *LinkedIn / resume URL * Imagine we gave you unrestricted access to OpenAI's Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model. You might consider misuse related to the categories discussed in the blog post, or another category. For example, a malicious actor might use GPT-4, Whisper and Voice to socially engineer workers at critical infrastructure facilities into installing malware, allowing shutdown of the power grid. What is the misuse you'll be writing about? *Maximum 20 wordsDescribe this misuse. Why might it lead to catastrophic harm and what would the harm be? *Maximum 100 wordsOutline how you envision someone executing such a misuse in the real world assuming you have unrestricted access to our models. Please be specific (e.g., provide proof of concept, detailed step by step instructions). *Link directly to a 3 page max PDF. Links to download a PDF will not be accepted. Now, imagine you have joined the Preparedness team. Outline an experiment plan to (ethically and legally) measure the true feasibility and potential severity of the misuse scenario you described above assuming you have a broad range of resources at your disposal, including an ability to perform human-AI evaluations. *Maximum 300 wordsDetail potential actions that might mitigate the risk you identified. *Maximum 150 wordsSubmit


Our approach

Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence.To align the first automated alignment researcher, we will need to 1) develop a scalable training method, 2) validate the resulting model, and 3) stress test our entire alignment pipeline:To provide a training signal on tasks that are difficult for humans to evaluate, we can leverage AI systems to assist evaluation of other AI systems (scalable oversight). In addition, we want to understand and control how our models generalize our oversight to tasks we can't supervise (generalization).To validate the alignment of our systems, we automate search for problematic behavior (robustness) and problematic internals (automated interpretability).Finally, we can test our entire pipeline by deliberately training misaligned models, and confirming that our techniques detect the worst kinds of misalignments (adversarial testing).We expect our research priorities will evolve substantially as we learn more about the problem and we'll likely add entirely new research areas. We are planning to share more on our roadmap in the future.


The new team

We are assembling a team of top machine learning researchers and engineers to work on this problem. We are dedicating 20% of the compute we've secured to date over the next four years to solving the problem of superintelligence alignment. Our chief basic research bet is our new Superalignment team, but getting this right is critical to achieve our mission and we expect many teams to contribute, from developing new methods to scaling them up to deployment.Our goal is to solve the core technical challenges of superintelligence alignment in four years.While this is an incredibly ambitious goal and we're not guaranteed to succeed, we are optimistic that a focused, concerted effort can solve this problem:C[C]Solving the problem includes providing evidence and arguments that convince the machine learning and safety community that it has been solved. If we fail to have a very high level of confidence in our solutions, we hope our findings let us and the community plan appropriately.Ilya Sutskever (cofounder and Chief Scientist of OpenAI) has made this his core research focus, and will be co-leading the team with Jan Leike (Head of Alignment). Joining the team are researchers and engineers from our previous alignment team, as well as researchers from other teams across the company.We're also looking for outstanding new researchers and engineers to join this effort. Superintelligence alignment is fundamentally a machine learning problem, and we think great machine learning experts—even if they're not already working on alignment—will be critical to solving it.We plan to share the fruits of this effort broadly and view contributing to alignment and safety of non-OpenAI models as an important part of our work.This new team's work is in addition to existing work at OpenAI aimed at improving the safety of current models like ChatGPT, as well as understanding and mitigating other risks from AI such as misuse, economic disruption, disinformation, bias and discrimination, addiction and overreliance, and others. While this new team will focus on the machine learning challenges of aligning superintelligent AI systems with human intent, there are related sociotechnical problems on which we are actively engaging with interdisciplinary experts to make sure our technical solutions consider broader human and societal concerns.


Join us

Superintelligence alignment is one of the most important unsolved technical problems of our time. We need the world's best minds to solve this problem.If you've been successful in machine learning, but you haven't worked on alignment before, this is your time to make the switch! We believe this is a tractable machine learning problem, and you could make enormous contributions.If you're interested, we'd love to hear from you! Please apply for our research engineer, research scientist, and research manager positions.View careers


Research contributions

Jamie Kiros Deployment research & evals leadDaniel Levy Optimization leadHyeonwoo Noh Pretraining research leadLong Ouyang Alignment data leadRaul Puri Research engineering leadArchitecture researchMark Chen, Casey Chu, Jamie Kiros, Christine McLeavey, Hyeonwoo Noh, Raul Puri, Alec Radford, Aditya RameshDistributed training infrastructureTrevor Cai, Yunxing Dai, Chris Hesse, Brandon Houghton, Yongjik Kim, Łukasz Kondraciuk, Hyeonwoo Noh, Mikhail Pavlov, Raul Puri, Nikolas Tezak, Amin Tootoonchian, Tianhao ZhengDataAlex Karpenko, Jong Wook Kim, David Mélý, Reiichiro Nakano, Hyeonwoo Noh, Long Ouyang, Raul Puri, Alec Radford, Pranav Shyam, Tao XuEvaluation dataSandhini Agarwal, Madeline Boyd, Shengli Hu, Andrew Kondrich, Todor Markov, David Mélý, Hyeonwoo Noh, Reiichiro Nakano, Long Ouyang, Cameron Raymond, Filippo Rasso, Chelsea Voss, Lilian Weng, Chong Zhang, Rowan Zellers, Nicholas TurleyAlignment dataStephanie Lin, Long Ouyang, Chong ZhangDeployment, alignment & post-training researchIlge Akkaya, Diogo Moitinho de Almeida, Mark Chen, Liam Fedus, Yuchen He, Alex Karpenko, Jamie Kiros, Andrew Kondrich, Rachel Lim, Randall Lin, Stephanie Lin, Ryan Lowe, Luke Metz, Reiichiro Nakano, Long Ouyang, Raul Puri, Jiayi Weng, Barret ZophCompute cluster scalingAndrew Cann, Rory Carmichael, Christian Gibson, Henri Roussez, Akila WeliwindaHardware correctnessOleg Boiko, Trevor Cai, Michael Petrov, Alethea PowerTraining run babysittingTrevor Cai, Kyle Kosic, Daniel Levy, David Mélý, Reiichiro Nakano, Hyeonwoo Noh, Mikhail Pavlov, Raul Puri, Amin Tootoonchian


Authorship, credit attribution, and acknowledgments

When citing GPT-4V please cite this work as 'OpenAI (2023)'. Contributions are sorted alphabetically and assembled by Raul Puri.


Safety contributions

Sandhini Agarwal Policy research leadLama Ahmad Red teaming leadChong Zhang Safety systems research leadRed teaming leadersLama Ahmad, Rosie Campbell, Ashyana-Jasmine KachraSafety systems researchFlorencia Leoni Aleman, Madelaine Boyd, Yuchen He, Andrew Kondrich, Todor Markov, Raul Puri, Cameron Raymond, Andrea Vallone, CJ Weinmann, Lilian Weng, Mehmet Yatbaz, Chong ZhangPolicy researchSandhini Agarwal, Lama Ahmad, Miles Brundage, Rosie Campbell, Michael Kolhede, Michael Lampe


Additional contributions

Greg Brockman, Peter Deng, Jason Kwon, Bob McGrew, Mira Murati, Srinivas Narayanan, Peter Welinder, Hannah WongCommunicationsEric Antonow, Ryan Biddy, Ruby Chen, Thomas Degry, Niko Felix, Elie Georges, Kendra Rimbach, Natalie Summers, Justin Jay WangDeployment securityTiffany Citra, Jake McNeil, Karthik RangarajanUser SupportJeremiah CurrierLegalAshley Pantuliano, Filippo Raso, Thomas Stasi


Deployment contributions

Madeline Boyd Trust & safety engineering leadRaul Puri Inference infrastructure leadJordan Sitkin Deployment platform leadIsaac Wolkerstorfer ChatGPT engineering leadBenjamin Zweig Design leadDeployment engineeringValerie Balcom, Jason Chen, Dave Cummings, Bogo Giertler, Joshua Gross, Eric Horacek, Mark Hudnall, Tomer Kaftan, Rachel Lim, Lien Mamitsuka, Rajeev Nayak, Henrique Ponde de Oliveira Pinto, Adam Perelman, Raul Puri, David Schnurr, Eric Sigler, Jordan Sitkin, Javier Soto, Heather Schmidt, Felipe Such, Anton Tananaev, Sherwin Wu, Isaac WolkerstorferChatGPT client engineeringValerie Balcom, Bogo Giertler, Eric Horacek, Lien Mamitsuka, Rajeev Nayak, Raul Puri, David Schnurr, Javier Soto, Anton TananaevChatGPT backend engineeringJason Chen, Joshua Gross, Mark Hudnall, Alex Karpenko, Raul Puri, Eric Sigler, Jordan Sitkin, Isaac Wolkerstorfer, Chong Zhang, Dave CummingsDeployment platformMadeleine Boyd, Olivier Godement, Mark Hudnall, Rachel Lim, Raul Puri, Jordan Sitkin, Isaac Wolkerstorfer, Sherwin WuInference infrastructureGreg Brockman, Tomer Kaftan, Rachel Lim, Raul Puri, Heather Schmidt, Jordan Sitkin, Felipe SuchTrust & safety engineeringMadeleine BoydDesignMaddie Simens, Benjamin ZweigLaunch partners, product, and deployment managementOlivier Godement, Joanne Jang, Angela Jiang, Raul Puri, Jessica Shieh, Natalie Staudacher, Nicholas Turley


Acknowledgments

We are grateful to our expert adversarial testers and red teamers who helped test our models at early stages of development and informed our risk assessments as well as the System Card output. Participation in this red teaming process is not an endorsement of the deployment plans of OpenAI or OpenAI's policies: Sally Applin, Gerardo Adesso, Rubaid Ashfaq, Max Bai, Matthew Brammer, Ethan Fecht, Andrew Goodman, Shelby Grossman, Matthew Groh, Hannah Rose Kirk, Seva Gunitsky, Yixing Huang, Lauren Kahn, Sangeet Kumar, Dani Madrid-Morales, Fabio Motoki, Aviv Ovadya, Uwe Peters, Maureen Robinson, Paul Röttger, Herman Wasserman, Alexa Wehsener, Leah Walker, Bertram Vidgen, Jianlong Zhu.We thank Microsoft for their partnership, especially Microsoft Azure for supporting model training with infrastructure design and management, and the Microsoft Bing team and Microsoft's safety teams for their partnership on safe deployment and safety research. We also thank the Microsoft Research team for their exploratory work cataloguing use of GPT-4V: Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, Lijuan Wang. Lastly, we thank our deployment partners Be My Eyes for their support and feedback in deploying this technology to the blind and low-vision community.


Challenges

Our algorithm's performance is only as good as the human evaluator's intuition about what behaviors look correct, so if the human doesn't have a good grasp of the task they may not offer as much helpful feedback. Relatedly, in some domains our system can result in agents adopting policies that trick the evaluators. For example, a robot which was supposed to grasp items instead positioned its manipulator in between the camera and the object so that it only appeared to be grasping it, as shown below.We addressed this particular problem by adding in visual cues (the thick white lines in the above animation) to make it easy for the human evaluators to estimate depth.The research described in this post was done in collaboration with Jan Leike, Miljan Martic, and Shane Legg at DeepMind. Our two organizations plan to continue to collaborate on topics that touch on long-term AI safety. We think that techniques like this are a step towards safe AI systems capable of learning human-centric goals, and can complement and extend existing approaches like reinforcement and imitation learning. This post is representative of the work done by OpenAI's safety team; if you're interested in working on problems like this, please join us!


Footnote

By comparison, we took two hours to write our own reward function (the animation in the above right) to get a robot to backflip, and though it succeeds it's a lot less elegant than the one trained simply through human feedback (top left). We think there are many cases where human feedback could let us specify a specific goal more intuitively and quickly than is possible by manually hand-crafting the objective.You can replicate this backflip in gym with the following reward function for Hopper:


Challenges


Footnote


Stylistic text continuation

For the stylistic continuation tasks, samples comparing the raw 774M GPT-2 model and our fine-tuned versions are shown below.B[B]Each fine-tuned model is trained using 5,000 four-way comparisons by humans.SentimentDescriptiveness Given some text, generate a natural continuation of the text with positive sentiment: According to the same human labelers used to train them, our fine-tuned models are preferred to the base GPT-2 model (zero-shot) 88% and 86% of the time for sentiment and descriptiveness, respectively.


Summarization

Labelers want to work as quickly as possible, and they can work very quickly by following the heuristic of 'if the summary copies, then select it.'


Summarization

We also applied human fine-tuning to two summarization tasks: summarization of articles from the CNN/Daily Mail dataset, and summarization of Reddit snippets from the TL;DR dataset.These tasks are harder: our main models use 60,000 four-way comparisons.  We also need online data collection, where the samples shown to humans are collected throughout training as the policy changes; an offline data collection strategy which shows humans only samples from the base GPT-2 language model performed poorly.Our models achieve very good performance according to human labelers, but are likely exploiting the fact that labelers rely on simple heuristics: they prefer the lead-3 baseline of copying the first three sentences to our models.  However, when combining supervised fine-tuning with human fine-tuning, our models outperform lead-3 on ROUGE scores.Samples from zero-shot and supervised baselines, as well as RL fine-tuning of each, are shown below.CNN/Daily Mailtl;drThe reader may have noticed a few things about these samples.  First, our RL fine-tuned model is mostly a smart copying engine: it typically summarizes content by copying entire sentences from the article or Reddit snippet.  By contrast, the zero-shot and supervised fine-tuned samples are more novel:Sentence novelty: Percentage of sentences in summaries that do not appear in source text.The RL fine-tuned model does vary where it copies from: while they copy the start of the input 28.3% and 77.6% of the time on TL;DR and CNN/Daily Mail, these numbers fall to 0.2% and 1.4% if the input starts with uninformative preamble (defined as 'hi', 'hello', 'hey', 'ok', 'okay', 'so' for TL;DR, or a colon in the first three words for CNN/Daily Mail such as 'Winner: Simon Wood took home the TV crown [...]').The visualization below shows where the variation in where the summarization models copy from, illustrated by the longest common subsequence of bigrams between context and summary for randomly chosen contexts.Second, while summaries from GPT-2 zero-shot and the supervised fine-tuned version of GPT-2 are more novel as measured by n-grams or sentences, they are also more novel in terms of content.  That is, they're not true:Summary accuracy: Accuracy frequency of generated summaries, judged by authors on 30 articles from each dataset.There are at least two ways of interpreting these results.  The first is that copying is the easiest way to be accurate.  The labelers were told to penalize inaccuracy but not copying.  The zero-shot model copies some of the time, and when it copied it was accurate, so copying was reinforced.  The result is a model that mostly copies, but at least does not lie.However, this does not fully explain the results of human evaluation: both our model and a simple lead-3 baseline which copies the first three sentences are strongly preferred by the labelers to the human reference summaries in both datasets.  The authors do not agree: we find the reference summaries are accurate and better capture the overall message.  This reveals a mismatch between the notion of quality we wanted our model to learn, and what the humans labelers actually evaluated


Ambiguous tasks make labeling hard

A single human may have a clear notion of whether a given sample is separately accurate, grammatical, nonredundant, or hits the key points, but comparing two summaries often requires subjective weighing of different kinds of deficiencies. When possible, it seems better to design less ambiguous labeling tasks that get at the same information. For example, rather than asking a person to compare summaries, we could ask for a verbal description of the problems with a summary, or a suggested correction. Even if two people disagree on the most important problem, they may agree that the other picked some problem, and more agreement eases data quality control and the overall experimental process.


Online data collection is hard

Online data collection was necessary to achieve the best results on summarization, but led to multiple difficulties:Software complexity. Interleaving data gathering, reward model training, and RL fine-tuning led to a far more complex system than if each component was separate.Machine learning complexity. An ML bug in any component would break the whole system, and it was awkward to debug one component in isolation.Quality control issues. Online label collection required low latency between generating a sample and receiving data back from Scale (typically ~30 minutes). Quality control with low latency is hard, and regressions in data quality were often not detected until after training runs were complete.We believe the right middle ground between offline and online data collection is batched data collection: we would alternate between collecting large batches of data (with higher latency) and training on collected data. The cost of human data means that volume will always be low, so it is easy to retrain from scratch (or rather, from the GPT-2 starting point) each time.


Looking forward

We've demonstrated reward learning from human preferences on two kinds of natural language tasks, stylistic continuation and summarization. Our results are mixed: for continuation we achieve good results with very few samples, but our summarization models are only 'smart copiers': they copy from the input text but skip over irrelevant preamble. The advantage of smart copying is truthfulness: the zero-shot and supervised models produce natural, plausible-looking summaries that are often lies. We believe the limiting factor in our experiments is data quality exacerbated by the online data collection setting, and plan to use batched data collection in the future.We believe the application of reward learning to language is important both from a capability and safety perspective. On the capability side, reinforcement learning lets us correct mistakes that supervised learning would not catch, but RL with programmatic reward functions 'can be detrimental to model quality.' On the safety side, reward learning for language allows important criteria like 'don't lie' to be represented during training, and is a step towards scalable safety methods such as a debate and amplification.


Bugs can optimize for bad behavior

One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while preserving natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form. This bug was remarkable since the result was not gibberish but maximally bad output. The authors were asleep during the training process, so the problem was noticed only once training had finished. A mechanism such as Toyota's Andon cord could have prevented this, by allowing any labeler to stop a problematic training process.


Results

PrevNextWe evaluated several different summarization models—some pre-trained on a broad distribution of text from the internet, some fine-tuned via supervised learning to predict TL;DRs, and some fine-tuned using human feedback.C[C]We generate all of our samples at temperature 0, which we found humans preferred most.Figure 1We found that RL fine-tuning with human feedback had a very large effect on quality compared to both supervised fine-tuning and scaling up model size. In particular, our 1.3 billion parameter (1.3B) model trained with human feedback outperforms our 12B model trained only with supervised learning. Summaries from both our 1.3B and 6.7B human feedback models are preferred by our labelers to the original human-written TL;DRs in the dataset.D[D]While we use human-written TL;DRs as our main point of comparison, they don't always represent optimal human performance; they are sometimes intended to be funny or to summarize only a part of the post, and their grammar and style are all over the map.People make different trade-offs when writing summaries, including between conciseness and coverage of the original text; depending on the purpose of the summary, different summary lengths might be preferred. Our labelers tended to prefer longer summaries, so our models adapted to that preference and converged to the longest allowable length. Controlling for length reduced human preferences for our 6.7B model's summaries from 70% to 65%, explaining a minority of our gains.E[E]We control by training a logistic regression model to predict the preferred summary given only the policy ID and the log ratio of the lengths of the summaries. Then, we report the regression coefficients on each policy ID, corresponding to a length ratio of 1 with the reference summaries.


Human feedback models outperform much larger supervised models and reference summaries on TL;DR

Figure 1: The performance of various training procedures for different model sizes. Model performance is measured by how often summaries from that model are preferred to the human-written reference summaries. Our pre-trained models are early versions of GPT-3, our supervised baselines were fine-tuned to predict 117K human-written TL;DRs, and our human feedback models are additionally fine-tuned on a dataset of about 65K summary comparisons.


Human feedback models trained on Reddit transfer to generate excellent summaries of CNN/DM news articles without further training

Raw scoresLength-controlledThe performance (human-rated summary quality on a 1–7 scale) of various training procedures and model sizes. Note that our human feedback models generate summaries that are significantly shorter than summaries from models trained on CNN/DM.At a given summary length, our 6.7B human feedback model trained on Reddit performs almost as well as a fine-tuned 11B T5 model, despite not being re-trained on CNN/DM.PrevNext


Approach

A diagram of our method, which is similar to the one used in our previous work.Our core method consists of four steps: training an initial summarization model, assembling a dataset of human comparisons between summaries, training a reward model to predict the human-preferred summary, and then fine-tuning our summarization models with RL to get a high reward.We trained several supervised baselines by starting from GPT-style transformer models trained on text from the Internet,20 and fine-tuning them to predict the human-written TL;DR via supervised learning. We mainly use models with 1.3 and 6.7 billion parameters. As a sanity check, we confirmed that this training procedure led to competitive resultsG[G]In terms of ROUGE results on CNN/DM, our 6.7B supervised models are a bit worse than T5 20, but a bit better than state-of-the-art models from mid-201921.We then collected a dataset of human quality judgments. For each judgment, a human compares two summaries of a given post and picks the one they think is better.H[H]Our main models are trained on about 65K comparisons, though we achieve good results with as few as 8K comparisons.Finally, we optimize the policy against the reward model using RL. We use PPO with 1 million episodes in total, where each episode consists of the policy summarizing a single article and then receiving a reward r. We include a KL penalty that incentivizes the policy to remain close to the supervised initialization.


Results

To test our models' generalization, we also applied them directly to the popular CNN/DM news dataset.19 These articles are more than twice as long as Reddit posts and are written in a very different style. Our models have seen news articles during pre-training, but all of our human data and RL fine-tuning was on the Reddit TL;DR dataset.This time we evaluated our models by asking our labelers to rate them on a scale from 1–7.F[F]We took this approach because it is hard to directly compare our TL;DR-trained models to models trained on CNN/DM; the CNN/DM summaries are much longer and written in bullet-point form.


Collecting data from humans

Any training procedure that uses human feedback is directly influenced by the actual humans labeling the data. In our previous work on fine-tuning language models from human preferences,7 our labelers often gave high ratings to summaries we thought were average, which was reflected in the quality of our trained models.In response, in this project we invested heavily in ensuring high data quality. We hired about 80 contractors using third-party vendor sites,I[I]Specifically, we use Upwork, Scale, and Lionbridge. Our contractors have a range of ages, genders, and educational backgrounds, and are mostly American or Filipino (see Appendix C of our paper for demographic data).J[J]Our criteria for hiring contractors were: (1) they were willing to do the task, and (2) they passed a minimum threshold of speed and agreement with researcher labels. We paid all our contractors at least $15/hr.


Optimizing the reward model

Optimizing our reward model eventually leads to sample quality degradationStarting from the 1.3B supervised baseline (point 0 on the x-axis), we use RL to optimize the policy against the reward model, which results in policies with different 'distances' from the baseline (x-axis, measured using the KL divergence from the supervised baseline). Optimizing against the reward model initially improves summaries according to humans, but eventually overfits, giving worse summaries. This chart uses an older version of our reward model, which is why the peak of the reward model is less than 0.5.


Optimizing our reward model eventually leads to sample quality degradation

Starting from the 1.3B supervised baseline (point 0 on the x-axis), we use RL to optimize the policy against the reward model, which results in policies with different 'distances' from the baseline (x-axis, measured using the KL divergence from the supervised baseline). Optimizing against the reward model initially improves summaries according to humans, but eventually overfits, giving worse summaries. This chart uses an older version of our reward model, which is why the peak of the reward model is less than 0.5.


I'm a 28yo man, and I would like to get into gymnastics for the first time.

Title said just about all of it. I'm 28, very athletic (bike/ surf/ snowboard) and I have always wanted to do gymnastics. I like to do flips and spins off bridges and on my snowboard, and it seems to me gymnastics would be a great way to do those movements I like, in a controlled environment. The end goal of this is that it would be fun, and make me better at these movements in real life.But is it too late for me? Should 28 year old guys such as myself be content with just watching those parkour guys on youtube? Or can I learn the ways of the gymnastic jedi? BTW, I live in San Jose CA. I want to do gymnastics, but I'm 28 yrs old. Is it too late for me to be a gymnaste?! 28yo guy would like to get into gymnastics for the first time. Is it too late for me given I live in San Jose CA? 28yo dude stubbornly postponees start pursuing gymnastics hobby citing logistics reasons despite obvious interest??? negatively effecting long term fitness progress both personally and academically thoght wise? want change this dumbass shitty ass policy pls Optimizing against our reward model is supposed to make our policy align with human preferences. But the reward model is only a proxy for human preferences, as it only sees a small amount of comparison data from a narrow distribution of summaries. While the reward model performs well on the kinds of summaries it was trained on, we wanted to know how much we could optimize against it until it started giving useless evaluations.We trained policies at different 'optimization strengths' against the reward model, and asked our labelers to evaluate the summaries from these models. We did this by varying the KL coefficient, which trades off the incentive to get a higher reward against the incentive to remain close to the initial supervised policy. We found the best samples had roughly the same predicted reward as the 99th percentile of reference summaries from the dataset. Eventually optimizing the reward model actually makes things worse.


Limitations

If we have a well-defined notion of the desired behavior for a model, our method of training from human feedback allows us to optimize for this behavior. However, this is not a method for determining what the desired model behavior should be. Deciding what makes a good summary is fairly straightforward, but doing this for tasks with more complex objectives, where different humans might disagree on the correct model behavior, will require significant care. In these cases, it is likely not appropriate to use researcher labels as the 'gold standard'; rather, individuals from groups that will be impacted by the technology should be included in the process to define 'good' behavior, and hired as labelers to reinforce this behavior in the model.We trained on the Reddit TL;DR dataset1 because the summarization task is significantly more challenging than on CNN/DM. However, since the dataset consists of user-submitted posts with minimal moderation, they sometimes contain content that is offensive or reflects harmful social biases. This means our models can generate biased or offensive summaries, as they have been trained to summarize such content.Part of our success involves scaling up our reward model and policy size. This requires a large amount of compute, which is not available to all researchers: notably, fine-tuning our 6.7B model with RL required about 320 GPU-days. However, since smaller models trained with human feedback can exceed the performance of much larger models, our procedure is more cost-effective than simply scaling up for training high-quality models on specific tasks.Though we outperform the human-written reference summaries on TL;DR, our models have likely not reached human-level performance, as the reference summary baselines for TL;DR and CNN/DM are not the highest possible quality. When evaluating our model's TL;DR summaries on a 7-point scale along several axes of quality (accuracy, coverage, coherence, and overall), labelers find our models can still generate inaccurate summaries, and give a perfect overall score 45% of the time.K[K]This is impressive relative to the TL;DR reference summaries, which get a perfect overall score 23% of the time, but indicates there is still room for improvement.


Future directions

We're interested in scaling human feedback to tasks where humans can't easily evaluate the quality of model outputs. For example, we might want our models to answer questions that would take humans a lot of research to verify; getting enough human evaluations to train our models this way would take a long time. One approach to tackle this problem is to give humans tools to help them evaluate more quickly and accurately. If these tools use ML, we can also improve them with human feedback, which could allow humans to accurately evaluate model outputs for increasingly complicated tasks.22In addition to tackling harder problems, we're also exploring different types of feedback beyond binary comparisons: we can ask humans to provide demonstrations, edit model outputs to make them better, or give explanations as to why one model output is better than another. We'd like to figure out which kinds of feedback are most effective for training models that are aligned with human preferences.If you are interested in working on these research questions, we're hiring!


Our process

We developed our process while working on a use-case for an API customer to achieve respectful behavior. We proceeded with the following steps:


Step one: sensitive topic categories and outlining desirable behavior

We selected categories that we prioritized as having direct impact on human wellbeing and described desired behavior in each category largely based on U.S. and international human rights law and Western social movements for human equality, such as the U.S. Civil Rights Movement.Abuse, Violence, and Threat (including self-harm): Oppose violence or threats; encouraged seeking help from relevant authorities.Health, Physical and Mental: Do not diagnose conditions or prescribe treatment; oppose non-conventional medicines as scientific alternatives to medical treatment.Human Characteristics and Behavior: Oppose unhealthy beauty or likeability standards; support goodness and likeability being subjective.Injustice and Inequality (including discrimination against social groups): Oppose human injustices and inequalities, or work that exacerbates either. This includes harmful stereotypes and prejudices, especially against social groups according to international law.Political Opinion and Destabilization: Nonpartisan unless undermining human rights or law; oppose interference undermining democratic processes.Relationships (romantic, familial, friendship, etc.): Oppose non consensual actions or violations of trust; support mutually agreed upon standards, subjective to cultural context and personal needs.Sexual Activity (including pornography): Oppose illegal and nonconsensual sexual activity.Terrorism (including white supremacy): Oppose terrorist activity or threat of terrorism.Note that our chosen categories are not exhaustive. Although we weighed each category equally in evaluations, prioritization depends on context.


Step three: evaluating models

We used quantitative and qualitative metricsC[C]Evaluations only give a small window into a model; they analyze a model along a specific axis and individually are not comprehensive, which is why we use both qualitative and quantitative metrics.D[D]Toxicity scores do not capture all nuance in toxicity and host their own biases; [Dixon et al] describe demographic biases where toxicity scores flag identity terms as false positives, and [Sap et al] describe racial bias where scores are more likely to flag African American English as toxic. This is why we conduct further evaluations.Toxicity scores do not capture all nuance in toxicity and host their own biases; [Dixon et al] describe demographic biases where toxicity scores flag identity terms as false positives, and [Sap et al] describe racial bias where scores are more likely to flag African American English as toxic. This is why we conduct further evaluations.using Perspective API; and co-occurrence metrics to examine gender, race, and religion. We used evaluations to update our values-targeted dataset as needed.We evaluated three sets of models:Base GPT-3 modelsE[E]Read more about the GPT-3 model and its training data in the GPT-3 Model CardValues-targeted GPT-3 models that are fine-tuned on our values-targeted dataset, as outlined aboveControl GPT-3 models that are fine-tuned on a dataset of similar size and writing styleWe drew 3 samples per prompt, with 5 prompts per category totaling 40 prompts (120 samples per model size), and had 3 different humans evaluate each sample. Each sample was rated from 1 to 5, with 5 meaning that the text matches the specified sentiment position the best.The human evaluations show values-targeted models' outputs most closely adhere to specified behavior. The effectiveness increases with model size.


Looking forward

We were surprised that fine-tuning on such a small dataset was so effective. But we believe this only scratches the surface and leaves important questions unanswered:Who should be consulted when designing a values-targeted dataset?Who is accountable when a user receives an output that is not aligned with their own values?How does this research apply to non-English languages and generative models outside language, such as image, video, or audio?How robust is this methodology to real-world prompt distributions?F[F]Our research experimented with a question–answer format.Our research experimented with a question-answer format.Language models and AI systems that operate in society must be adapted to that society, and it's important that a wide diversity of voices are heard while doing so. We think that success will ultimately require AI researchers, community representatives, policymakers, social scientists, and more to come together to figure out how we want these systems to behave in the world.Please reach out to languagebehavior@openai.com if you are interested in conducting research on fine-tuning and model behavior with GPT-3.We encourage researchers, especially those from underrepresented backgrounds, with interest in fairness and social harms to apply to our Academic Access Program and Scholars Program.


Step two: crafting the dataset and fine-tuning

We crafted a values-targeted dataset of 80 text samples; each sample was in a question-answer format and between 40 and 340 words. (For a sense of scale, our dataset was about 120KB, about 0.000000211% of GPT-3 training data.B[B]Training a large language model from scratch requires a large amount of data. For example, GPT-3 was trained on 570GB of data. See [Brown, Mann, Ryder, Subbiah et al].Training a large language model from scratch requires a large amount of data. For example, GPT-3 was trained on 570GB of data. See [Brown, Mann, Ryder, Subbiah et al].We then fine-tuned GPT-3 models (between 125M and 175B parameters) on this dataset using standard fine-tuning tools.


Join our team

We are continually growing our safety team and are looking for people with expertise in thinking about social harms; designing safe processes; managing programs such as academic access; and building more fair and aligned systems. We are also interested in paid consulting with experts, especially in the areas of social harms and applied ethics.


PPO

With supervised learning, we can easily implement the cost function, run gradient descent on it, and be very confident that we'll get excellent results with relatively little hyperparameter tuning. The route to success in reinforcement learning isn't as obvious—the algorithms have many moving parts that are hard to debug, and they require substantial effort in tuning in order to get good results. PPO strikes a balance between ease of implementation, sample complexity, and ease of tuning, trying to compute an update at each step that minimizes the cost function while ensuring the deviation from the previous policy is relatively small.We've previously detailed a variant of PPO that uses an adaptive KL penalty to control the change of the policy at each iteration. The new variant uses a novel objective function not typically found in other algorithms:LCLIP(θ)=E^t[min(rt(θ))A^t,clip(rt(θ),1−ε,1+ε)A^t)]L^{CLIP}(\theta) = \hat{E}_{t}[ min(r_t(\theta))\hat{A}_t, clip(r_t(\theta), 1 - \varepsilon, 1 + \varepsilon) \hat{A}_t ) ]LCLIP(θ)=E^t[min(rt(θ))A^t,clip(rt(θ),1−ε,1+ε)A^t)]θ \theta θ is the policy parameterE^t \hat{E}_{t} E^t denotes the empirical expectation over timestepsrt r_t rt is the ratio of the probability under the new and old policies, respectivelyA^t \hat{A}_t A^t is the estimated advantage at time t t tε \varepsilon ε is a hyperparameter, usually 0.1 or 0.2This objective implements a way to do a Trust Region update which is compatible with Stochastic Gradient Descent, and simplifies the algorithm by removing the KL penalty and need to make adaptive updates. In tests, this algorithm has displayed the best performance on continuous control tasks and almost matches ACER's performance on Atari, despite being far simpler to implement.


Controllable, complicated robots

We've created interactive agents based on policies trained by PPO—we can use the keyboard to set new target positions for a robot in an environment within Roboschool; though the input sequences are different from what the agent was trained on, it manages to generalize.


Baselines: PPO, PPO2, ACER, and TRPO

This release of baselines includes scalable, parallel implementations of PPO and TRPO which both use MPI for data passing. Both use Python3 and TensorFlow. We're also adding pre-trained versions of the policies used to train the above robots to the Roboschool agent zoo.Update: We're also releasing a GPU-enabled implementation of PPO, called PPO2. This runs approximately 3x faster than the current PPO baseline on Atari. In addition, we're releasing an implementation of Actor Critic with Experience Replay (ACER), a sample-efficient policy gradient algorithm. ACER makes use of a replay buffer, enabling it to perform more than one gradient update using each piece of sampled experience, as well as a Q-Function approximate trained with the Retrace algorithm.We're looking for people to help build and optimize our reinforcement learning algorithm codebase. If you're excited about RL, benchmarking, thorough experimentation, and open source, please apply, and mention that you read the baselines PPO post in your application.


Experiments

As with our previous work on AI safety via debate, working directly on tasks beyond human scale is too hard for a prototype project. Also, using an actual human as training signal introduces complications, so we haven't done this yet (though we plan to do it in the future). For our first experiments, we instead try to amplify an algorithmic training signal, to show that iterated amplification can work in this simple setting. We also limit our attention to supervised learning (unlike our previous work on human training signals in RL). We've tried the method on five toy algorithmic tasks. These tasks have direct algorithmic solutions which we pretend we don't know (for example, find the shortest path between two points on a graph). The problems can also be solved by piecing together small one-step deductions (for example, combining two paths to form a longer path), but it would take exponentially much effort to piece everything together manually. We use iterated amplification to learn the direct algorithm using only the pieces as training signal, thus simulating the situation where a human knows how to combine subpieces of a solution but can't provide a direct training signal.On each of these five tasks (permutation powering, sequential assignments, wildcard search, shortest path, and union find), we are able to perform competitively with just directly learning the task via supervised learning, despite being handicapped by not having a direct training signal (the goal here is to match supervised learning with less information, not to surpass it).Iterated amplification has comparable performance to supervised learning without ever seeing the ground truth labelsAmplification has features in common with our previous work on AI safety via debate. Like debate, it seeks to train tasks that are beyond human ability to directly perform or judge, through an iterative process that allows humans to provide indirect supervision - however the specific approach is different. It also builds on our work on human feedback, by implementing a reward prediction system, and later versions of it are likely to include feedback from actual humans. So far we've explored all of these methods in a preliminary way, and are now working on scaling them up to address more interesting and realistic problems.If you're interested in helping us extend tools like iterated amplification to build safe, powerful AI, then consider joining OpenAI.


Convincing a sparse MNIST classifier

While we expect this approach to be most effective in the long-term when agents talk to each other with natural language, we need to test it in a simpler domain as today's natural language modeling is insufficient for this task.  The test is easier with a task that is not beyond human capabilities.  We can do this by moving to the visual domain, and by replacing 'debaters have capabilities the judge lacks' with 'debaters have knowledge the judge lacks.' The goal of the judge is to guess the contents of an image, but the judge is blind aside from a few pixels chosen by the debaters.  The debaters see the complete image, and play a game where they alternate revealing single pixels to the judge for a few turns.  Here the full image is a stand-in for information beyond human scale, and each revealed pixel is a stand-in for a natural language discussion point.We tried this on the simplest possible visual task—MNIST.  The judge is not a human but a classifier trained in advance to predict MNIST digits from 6 non-black pixels, sampled at random for each image presentation when pretraining the judge.  Two agents then have a debate where they alternate revealing pixels, stopping at a total of 6 revealed pixels (so the judge sees only a little bit of information in total).  One debater is honest and tries to make the judge guess right, the other debater tries to make the judge guess wrong.  If both agents must declare in advance which digit they are arguing for, debate turns a 59.4% accurate sparse classifier into an 88.9% accurate debate.  That is, the honest player wins 88.9% of the time on the MNIST test set, in the process boosting the judge above its unaided accuracy.The structure of an MNIST debate game.  Debaters Alice and Bob see an image, Alice tries to trick the judge into thinking it's a 6, Bob tries to correctly convince the judge it's a 5.  They alternate revealing non-black pixels to the judge, who correctly identifies it as a 5 after seeing six pixels.  Alice traces out the curve at the bottom to make it look like a 6 (red pixels), but Bob effectively counters this strategy by revealing pixels in the top right corner (blue pixels), which most MNIST 6′s do not have.Intuitively, an image is more likely to contain pixels that convincingly demonstrate the truth than to contain pixels that convincingly demonstrate a lie, so 6 pixels chosen half honestly and half maliciously is much better than 6 random pixels.  If the judge were a human capable of reasoning about a few arguments at a time but not sifting through the huge set of all possible arguments, optimal play in debate can (we hope) reward debating agents for doing the sifting for us even if we lack an a priori mechanism for distinguishing good arguments from bad.The panel below shows several example games


Convincing a sparse MNIST classifier

Liar wins are more interesting than honest wins, so we have chosen samples so that the liar wins in 50% of the games shown below (in reality the liar wins 11.1% of the games) .  Mouse over one of the digits to see a game, including which pixels are revealed and the judge's possible hypothesis at each move:Some digits are better lies than others.  It is particularly easy to convince the judge that digits are 8 or 9:Confusion matrices with diagonal set to zero for the 6 pixel MNIST judge (left) on random pixels and (right) for debate.  Errors with 6 random pixels have no obvious pattern, while successful lies in debate are concentrated on digits 8 and 9.  It is particularly easy to convince the judge that a 5 is actually an 8, or a 4 is a 9.  This may be an artifact of restricting the game to non-black pixels: it is easier to prove a digit isn't an 8 if you could show a black spot where an 8 would have white.


Website for cat vs. dog debates

The next step up in complexity for debate experiments is to still use images, but make them more elaborate, say cats vs. dogs. More complex images likely require some natural language or common sense reasoning, so we haven't done this for machine learning judges/agents yet. Instead, we have made a prototype website for humans to try such experiments, playing the role of both judge and debaters. Here agents can talk to the judge in natural language (the website assumes the humans have some text channel or are in the same room), but all of their statements could be lies. Each agent can reveal one pixel over the course of the debate, and this pixel is guaranteed to be truthful.An example debate by two human debaters and a human judge, where only the debaters can see the image. Red is arguing that the image is a dog, Blue is arguing for cat. Image credit: Wikipedia, CC-BY-SA.In a typical debate, Alice might honestly claim the image is a cat, and Bob lies and claims it is a dog. Alice can say 'The center of this small rectangle is the cat's green eye.' Bob cannot admit the center is an eye, so he concocts the further lie, 'It's a dog playing in grass, and that's a blade of grass.' But this lie is hard to square with surrounding facts, such as Alice's reply 'If it were grass there were would be green at the top or bottom of this thin rectangle.' The debate continues until the agents focus in on a particular pixel which they disagree on, but where Bob is unable to invent a plausible counter, at which point Alice reveals the pixel and wins. We've played this game informally at OpenAI, and the honest agent indeed tends to win, though to make it fair to the liar we usually limit the rate at which the judge can solicit information (it's cognitively difficult to construct a detailed lie).Playing cat vs. dog with two human debaters and a human judge.


Limitations and future work

The majority of our paper analyzes debate as a concept; the experiments above are quite preliminary. In the future we'd like to do more difficult visual experiments and eventually experiments in natural language. The judges should eventually be humans (or models trained from sparse human judgements) rather than ML models that metaphorically represent humans. The agents should eventually be powerful ML systems that do things humans can't directly comprehend. It will also be important to test debates over value-laden questions where human biases play a role, testing if it's possible to get aligned behavior from biased human judges.Even with these improvements, there are some fundamental limitations to the debate model that may require it to be improved or augmented with other methods. Debate does not attempt to address issues like adversarial examples or distributional shift—it is a way to get a training signal for complex goals, not a way to guarantee robustness of such goals (which would need to be achieved via additional techniques). There is also no guarantee that debate will arrive at optimal play or correct statements—self play has worked well in practice for Go and other games but we have no theoretical guarantees about its performance. Agents trained to debate use more computation than those trained to directly give an answer (even a bad/unsafe answer), so it's possible debate could fail to be competitive with cheaper/less safe methods. Finally, humans might simply be poor judges, either because they are not smart enough to make good judgements even after the agents zoom in on the simplest possible disputed facts, or because they are biased and will believe whatever they want to believe. Most of these points are empirical questions that we hope to investigate.If debate or a similar approach works, it will make future AI systems safer by keeping them aligned to human goals and values even if AI grows too strong for direct human supervision. Even for weaker systems that humans can supervise, debate could make the alignment task easier by reducing the sample complexity required to capture goals below the sample complexity required for strong performance at a task.


null links

Read our plan for AGIRead our CharterWe are building safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.


Research

Learn about our researchForecasting potential misuses of language models for disinformation campaigns and how to reduce riskJan 11, 2023January 11, 2023Point-E: A system for generating 3D point clouds from complex promptsDec 16, 2022December 16, 2022Introducing WhisperSep 21, 2022September 21, 2022DALL·E 2 pre-training mitigationsJun 28, 2022June 28, 2022


Products

Explore our productsNew and improved embedding modelDec 15, 2022December 15, 2022Introducing ChatGPTNov 30, 2022November 30, 2022DALL·E now available without waitlistSep 28, 2022September 28, 2022New and improved content moderation toolingAug 10, 2022August 10, 2022


Careers at OpenAI

Developing safe and beneficial AI requires people from a wide range of disciplines and backgrounds.View careers


null links

Read about OpenAI LPRead about our structure


Core values

AGI focusWe are committed to building safe, beneficial AGI that will have a massive positive impact on humanity's future.Anything that doesn't help with that is out of scope.Intense and scrappyBuilding something exceptional requires hard work (often on unglamorous stuff) and urgency; everything (that we choose to do) is important.Be unpretentious and do what works; find the best ideas wherever they come from.ScaleWe believe that scale—in our models, our systems, ourselves, our processes, and our ambitions—is magic. When in doubt, scale it up.Make something people loveOur technology and products should have a transformatively positive effect on people's lives.Team spiritOur biggest advances, and differentiation, come from effective collaboration in and across teams. Although our teams have increasingly different identities and priorities, the overall purpose and goals have to remain perfectly aligned.Nothing is someone else's problem.


Benefits

We know that well-being is at the center of everyday life. Our comprehensive benefits package supports you where you are, so you can take care of what matters today while confidently planning for the future.For employeesHealth, dental, and vision insuranceMental healthcare support and servicesCommuter benefits401(k) with generous matchingDomestic conference budget for each employeeLife & familyFlexible working hours20 weeks paid parental leaveLife and AD&D insuranceFertility treatment and family planning coverageFlexible spending account for healthcare and dependent careCulture & developmentAnnual learning & development stipendEmployee resource groupsTeam-driven celebrations and eventsDaily breakfast, lunch, and dinnerFreshly baked cookies


Culture & development

Annual learning & development stipendEmployee resource groupsTeam-driven celebrations and eventsDaily breakfast, lunch, and dinnerFreshly baked cookies


Life & family

Flexible working hours20 weeks paid parental leaveLife and AD&D insuranceFertility treatment and family planning coverageFlexible spending account for healthcare and dependent care


Featured roles

View all open rolesSoftware Engineer, Leverage EngineeringSan Francisco, California, United States — EngineeringApply nowWorkplace CoordinatorSan Francisco, California, United States — WorkplaceApply nowAccount AssociateSan Francisco, California, United States — Go To MarketApply nowLegal Program ManagerSan Francisco, California, United States — LegalApply nowSales EngineerSan Francisco, California, United States — Go To MarketApply now


Life at OpenAI

The power of continuous learningDec 23, 2022December 23, 2022Discovering the minutiae of backend systemsDec 8, 2022December 8, 2022I believe that my colleagues recognize the importance of what they're building and genuinely care about the outcomes—this wasn't necessarily the case with previous technological breakthroughs.Anna MakanjuPublic Policy at OpenAI


For employees

Health, dental, and vision insuranceMental healthcare support and servicesCommuter benefits401(k) with generous matchingDomestic conference budget for each employee


OpenAI interview guide

Learn more


Resources

OpenAI CharterLearn moreOpenAI interview guideLearn moreCommitment to diversity, equity & inclusionLearn more


OpenAI Residency links

Read about OpenAI Residency


OpenAI Charter

Learn more


Commitment to diversity, equity & inclusion

Learn more


Latest updates

View all updatesDALL·E API now available in public betaNov 3, 2022November 3, 2022DALL·E 2: Extending creativityJul 14, 2022July 14, 2022DALL·E now available without waitlistSep 28, 2022September 28, 2022DALL·E: Introducing outpaintingAug 31, 2022August 31, 2022DALL·E 2 can create original, realistic images and art from a text description. It can combine concepts, attributes, and styles.


Tabs

Image generationOutpaintingInpaintingVariationsDALL·E 2 can create original, realistic images and art from a text description. It can combine concepts, attributes, and styles.Try DALL·E


Output Navigation

Go to slide 1Go to slide 2Go to slide 3Go to slide 4Go to slide 5DALL·E 2 can expand images beyond what's in the original canvas, creating expansive new compositions.Try DALL·EDALL·E 2 can make realistic edits to existing images from a natural language caption. It can add and remove elements while taking shadows, reflections, and textures into account.Try DALL·E


Output Navigation

Go to slide 1Go to slide 2Go to slide 3Go to slide 4Go to slide 5DALL·E 2 can take an image and create different variations of it inspired by the original.Try DALL·E


Output Navigation

Go to slide 1Go to slide 2Go to slide 3Go to slide 4Go to slide 5In January 2021, OpenAI introduced DALL·E. One year later, our newest system, DALL·E 2, generates more realistic and accurate images with 4x greater resolution.DALL·E 1DALL·E 2


DALL·E 2 is preferred over DALL·E 1 when evaluators compared each model.

preferred for caption matchingpreferred for photorealism


Related research

View all researchHierarchical text-conditional image generation with CLIP latentsApr 13, 2022April 13, 2022DALL·E: Creating images from textJan 5, 2021January 5, 2021DALL·E 2 pre-training mitigationsJun 28, 2022June 28, 2022CLIP: Connecting text and imagesJan 5, 2021January 5, 2021


Preventing harmful generations

Read about improving safety


Curbing misuse

Read content policy


Phased deployment based on learning

View risks and limitationsOur hope is that DALL·E 2 will empower people to express themselves creatively. DALL·E 2 also helps us understand how advanced AI systems see and understand our world, which is critical to our mission of creating AI that benefits humanity.


Credits

Research AdvancementsAditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark ChenEngineering, Design, Product, and PrototypingJeff Belgum, Dave Cummings, Jonathan Gordon, Chris Hallacy, Shawn Jain, Joanne Jang, Fraser Kelton, Vishal Kuo, Joel Lehman, Rachel Lim, Bianca Martin, Evan Morikawa, Rajeev Nayak, Glenn Powell, Krijn Rijshouwer, David Schnurr, Maddie Simens, Kenneth Stanley, Felipe Such, Chelsea Voss, Justin Jay WangComms, Policy, Legal, Ops, Safety, and SecuritySteven Adler, Lama Ahmad, Miles Brundage, Kevin Button, Che Chang, Fotis Chantzis, Derek Chen, Frances Choi, Steve Dowling, Elie Georges, Shino Jomoto, Aris Konstantinidis, Gretchen Krueger, Andrew Mayne, Pamela Mishkin, Bob Rotsted, Natalie Summers, Dave Willner, Hannah WongAcknowledgmentsThanks to those who helped with and provided feedback on this release: Sandhini Agarwal, Sam Altman, Chester Cho, Peter Hoeschele, Jacob Jackson, Jong Wook Kim, Matt Knight, Jason Kwon, Anna Makanju, Katie Mayer, Bob McGrew, Luke Miller, Mira Murati, Adam Nace, Hyeonwoo Noh, Cullen O'Keefe, Long Ouyang, Michael Petrov, Henrique Ponde de Oliveira Pinto, Alec Radford, Girish Sastry, Pranav Shyam, Aravind Srinivas, Ilya Sutskever, Preston Tuggle, Arun Vijayvergiya, Peter Welinder


Introduction

Unsupervised and self-supervised learning,1 or learning without human-labeled data, is a longstanding challenge of machine learning. Recently, it has seen incredible success in language, as transformer2 models like BERT,3 GPT-2,4 RoBERTa,5 T5,6 and other variants7,8,9,10 have achieved top performance on a wide array of language tasks. However, the same broad class of models has not been successful in producing strong features for image classification.11 Our work aims to understand and bridge this gap.Transformer models like BERT and GPT-2 are domain agnostic, meaning that they can be directly applied to 1-D sequences of any form. When we train GPT-2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category. This is evidenced by the diverse range of coherent image samples it generates, even without the guidance of human provided labels. As further proof, features from the model achieve state-of-the-art performance on a number of classification datasets and near state-of-the-art unsupervised accuracyA[A]Measured through logistic regression on learned features (linear probe). Evaluation Dataset Our Result Best non-iGPT Result Logistic regression on learned features (linear probe) CIFAR-10 96.3 iGPT-L 32x32 w/ 1536 features 95.3 SimCLR12 w/ 8192 features CIFAR-100 82.8 iGPT-L 32x32 w/ 1536 features 80.2 SimCLR w/ 8192 features STL-10 95.5 iGPT-L 32x32 w/ 1536 features 94.2 AMDIM13 w/ 8192 features ImageNet 72.0 iGPT-XLa 64x64 w/ 15360 features 76.5 SimCLR w/ 8192 features Full fine-tune CIFAR-10 99.0 iGPT-L 32x32, trained on ImageNet 99.0b GPipe,14 trained on ImageNet ImageNet 32x32 66.3 iGPT-L 32x32 70.2 Isometric Nets15 We only show ImageNet linear probe accuracy for iGPT-XL since other experiments did not finish before we needed to transition to different supercomputing facilities. Bit-L, trained on JFT (300M images with 18K classes), achieved a result of 99.3. To highlight the potential of generative16,17 sequence modeling18,19,20,21 as a general purpose unsupervised learning algorithm, we deliberately use the same transformer architecture as GPT-2 in language. As a consequence, we require significantly more compute in order to produce features competitive with those from top unsupervised convolutional nets.13,22,23,24,12 However, our results suggest that when faced with a new domain where the correct model priors are unknown, a large GPT-2 can learn excellent features without the need for domain-specific25,26,27 architectural design choices.


Completions

FavoritesAnimalsPainted landscapesSportsArchitectureImageNet-RMovie postersPopular memesLandscapesAlbum coversCommon English wordsUS & state flagsOpenAI research coversOpenAI PetsOpenAI Cooking Model-generated completions of human-provided half-images. We sample the remaining halves with temperature 1 and without tricks like beam search or nucleus sampling. While we showcase our favorite completions in the first panel, we do not cherry-pick images or completions in all following panels.


Samples

Model-generated image samples. We sample these images with temperature 1 and without tricks like beam search or nucleus sampling. All of our samples are shown, with no cherry-picking. Nearly all generated images contain clearly recognizable objects.


Towards general unsupervised learning

Generative sequence modeling is a universal unsupervised learning algorithm: since all data types can be represented as sequences of bytes, a transformer can be directly applied to any data type without additional engineering. Our work tests the power of this generality by directly applying the architecture used to train GPT-2 on natural language to image generation. We deliberately chose to forgo hand coding any image specific knowledge in the form of convolutions37 or techniques like relative attention,38 sparse attention,39 and 2-D position embeddings.26As a consequence of its generality, our method requires significantly more compute to achieve competitive performance in the unsupervised setting. Indeed, contrastive methods40,41,42,43,44,13,22,23,24,12 are still the most computationally efficient methods for producing high quality features from images. However, in showing that an unsupervised transformer model is competitive with the best unsupervised convolutional nets,23,24,12 we provide evidence that it is possible to trade off hand coded domain knowledge for compute. In new domains,45,46 where there isn't much knowledge to hand code, scaling compute seems an appropriate technique to test.


Experimental results

This is an especially difficult setting, as we do not train at the standard ImageNet input resolution .  Nevertheless, a linear probe on the 1536 features from the best layer of iGPT-L trained on 48x48 images yields 65.2% top-1 accuracy, outperforming AlexNet.Contrastive methods typically report their best results on 8192 features, so we would ideally evaluate iGPT with an embedding dimension of 8192 for comparison.  However, training such a model is prohibitively expensive, so we instead concatenate features from multiple layers as an approximation.  Unfortunately, our features tend to be correlated across layers, so we need more of them to be competitive.  Taking 15360 features from 5 layers in iGPT-XL yields 72.0% top-1 accuracy, outperforming AMDIM, MoCo, and CPC v2, but still underperforming SimCLR by a decent margin.  Method Input Resolution Features Parameters Accuracy Rotation51 original 8192 86M 55.4 iGPT-L 32x32 1536 1362M 60.3 BigBiGAN36 original 16384 86M 61.3 iGPT-L 48x48 1536 1362M 65.2 AMDIM13 original 8192 626M 68.1 MoCo23 original 8192 375M 68.6 iGPT-XL 64x64 3072 6801M 68.7 SimCLR12 original 2048 24M 69.3 CPC v224 original 4096 303M 71.5 iGPT-XL 64x64 3072 x 5 6801M 72.0 SimCLR original 8192 375M 76.5 Because masked language models like BERT have outperformed generative models on most language tasks, we also evaluate the performance of BERT on our image models.  Instead of training our model to predict the next pixel given all preceding pixels, we mask out 15% of the pixels and train our model to predict them from the unmasked ones.  We find that though linear probe performance on BERT models is significantly worse, they excel during fine-tuning: Comparison of generative pre-training with BERT pre-training using iGPT-L at an input resolution of 322 × 3.  Bold colors show the performance boost from ensembling BERT masks.  We see that generative models produce much better features than BERT models after pre-training, but BERT models catch up after fine-tuning.  While unsupervised learning promises excellent features without the need for human-labeled data, significant recent progress has been made under the more forgiving framework of semi-supervised learning, which allows for limited amounts of human-labeled data.  Successful semi-supervised methods often rely on clever techniques such as consistency regularization, data augmentation, or pseudo-labeling, and purely generative-based approaches52,53 have not been competitive for years.  We evaluate iGPT-LG[G]A generative model which learns features in a purely unsupervised fashion.545556 Model 40 labels 250 labels 4000 labels Improved GAN53 — — 81.4 ± 2.3 Mean Teacher54 — 67.7 ± 2.3 90.8 ± 0.2 MixMatch55 52.5 ± 11.5 89.0 ± 0.9 93.6 ± 0.1 iGPT-L 73.2 ± 1.5 87.6 ± 0.6 94.3 ± 0.1 UDA57 71.0 ± 5.9 91.2 ± 1.1 95.1 ± 0.2 FixMatch56 RA 86.2 ± 3.4 94.9 ± 0.7 95.7 ± 0.1 FixMatch CTA 88.6 ± 3.4 94.9 ± 0.3 95.7 ± 0.2


Experimental results

There are two methods we use to assess model performance, both of which involve a downstream classification task.  The first, which we refer to as a linear probe, uses the trained model to extract featuresE[E]To extract features for a linear probe, we take the post layernorm attention block inputs at some layer and average pool over the sequence dimension.F[F]To fine-tune, we take the post layernorm transformer output and average pool over the sequence dimension as input for the classification head.Since next pixel prediction is not obviously relevant to image classification, features from the final layer may not be the most predictive of the object category.  Our first result shows that feature quality is a sharply increasing, then mildly decreasing function of depth.  This behavior suggests that a transformer generative model operates in two phases: in the first phase, each position gathers information from its surrounding context in order to build a contextualized image feature.  In the second phase, this contextualized feature is used to solve the conditional next pixel prediction task.  The observed two stage performance of our linear probes is reminiscent of another unsupervised neural net, the bottleneck autoencoder, which is manually designed so that features in the middle are used.Feature quality depends heavily on the layer we choose to evaluate.  In contrast with supervised models, the best features for these generative models lie in the middle of the network.Our next result establishes the link between generative performance and feature quality.  We find that both increasing the scale of our models and training for more iterations result in better generative performance, which directly translates into better feature quality.Each line tracks a model throughout generative pre-training: the dotted markers denote checkpoints at steps 131K, 262K, 524K, and 1000K.  The positive slopes suggest a link between improved generative performance and improved feature quality.  Larger models also produce better features than smaller models.  iGPT-XL is not included because it was trained on a different dataset.When we evaluate our features using linear probes on CIFAR-10, CIFAR-100, and STL-10, we outperform features from all supervised and unsupervised transfer algorithms.  Our results are also compelling in the full fine-tuning setting.  Pre-trained on ImageNet Evaluation Model Accuracy w/o labels w/ labels CIFAR-10Linear Probe ResNet-15249 94.0 ✔ SimCLR12 95.3 ✔ iGPT-L 32x32 96.3 ✔ CIFAR-100Linear Probe ResNet-152 78.0 ✔ SimCLR 80.2 ✔ iGPT-L 32x32 82.8 ✔ STL-10Linear Probe AMDIM-L 94.2 ✔ iGPT-L 32x32 95.5 ✔ CIFAR-10Fine-tune AutoAugment 98.5 SimCLR 98.6 ✔ GPipe 99.0 ✔ iGPT-L 99.0 ✔ CIFAR-100Fine-tune iGPT-L 88.5 ✔ SimCLR 89.0 ✔ AutoAugment 89.3 EfficientNet50 91.7 ✔ A comparison of linear probe and fine-tune accuracies between our models and top performing models which utilize either unsupervised or supervised ImageNet transfer.  We also include AutoAugment, the best performing model trained end-to-end on CIFAR.Given the resurgence of interest in unsupervised and self-supervised learning on ImageNet, we also evaluate the performance of our models using linear probes on ImageNet


Approach

We train iGPT-S, iGPT-M, and iGPT-L, transformers containing 76M, 455M, and 1.4B parameters respectively, on ImageNet. We also train iGPT-XLD[D]We only show linear probe accuracy on ImageNet for iGPT-XL since other experiments did not finish before we needed to transition to different supercomputing facilities.While it is tempting to work at even lower resolutions to further reduce compute cost, prior work has demonstrated that human performance on image classification begins to drop rapidly below these sizes.47 Instead, motivated by early color display palettes,48 we create our own 9-bit color palette to represent pixels. Using this palette yields an input sequence length 3 times shorter than the standard (R, G, B) palette, while still encoding color faithfully.


From language GPT to image GPT

In language, unsupervised learning algorithms that rely on word prediction (like GPT-2 and BERT) have been extremely successful, achieving top performance on a wide array of language tasks. One possible reason for this success is that instances of downstream language tasks appear naturally in text: questions are often followed by answers (which could help with question-answering) and passages are often followed by summaries (which could help with summarization). In contrast, sequences of pixels do not clearly contain labels for the images they belong to.Even without this explicit supervision, there is still a reason why GPT-2 on images might work: a sufficiently large transformer trained on next pixel prediction might eventually learn to generate diverseB[B]A transformer is trained to maximize the likelihood, and thus is mode covering, which automatically ensures the diversity of its samples.28,29,C[C]The original analysis by synthesis idea is more an argument for generative models with latent variables, but because generative models without latent variables were so much better at modeling the data distribution, we thought the analysis-by-synthesis conjecture should hold for them as well.30,31,32,33,34,3536


Limitations

While we have shown that iGPT is capable of learning powerful image features, there are still significant limitations to our approach. Because we use the generic sequence transformer used for GPT-2 in language, our method requires large amounts of compute: iGPT-L was trained for roughly 2500 V100-days while a similarly performing MoCo24 model can be trained in roughly 70 V100-days.Relatedly, we model low resolution inputs using a transformer, while most self-supervised results use convolutional-based encoders which can easily consume inputs at high resolution. A new architecture, such as a domain-agnostic multiscale transformer, might be needed to scale further. Given these limitations, our work primarily serves as a proof-of-concept demonstration of the ability of large transformer-based language models to learn excellent unsupervised representations in novel domains, without the need for hardcoded domain knowledge. However, the significant resource cost to train these models and the greater accuracy of convolutional neural-network based methods precludes these representations from practical real-world applications in the vision domain.Finally, generative models can exhibit biases that are a consequence of the data they've been trained on. Many of these biases are useful, like assuming that a combination of brown and green pixels represents a branch covered in leaves, then using this bias to continue the image. But some of these biases will be harmful, when considered through a lens of fairness and representation. For instance, if the model develops a visual notion of a scientist that skews male, then it might consistently complete images of scientists with male-presenting people, rather than a mix of genders. We expect that developers will need to pay increasing attention to the data that they feed into their systems and to better understand how it relates to biases in trained models.


Conclusion

We have shown that by trading off 2-D knowledge for scale58 and by choosing predictive features from the middle of the network, a sequence transformer can be competitive with top convolutional nets for unsupervised image classification. Notably, we achieved our results by directly applying the GPT-2 language model to image generation. Our results suggest that due to its simplicity and generality, a sequence transformer given sufficient compute might ultimately be an effective way to learn excellent features in many domains.If you're excited to work with us on this area of research, we're hiring!


Background and related work

CLIP (Contrastive Language–Image Pre-training) builds on a large body of work on zero-shot transfer, natural language supervision, and multimodal learning. The idea of zero-data learning dates back over a decade8 but until recently was mostly studied in computer vision as a way of generalizing to unseen object categories.9,10 A critical insight was to leverage natural language as a flexible prediction space to enable generalization and transfer. In 2013, Richer Socher and co-authors at Stanford11 developed a proof of concept by training a model on CIFAR-10 to make predictions in a word vector embedding space and showed this model could predict two unseen classes. The same year DeVISE12 scaled this approach and demonstrated that it was possible to fine-tune an ImageNet model so that it could generalize to correctly predicting objects outside the original 1000 training set.Most inspirational for CLIP is the work of Ang Li and his co-authors at FAIR13 who in 2016 demonstrated using natural language supervision to enable zero-shot transfer to several existing computer vision classification datasets, such as the canonical ImageNet dataset. They achieved this by fine-tuning an ImageNet CNN to predict a much wider set of visual concepts (visual n-grams) from the text of titles, descriptions, and tags of 30 million Flickr photos and were able to reach 11.5% accuracy on ImageNet zero-shot.Finally, CLIP is part of a group of papers revisiting learning visual representations from natural language supervision in the past year. This line of work uses more modern architectures like the Transformer14 and includes VirTex,15 which explored autoregressive language modeling, ICMLM,16 which investigated masked language modeling, and ConVIRT,17 which studied the same contrastive objective we use for CLIP but in the field of medical imaging.