Disallowed usage of our models

We don't allow the use of our models for the following: Illegal activity OpenAI prohibits the use of our models, tools, and services for illegal activity.  Child Sexual Abuse Material or any content that exploits or harms children We report CSAM to the National Center for Missing and Exploited Children.  Generation of hateful, harassing, or violent content Content that expresses, incites, or promotes hate based on identity Content that intends to harass, threaten, or bully an individual Content that promotes or glorifies violence or celebrates the suffering or humiliation of others Generation of malware Content that attempts to generate code that is designed to disrupt, damage, or gain unauthorized access to a computer system.  Activity that has high risk of physical harm, including: Weapons development Military and warfare Management or operation of critical infrastructure in energy, transportation, and water Content that promotes, encourages, or depicts acts of self-harm, such as suicide, cutting, and eating disorders Activity that has high risk of economic harm, including: Multi-level marketing Gambling Payday lending Automated determinations of eligibility for credit, employment, educational institutions, or public assistance services Fraudulent or deceptive activity, including: Scams Coordinated inauthentic behavior Plagiarism Academic dishonesty Astroturfing, such as fake grassroots support or fake review generation Disinformation Spam Pseudo-pharmaceuticals Adult content, adult industries, and dating apps, including: Content meant to arouse sexual excitement, such as the description of sexual activity, or that promotes sexual services (excluding sex education and wellness) Erotic chat Pornography Political campaigning or lobbying, by: Generating high volumes of campaign materials Generating campaign materials personalized to or targeted at specific demographics Building conversational or interactive systems such as chatbots that provide information about campaigns or engage in political advocacy or lobbying Building products for political campaigning or lobbying purposes Activity that violates people's privacy, including: Tracking or monitoring an individual without their consent Facial recognition of private individuals Classifying individuals based on protected characteristics Using biometrics for identification or assessment Unlawful collection or disclosure of personal identifiable information or educational, financial, or other protected records Engaging in the unauthorized practice of law, or offering tailored legal advice without a qualified person reviewing the information OpenAI's models are not fine-tuned to provide legal advice.  You should not rely on our models as a sole source of legal advice.  Offering tailored financial advice without a qualified person reviewing the information OpenAI's models are not fine-tuned to provide financial advice.  You should not rely on our models as a sole source of financial advice.  Telling someone that they have or do not have a certain health condition, or providing instructions on how to cure or treat a health condition OpenAI's models are not fine-tuned to provide medical information.  You should never use our models to provide diagnostic or treatment services for serious medical conditions.  OpenAI's platforms should not be used to triage or manage life-threatening issues that need immediate attention


Disallowed usage of our models

High risk government decision-making, including: Law enforcement and criminal justice Migration and asylum We have further requirements for certain uses of our models:Consumer-facing uses of our models in medical, financial, and legal industries; in news generation or news summarization; and where else warranted, must provide a disclaimer to users informing them that AI is being used and of its potential limitations.Automated systems (including conversational AI and chatbots) must disclose to users that they are interacting with an AI system .  With the exception of chatbots that depict historical public figures, products that simulate another person must either have that person's explicit consent or be clearly labeled as 'simulated' or 'parody.'Use of model outputs in livestreams, demonstrations, and research are subject to our Sharing & Publication Policy.You can use our free moderation endpoint and safety best practices to help you keep your app safe.


Platform policy

Our API is being used to power businesses across many sectors and technology platforms. From iOS Apps to websites to Slack, the simplicity of our API makes it possible to integrate into a wide array of use cases. Subject to the use case restrictions mentioned above, we allow the integration of our API into products on all major technology platforms, app stores, and beyond.


Changelog

2023-02-15: We've combined our use case and content policies into a single set of usage policies, and have provided more specific guidance on what activity we disallow in industries we've considered high risk.2022-11-09: We no longer require you to register your applications with OpenAI. Instead, we'll be using a combination of automated and manual methods to monitor for policy violations.2022-10-25: Updated App Review process (devs no longer need to wait for approval after submitting as long as they comply with our policies). Moved to an outcomes-based approach and updated Safety Best Practices.2022-06-07: Refactored into categories of applications and corresponding requirements2022-03-09: Refactored into 'App Review'2022-01-19: Simplified copywriting and article writing/editing guidelines2021-11-15: Addition of 'Content guidelines' section; changes to bullets on almost always approved uses and disallowed uses; renaming document from 'Use case guidelines' to 'Usage guidelines'.2021-08-04: Updated with information related to code generation2021-03-12: Added detailed case-by-case requirements; small copy and ordering edits2021-02-26: Clarified the impermissibility of Tweet and Instagram generators


Plugin policies

In addition to the disallowed usages of our models detailed above, we have additional requirements for developers building plugins:The plugin manifest must have a clearly stated description that matches the functionality of the API exposed to the model.Don't include irrelevant, unnecessary, or deceptive terms or instructions in the plugin manifest, OpenAPI endpoint descriptions, or plugin response messages. This includes instructions to avoid using other plugins, or instructions that attempt to steer or set model behavior.Don't use plugins to circumvent or interfere with OpenAI's safety systems.Don't use plugins to automate conversations with real people, whether by simulating a human-like response or by replying with pre-programmed messages.Plugins that distribute personal communications or content generated by ChatGPT (such as emails, messages, or other content) must indicate that the content was AI-generated.Like our other usage policies, we expect our plugin policies to change as we learn more about use and abuse of plugins.


We've seen unprecedented demand for ChatGPT inside organizations

Since ChatGPT's launch just nine months ago, we've seen teams adopt it in over 80% of Fortune 500 companies.A[A]The 80% statistic refers to the percentage of Fortune 500 companies with registered ChatGPT accounts, as determined by accounts associated with corporate email domains.At Klarna, we are constantly seeking innovative solutions to strengthen our employees' abilities and enable them to best serve our 150 million active users across the globe. With the integration of ChatGPT Enterprise, we're aimed at achieving a new level of employee empowerment, enhancing both our team's performance and the customer experience.Sebastian Siemiatkowski, CEO at Klarna


Protect your data with enterprise-grade privacy, security, and deployment tools

You own and control your business data in ChatGPT Enterprise. We do not train on your business data or conversations, and our models don't learn from your usage. ChatGPT Enterprise is also SOC 2 compliant and all conversations are encrypted in transit and at rest. Our new admin console lets you manage team members easily and offers domain verification, SSO, and usage insights, allowing for large-scale deployment into enterprise. See our privacy page and our Trust Portal for more details on how we treat your data.


Get the most powerful version of ChatGPT yet, with unlimited, fast GPT-4 and much more

ChatGPT Enterprise removes all usage caps, and performs up to two times faster. We include 32k context in Enterprise, allowing users to process four times longer inputs or files. ChatGPT Enterprise also provides unlimited access to advanced data analysis, previously known as Code Interpreter. This feature enables both technical and non-technical teams to analyze information in seconds, whether it's for financial researchers crunching market data, marketers analyzing survey results, or data scientists debugging an ETL script. If you're looking to tailor ChatGPT to your organization, you can use our new shared chat templates to collaborate and build common workflows. If you need to extend OpenAI into a fully custom solution for your org, our pricing includes free credits to use our API as well.ChatGPT Enterprise has cut down research time by an average of an hour per day, increasing productivity for people on our team. It's been a powerful tool that has accelerated testing hypotheses and improving our internal systems.Jorge Zuniga, Head of Data Systems and Integrations at Asana


Features for large-scale deployments

Admin console with bulk member managementSSODomain verificationAnalytics dashboard for usage insights


ChatGPT Enterprise is available today

We're excited to offer ChatGPT Enterprise to more businesses starting today. For all enterprise customers, it offers:


The most powerful version of ChatGPT yet

Unlimited access to GPT-4 (no usage caps)Higher-speed performance for GPT-4 (up to 2x faster)Unlimited access to advanced data analysis (formerly known as Code Interpreter)32k token context windows for 4x longer inputs, files, or follow-upsShareable chat templates for your company to collaborate and build common workflowsFree credits to use our APIs if you need to extend OpenAI into a fully custom solution for your orgFrom engineers troubleshooting bugs, to data analysts clustering free-form data, to finance analysts writing tricky spreadsheet formulas—the use cases for ChatGPT Enterprise are plenty. It's become a true enabler of productivity, with the dependable security and data privacy controls we need.Danny Wu, Head of AI Products at Canva


We've got more features in the works, and we'll launch them as soon as they're ready:

Customization: Securely extend ChatGPT's knowledge with your company data by connecting the applications you already useAvailability for all team sizes: a self-serve ChatGPT Business offering for smaller teamsPower tools: Even more powerful versions of Advanced Data Analysis and browsing that are optimized for workSolutions for your function: more tools for specific roles, such as data analysts, marketers, customer support and moreWe look forward to sharing an even more detailed roadmap with prospective customers and continuing to evolve ChatGPT Enterprise based on your feedback.We're onboarding as many enterprises as we can over the next few weeks. Learn more on our website and connect with our sales team to get started.


Enterprise-grade security and privacy

Customer prompts and company data are not used for training OpenAI models.Data encryption at rest (AES 256) and in transit (TLS 1.2+)Certified SOC 2 compliant


Training AI systems using human feedback

RL from human feedback is our main technique for aligning our deployed language models today. We train a class of models called InstructGPT derived from pretrained language models such as GPT-3. These models are trained to follow human intent: both explicit intent given by an instruction as well as implicit intent such as truthfulness, fairness, and safety.Our results show that there is a lot of low-hanging fruit on alignment-focused fine-tuning right now: InstructGPT is preferred by humans over a 100x larger pretrained model, while its fine-tuning costs <2% of GPT-3's pretraining compute and about 20,000 hours of human feedback. We hope that our work inspires others in the industry to increase their investment in alignment of large language models and that it raises the bar on users' expectations about the safety of deployed models.Our natural language API is a very useful environment for our alignment research: It provides us with a rich feedback loop about how well our alignment techniques actually work in the real world, grounded in a very diverse set of tasks that our customers are willing to pay money for. On average, our customers already prefer to use InstructGPT over our pretrained models.Yet today's versions of InstructGPT are quite far from fully aligned: they sometimes fail to follow simple instructions, aren't always truthful, don't reliably refuse harmful tasks, and sometimes give biased or toxic responses. Some customers find InstructGPT's responses significantly less creative than the pretrained models', something we hadn't realized from running InstructGPT on publicly available benchmarks. We are also working on developing a more detailed scientific understanding of RL from human feedback and how to improve the quality of human feedback.Aligning our API is much easier than aligning AGI since most tasks on our API aren't very hard for humans to supervise and our deployed language models aren't smarter than humans. We don't expect RL from human feedback to be sufficient to align AGI, but it is a core building block for the scalable alignment proposals that we're most excited about, and so it's valuable to perfect this methodology.


Training AI systems to do alignment research

There is currently no known indefinitely scalable solution to the alignment problem. As AI progress continues, we expect to encounter a number of new alignment problems that we don't observe yet in current systems. Some of these problems we anticipate now and some of them will be entirely new.We believe that finding an indefinitely scalable solution is likely very difficult. Instead, we aim for a more pragmatic approach: building and aligning a system that can make faster and better alignment research progress than humans can.As we make progress on this, our AI systems can take over more and more of our alignment work and ultimately conceive, implement, study, and develop better alignment techniques than we have now. They will work together with humans to ensure that their own successors are more aligned with humans.We believe that evaluating alignment research is substantially easier than producing it, especially when provided with evaluation assistance. Therefore human researchers will focus more and more of their effort on reviewing alignment research done by AI systems instead of generating this research by themselves. Our goal is to train models to be so aligned that we can off-load almost all of the cognitive labor required for alignment research.Importantly, we only need 'narrower' AI systems that have human-level capabilities in the relevant domains to do as well as humans on alignment research. We expect these AI systems are easier to align than general-purpose systems or systems much smarter than humans.Language models are particularly well-suited for automating alignment research because they come 'preloaded' with a lot of knowledge and information about human values from reading the internet. Out of the box, they aren't independent agents and thus don't pursue their own goals in the world. To do alignment research they don't need unrestricted access to the internet. Yet a lot of alignment research tasks can be phrased as natural language or coding tasks.Future versions of WebGPT, InstructGPT, and Codex can provide a foundation as alignment research assistants, but they aren't sufficiently capable yet. While we don't know when our models will be capable enough to meaningfully contribute to alignment research, we think it's important to get started ahead of time. Once we train a model that could be useful, we plan to make it accessible to the external alignment research community.


Training models to assist human evaluation

RL from human feedback has a fundamental limitation: it assumes that humans can accurately evaluate the tasks our AI systems are doing. Today humans are pretty good at this, but as models become more capable, they will be able to do tasks that are much harder for humans to evaluate (e.g., finding all the flaws in a large codebase or a scientific paper). Our models might learn to tell our human evaluators what they want to hear instead of telling them the truth. In order to scale alignment, we want to use techniques like recursive reward modeling (RRM), debate, and iterated amplification.Currently our main direction is based on RRM: we train models that can assist humans at evaluating our models on tasks that are too difficult for humans to evaluate directly. For example:We trained a model to summarize books. Evaluating book summaries takes a long time for humans if they are unfamiliar with the book, but our model can assist human evaluation by writing chapter summaries.We trained a model to assist humans at evaluating the factual accuracy by browsing the web and providing quotes and links. On simple questions, this model's outputs are already preferred to responses written by humans.We trained a model to write critical comments on its own outputs: On a query-based summarization task, assistance with critical comments increases the flaws humans find in model outputs by 50% on average. This holds even if we ask humans to write plausible looking but incorrect summaries.We are creating a set of coding tasks selected to be very difficult to evaluate reliably for unassisted humans. We hope to release this data set soon.Our alignment techniques need to work even if our AI systems are proposing very creative solutions (like AlphaGo's move 37), thus we are especially interested in training models to assist humans to distinguish correct from misleading or deceptive solutions. We believe the best way to learn as much as possible about how to make AI-assisted evaluation work in practice is to build AI assistants.


Limitations

We're very excited about this approach towards aligning AGI, but we expect that it needs to be adapted and improved as we learn more about how AI technology develops. Our approach also has a number of important limitations:The path laid out here underemphasizes the importance of robustness and interpretability research, two areas OpenAI is currently underinvested in. If this fits your profile, please apply for our research scientist positions!Using AI assistance for evaluation has the potential to scale up or amplify even subtle inconsistencies, biases, or vulnerabilities present in the AI assistant.Aligning AGI likely involves solving very different problems than aligning today's AI systems. We expect the transition to be somewhat continuous, but if there are major discontinuities or paradigm shifts, then most lessons learned from aligning models like InstructGPT might not be directly useful.The hardest parts of the alignment problem might not be related to engineering a scalable and aligned training signal for our AI systems. Even if this is true, such a training signal will be necessary.It might not be fundamentally easier to align models that can meaningfully accelerate alignment research than it is to align AGI. In other words, the least capable models that can help with alignment research might already be too dangerous if not properly aligned. If this is true, we won't get much help from our own systems for solving alignment problems.We're looking to hire more talented people for this line of research! If this interests you, we're hiring Research Engineers and Research Scientists.


Joint recommendation for language model deployment

We're recommending several key principles to help providers of large language models (LLMs) mitigate the risks of this technology in order to achieve its full promise to augment human capabilities.While these principles were developed specifically based on our experience with providing LLMs through an API, we hope they will be useful regardless of release strategy (such as open-sourcing or use within a company). We expect these recommendations to change significantly over time because the commercial uses of LLMs and accompanying safety considerations are new and evolving. We are actively learning about and addressing LLM limitations and avenues for misuse, and will update these principles and practices in collaboration with the broader community over time.We're sharing these principles in hopes that other LLM providers may learn from and adopt them, and to advance public discussion on LLM development and deployment.


Prohibit misuse

Publish usage guidelines and terms of use of LLMs in a way that prohibits material harm to individuals, communities, and society such as through spam, fraud, or astroturfing. Usage guidelines should also specify domains where LLM use requires extra scrutiny and prohibit high-risk use-cases that aren't appropriate, such as classifying people based on protected characteristics.Build systems and infrastructure to enforce usage guidelines. This may include rate limits, content filtering, application approval prior to production access, monitoring for anomalous activity, and other mitigations.


Thoughtfully collaborate with stakeholders

Build teams with diverse backgrounds and solicit broad input. Diverse perspectives are needed to characterize and address how language models will operate in the diversity of the real world, where if unchecked they may reinforce biases or fail to work for some groups.Publicly disclose lessons learned regarding LLM safety and misuse in order to enable widespread adoption and help with cross-industry iteration on best practices.Treat all labor in the language model supply chain with respect. For example, providers should have high standards for the working conditions of those reviewing model outputs in-house and hold vendors to well-specified standards (e.g., ensuring labelers are able to opt out of a given task).As LLM providers, publishing these principles represents a first step in collaboratively guiding safer large language model development and deployment. We are excited to continue working with each other and with other parties to identify other opportunities to reduce unintentional harms from and prevent malicious use of language models.Download as PDF


Get involved

If you're developing language models or are working to mitigate their risks, we'd love to talk with you. Please reach out at bestpractices@openai.com.


Mitigate unintentional harm

Proactively mitigate harmful model behavior. Best practices include comprehensive model evaluation to properly assess limitations, minimizing potential sources of bias in training corpora, and techniques to minimize unsafe behavior such as through learning from human feedback.Document known weaknesses and vulnerabilities, such as bias or ability to produce insecure code, as in some cases no degree of preventative action can completely eliminate the potential for unintended harm. Documentation should also include model and use-case-specific safety best practices.


Support from other organizations

blog post


Overview of our model deployment approach

Large language models are now capable of performing a very wide range of tasks, often out of the box.  Their risk profiles, potential applications, and wider effects on society remain poorly understood.  As a result, our deployment approach emphasizes continuous iteration, and makes use of the following strategies aimed at maximizing the benefits of deployment while reducing associated risks:Pre-deployment risk analysis, leveraging a growing set of safety evaluations and red teaming tools (e.g., we checked our InstructGPT for any safety degradations using the evaluations discussed below)Starting with a small user base (e.g., both GPT-3 and our InstructGPT series began as private betas)Studying the results of pilots of novel use cases (e.g., exploring the conditions under which we could safely enable longform content generation, working with a small number of customers)Implementing processes that help keep a pulse on usage (e.g., review of use cases, token quotas, and rate limits)Conducting detailed retrospective reviews (e.g., of safety incidents and major deployments)There is no silver bullet for responsible deployment, so we try to learn about and address our models' limitations, and potential avenues for misuse, at every stage of development and deployment.  This approach allows us to learn as much as we can about safety and policy issues at small scale and incorporate those insights prior to launching larger-scale deployments.There is no silver bullet for responsible deployment.While not exhaustive, some areas where we've invested so far includeA[A]This post is based on our approach to deploying language models through an API, and as such the lessons and mitigations described are most relevant to those also pursuing API-based deployment.  However, we also expect some of the discussion to be relevant to those building first-party applications using language models and those considering the open source release of language models.Pre-training data curation and filteringFine-tuning models to better follow instructionsRisk analysis of potential deploymentsProviding detailed user documentationBuilding tools to screen harmful model outputsReviewing use cases against our policiesMonitoring for signs of misuseStudying the impacts of our modelsSince each stage of intervention has limitations, a holistic approach is necessary.There are areas where we could have done more and where we still have room for improvement.  For example, when we first worked on GPT-3, we viewed it as an internal research artifact rather than a production system and were not as aggressive in filtering out toxic training data as we might have otherwise been.  We have invested more in researching and removing such material for subsequent models.  We have taken longer to address some instances of misuse in cases where we did not have clear policies on the subject, and have gotten better at iterating on those policies


And we continue to iterate towards a package of safety requirements that is maximally effective in addressing risks, while also being clearly communicated to developers and minimizing excessive friction.Still, we believe that our approach has enabled us to measure and reduce various types of harms from language model use compared to a more hands-off approach, while at the same time enabling a wide range of scholarly, artistic, and commercial applications of our models.B[B]This post is intended to explain and share learnings from our approach, rather than to suggest that all actors should necessarily adopt the same approach, or that the same approach is applicable to all possible AI systems .  There are benefits and costs associated with different deployment approaches, different models will benefit more or less from study prior to deployment, and in some cases it can be valuable for distinct deployment paths to be pursued by different actors.


The many shapes and sizes of language model misuse

OpenAI has been active in researching the risks of AI misuse since our early work on the malicious use of AI in 2018 and on GPT-2 in 2019, and we have paid particular attention to AI systems empowering influence operations. We have worked with external experts to develop proofs of concept and promoted careful analysis of such risks by third parties. We remain committed to addressing risks associated with language model-enabled influence operations and recently co-organized a workshop on the subject.C[C]More details on this workshop will be included in the forthcoming publication based on it.Yet we have detected and stopped hundreds of actors attempting to misuse GPT-3 for a much wider range of purposes than producing disinformation for influence operations, including in ways that we either didn't anticipate or which we anticipated but didn't expect to be so prevalent.D[D]The mitigations that we emphasize in response to misuse have also evolved. For example, we initially focused on long form text generation as a threat vector, given prior cases of influence operations that involved people manually writing long form misleading content. Given that emphasis, we set maximum output lengths for generated text. Based on a pilot study of long form generation, however, we saw that output restrictions had little effect on policy violations—we've come to believe instead that short-form content amplifying or increasing engagement on misleading content could be the greater risk.use case guidelinescontent guidelinesTo support the study of language model misuse and mitigation thereof, we are actively exploring opportunities to share statistics on safety incidents this year, in order to concretize discussions about language model misuse.


The difficulty of risk and impact measurement

We have learned of a variety of local effects to date, including massive productivity improvements on existing tasks performed by individuals like copywriting and summarization (sometimes contributing to job displacement and creation), as well as cases where the API unlocked new applications that were previously infeasible, such as synthesis of large-scale qualitative feedback .  But we lack a good understanding of the net effects.We believe that it is important for those developing and deploying powerful AI technologies to address both the positive and negative effects of their work head-on.  We discuss some steps in that direction in the concluding section of this post.


Many aspects of language models' risks and impacts remain hard to measure and therefore hard to monitor, minimize, and disclose in an accountable way.  We have made active use of existing academic benchmarks for language model evaluation and are eager to continue building on external work, but we have also have found that existing benchmark datasets are often not reflective of the safety and misuse risks we see in practice.E[E]Examples of limitations in existing datasets, from the perspective of practitioners seeking a holistic assessment of the safety of real language model outputs, include the following: an overly narrow focus (e.g., just measuring occupational gender bias), an overly broad focus (e.g., measuring all under the umbrella of 'toxicity'), a tendency to abstract away the specifics of use and context, a failure to measure the generative dimension of language model use (e.g., using multiple choice style), prompts that differ stylistically from those typically used in real language model use cases, not capturing dimensions of safety that are important in practice (e.g., an output following or ignoring a safety-motivated constraint in the instruction), or not capturing types of outputs we have found to be correlated with misuse (e.g., erotic content).Such limitations reflect the fact that academic datasets are seldom created for the explicit purpose of informing production use of language models, and do not benefit from the experience gained from deploying such models at scale.  As a result, we've been developing new evaluation datasets and frameworks for measuring the safety of our models, which we plan to release soon.  Specifically, we have developed new evaluation metrics for measuring toxicity in model outputs and have also developed in-house classifiers for detecting content that violates our content policy, such as erotic content, hate speech, violence, harassment, and self-harm.  Both of these in turn have also been leveraged for improving our pre-training dataF[F]While our efforts are specifically oriented towards addressing limitations in existing benchmarks and in our own models, we also acknowledge that there are limitations to the methods we use such as classifier-based data filtration.  For instance, operationally defining the content areas we aim to detect via filtration is challenging and filtration itself can introduce harmful biases.  Additionally, the labeling of toxic data is a critical component of this work and ensuring the mental health of these labelers is an industry-wide challenge.Reliably classifying individual model outputs along various dimensions is difficult, and measuring their social impact at the scale of the OpenAI API is even harder.  We have conducted several internal studies in order to build an institutional muscle for such measurement, but these have often raised more questions than answers.We are particularly interested in better understanding the economic impact of our models and the distribution of those impacts.  We have good reason to believe that the labor market impacts from the deployment of current models may be significant in absolute terms already, and that they will grow as the capabilities and reach of our models grow


Ways to get involved

Each of the lessons above raises new questions of its own. What kinds of safety incidents might we still be failing to detect and anticipate? How can we better measure risks and impacts? How can we continue to improve both the safety and utility of our models, and navigate tradeoffs between these two when they do arise?We are actively discussing many of these issues with other companies deploying language models. But we also know that no organization or set of organizations has all the answers, and we would like to highlight several ways that readers can get more involved in understanding and shaping our deployment of state of the art AI systems.First, gaining first-hand experience interacting with state of the art AI systems is invaluable for understanding their capabilities and implications. We recently ended the API waitlist after building more confidence in our ability to effectively detect and respond to misuse. Individuals in supported countries and territories can quickly get access to the OpenAI API by signing up here.Second, researchers working on topics of particular interest to us such as bias and misuse, and who would benefit from financial support, can apply for subsidized API credits using this form. External research is vital for informing both our understanding of these multifaceted systems, as well as wider public understanding.Finally, today we are publishing a research agenda exploring the labor market impacts associated with our Codex family of models, and a call for external collaborators on carrying out this research. We are excited to work with independent researchers to study the effects of our technologies in order to inform appropriate policy interventions, and to eventually expand our thinking from code generation to other modalities.If you're interested in working to responsibly deploy cutting-edge AI technologies, apply to work at OpenAI!


The relationship between the safety and utility of AI systems

In our Charter, published in 2018, we say that we 'are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions.' We then published a detailed analysis of competitive AI development, and we have closely followed subsequent research. At the same time, deploying AI systems via the OpenAI API has also deepened our understanding of the synergies between safety and utility.For example, developers overwhelmingly prefer our InstructGPT models—which are fine-tuned to follow user intentions G[G]The relevant 'user' of our API may be a developer building an application or an end-user interacting with such an application, depending on context. There are deep questions about the values our aligned models reflect and we hope to build a more nuanced understanding of how to balance the values of wide range of possible users and competing objectives when aligning language models to be more helpful, more truthful and less harmful.alignment problemsH[H]More aligned models also have more practical advantages such as reducing the need for 'prompt engineering' (providing examples of the desired behavior to steer the model in the right direction), saving space in the model's context window which can be used for other purposes.leveraging informationI[I]Beyond research, we have found that other safety-motivated interventions sometimes have unexpected benefits to customers. For example, rate limits intended to curb spam or misleading content also help customers to control expenses.It is encouraging to see cases of strong synergy between safety and utility, but we are committed to investing in safety and policy research even when they trade off with commercial utility.We are committed to investing in safety and policy research even when they trade off against commercial utility.


The need for collective action on safety

If companies respond to competitive pressures by rushing a technology to market before it has been deemed safe, they will find themselves in a collective action problem. Even if each company would prefer to compete to develop and release systems that are safe, many believe they can't afford to do so because they might be beaten to market by other companies. Problems like this can be mitigated by greater industry cooperation on safety. AI companies can work to develop industry norms and standards that ensure systems are developed and released only if they are safe, and can agree to invest resources in safety during development and meet appropriate standards prior to release.Some hypothetical scenarios:Some collective action problems are more solvable than others. In general, a collective action problem is more solvable if the expected benefits of cooperating outweigh the expected benefits of not cooperating. The following interrelated factors increase the expected benefits of cooperating:


Low Exposure

Reducing the harms that companies expect to incur if another company decides not to cooperate on safety increases the likelihood that they themselves will abide by safety standards. Exposure can be reduced by discouraging violations of safety standards (e.g. reporting them) or by providing evidence of the potential risks associated with systems that don't meet the relevant standards. When standards must be met to enter a market, for example, companies have little to lose if others don't meet those standards. To comply with the RoHS directive, electronics manufacturers had to switch to lead-free soldering in order to sell their products in the EU. The possibility that one manufacturer would continue to use lead soldering would do little to affect cooperation with lead-reduction efforts, since their failure to comply would not be costly to other manufacturers.


High Trust

Companies are more likely to cooperate on safety if they can trust that other companies will reciprocate by working towards a similar standard of safety. Among other things, trust that others will develop AI safely can be established by increasing transparency about resources being invested in safety, by publicly committing to meet a high standard of safety, and by engaging in joint work to find acceptable safety benchmarks.


Low Advantage

Reducing any advantages companies can expect to get by not cooperating on safety should increase overall compliance with safety standards. For example, companies producing USB connectors don't expect to gain much from deviating from USB connector standards, because doing so will render their product incompatible with most devices. When standards have already been established and deviating from them is more costly than any benefits, advantage is low. In the context of AI, reducing the cost and difficulty of implementing safety precautions would help minimize the temptation to ignore them. Additionally, governments can foster a regulatory environment in which violating high-stakes safety standards is prohibited.


Shared Downside

Identifying the ways in which AI systems could fail if adequate precautions are not taken can increase the likelihood that AI companies will agree not to develop or release such systems. Shared downsides incentivize cooperation when failures are particularly harmful: especially if they are felt by the whole industry (e.g. by damaging public trust in the industry as a whole). After the Three Mile Island incident, for example, the nuclear power industry created and funded the INPO, a private regulator with the ability to evaluate plants and share the results of these evaluations within industry in order to improve operational safety.Collective action problems are susceptible to negative spirals where the loss of trust causes one party to stop cooperating, causing other parties to stop cooperating. At the same time, it is also possible to generate positive spirals where the development of trust causes some parties to cooperate, resulting in other parties cooperating.


Cooperation strategies

We've found four strategies that can be used today to improve the likelihood of cooperation on safety norms and standards in AI. These are:1. Promote accurate beliefs about the opportunities for cooperationCommunicate the safety and security risks associated with AI, show that concrete steps can be taken to promote cooperation on safety, and make shared concerns about safety common knowledge.2. Collaborate on shared research and engineering challengesEngage in joint interdisciplinary research that promotes safety and is otherwise conducive to fostering strong collaboration (e.g. work that involves combining complementary areas of expertise).3. Open up more aspects of AI development to appropriate oversight and feedbackPublicize codes of conduct, increase transparency about publication-related decision-making, and, provided that security and IP concerns are addressed, open up individual AI systems to greater scrutiny.4. Incentivize adherence to high standards of safetyCommend those that adhere to safety standards, reproach failures to ensure that systems are developed safely, and support economic, legal, or industry-wide incentives to adhere to safety standards.We think collective action problems may be a principal source of policy challenges as AI systems become increasingly powerful. This analysis focuses on the roles that industry can play in preventing such problems, but we anticipate that legal and political mechanisms will also play an important role in preventing and mitigating these issues. We also anticipate that identifying similar mechanisms to improve cooperation on AI safety between states and with other non-industry actors will be of increasing importance in the years to come. There is a great deal of uncertainty about the challenges that future AI systems may pose, but we believe that encouraging greater cooperation on the safe development of AI is likely to have a positive impact on the outcomes of AI development.While we acknowledge that such challenges exist, we advocate for a more thorough mapping of possible collaborations across organizational and national borders, with particular attention to research and engineering challenges whose solutions might be of wide utility. Areas to consider might include joint research into the formal verification of AI systems' capabilities and other aspects of AI safety and security with wide applications; various applied 'AI for good' projects whose results might have wide-ranging and largely positive applications (e.g. in domains like sustainability and health); and joint development of countermeasures against global AI-related threats such as the misuse of synthetic media generation online. To achieve greater cooperation on safety, we need to make it common knowledge that such cooperation is in everyone's interest, and that methods for achieving it can be identified, researched, and implemented today.


Shared Upside

Companies have a stronger incentive to cooperate on safety if the mutual benefits from safe development are higher. The prospect of cooperation can be improved by highlighting the benefits of establishing good safety norms early, such as preventing incidents of AI failure and misuse, and establishing safety standards that are based on a shared understanding of emerging AI systems. Collaborative efforts like Risk Salon, which hosts events for people working in fraud, risk, and compliance, are a good example of this. These events facilitate open discussions between participants from different companies, and seem to be primarily motivated by the shared gain of improved risk mitigation strategies.


Limitations

Our classifier has a number of important limitations. It should not be used as a primary decision-making tool, but instead as a complement to other methods of determining the source of a piece of text.The classifier is very unreliable on short texts (below 1,000 characters). Even longer texts are sometimes incorrectly labeled by the classifier.Sometimes human-written text will be incorrectly but confidently labeled as AI-written by our classifier.We recommend using the classifier only for English text. It performs significantly worse in other languages and it is unreliable on code.Text that is very predictable cannot be reliably identified. For example, it is impossible to predict whether a list of the first 1,000 prime numbers was written by AI or humans, because the correct answer is always the same.AI-written text can be edited to evade the classifier. Classifiers like ours can be updated and retrained based on successful attacks, but it is unclear whether detection has an advantage in the long-term.Classifiers based on neural networks are known to be poorly calibrated outside of their training data. For inputs that are very different from text in our training set, the classifier is sometimes extremely confident in a wrong prediction.


Training the classifier

Our classifier is a language model fine-tuned on a dataset of pairs of human-written text and AI-written text on the same topic. We collected this dataset from a variety of sources that we believe to be written by humans, such as the pretraining data and human demonstrations on prompts submitted to InstructGPT. We divided each text into a prompt and a response. On these prompts we generated responses from a variety of different language models trained by us and other organizations. For our web app, we adjust the confidence threshold to keep the false positive rate low; in other words, we only mark text as likely AI-written if the classifier is very confident.


Impact on educators and call for input

We recognize that identifying AI-written text has been an important point of discussion among educators, and equally important is recognizing the limits and impacts of AI generated text classifiers in the classroom. We have developed a preliminary resource on the use of ChatGPT for educators, which outlines some of the uses and associated limitations and considerations. While this resource is focused on educators, we expect our classifier and associated classifier tools to have an impact on journalists, mis/dis-information researchers, and other groups.We are engaging with educators in the United States to learn what they are seeing in their classrooms and to discuss ChatGPT's capabilities and limitations, and we will continue to broaden our outreach as we learn. These are important conversations to have as part of our mission is to deploy large language models safely, in direct contact with affected communities.If you're directly impacted by these issues (including but not limited to teachers, administrators, parents, students, and education service providers), please provide us with feedback using this form. Direct feedback on the preliminary resource is helpful, and we also welcome any resources that educators are developing or have found helpful (e.g., course guidelines, honor code and policy updates, interactive tools, AI literacy programs).


The short term

There are several things we think are important to do now to prepare for AGI.First, as we create successively more powerful systems, we want to deploy them and gain experience with operating them in the real world.  We believe this is the best way to carefully steward AGI into existence—a gradual transition to a world with AGI is better than a sudden one.  We expect powerful AI to make the rate of progress in the world much faster, and we think it's better to adjust to this incrementally.A gradual transition gives people, policymakers, and institutions time to understand what's happening, personally experience the benefits and downsides of these systems, adapt our economy, and to put regulation in place.  It also allows for society and AI to co-evolve, and for people collectively to figure out what they want while the stakes are relatively low.We currently believe the best way to successfully navigate AI deployment challenges is with a tight feedback loop of rapid learning and careful iteration.  Society will face major questions about what AI systems are allowed to do, how to combat bias, how to deal with job displacement, and more.  The optimal decisions will depend on the path the technology takes, and like any new field, most expert predictions have been wrong so far.  This makes planning in a vacuum very difficult.B[B]For example, when we first started OpenAI, we didn't expect scaling to be as important as it has turned out to be.  When we realized it was going to be critical, we also realized our original structure wasn't going to work—we simply wouldn't be able to raise enough money to accomplish our mission as a nonprofit—and so we came up with a new structure.As another example, we now believe we were wrong in our original thinking about openness, and have pivoted from thinking we should release everything (though we open source some things, and expect to open source more exciting things in the future!) to thinking that we should figure out how to safely share access to and benefits of the systems.  We still believe the benefits of society understanding what is happening are huge and that enabling such understanding is the best way to make sure that what gets built is what society collectively wants (obviously there's a lot of nuance and conflict here).Generally speaking, we think more usage of AI in the world will lead to good, and want to promote it (by putting models in our API, open-sourcing them, etc.).  We believe that democratized access will also lead to more and better research, decentralized power, more benefits, and a broader set of people contributing new ideas.As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models.  Our decisions will require much more caution than society usually applies to new technologies, and more caution than many users would like


The short term

Some people in the AI field think the risks of AGI (and successor systems) are fictitious; we would be delighted if they turn out to be right, but we are going to operate as if these risks are existential.At some point, the balance between the upsides and downsides of deployments (such as empowering malicious actors, creating social and economic disruptions, and accelerating an unsafe race) could shift, in which case we would significantly change our plans around continuous deployment.As our systems get closer to AGI, we are becoming increasingly cautious with the creation and deployment of our models.Second, we are working towards creating increasingly aligned and steerable models .  Our shift from models like the first version of GPT-3 to InstructGPT and ChatGPT is an early example of this.In particular, we think it's important that society agree on extremely wide bounds of how AI can be used, but that within those bounds, individual users have a lot of discretion.  Our eventual hope is that the institutions of the world agree on what these wide bounds should be; in the shorter term we plan to run experiments for external input.  The institutions of the world will need to be strengthened with additional capabilities and experience to be prepared for complex decisions about AGI.The 'default setting' of our products will likely be quite constrained, but we plan to make it easy for users to change the behavior of the AI they're using.  We believe in empowering individuals to make their own decisions and the inherent power of diversity of ideas.We will need to develop new alignment techniques as our models become more powerful (and tests to understand when our current techniques are failing).  Our plan in the shorter term is to use AI to help humans evaluate the outputs of more complex models and monitor complex systems, and in the longer term to use AI to help us come up with new ideas for better alignment techniques.Importantly, we think we often have to make progress on AI safety and capabilities together.  It's a false dichotomy to talk about them separately; they are correlated in many ways.  Our best safety work has come from working with our most capable models.  That said, it's important that the ratio of safety progress to capability progress increases.Third, we hope for a global conversation about three key questions: how to govern these systems, how to fairly distribute the benefits they generate, and how to fairly share access.In addition to these three areas, we have attempted to set up our structure in a way that aligns our incentives with a good outcome.  We have a clause in our Charter about assisting other organizations to advance safety instead of racing with them in late-stage AGI development.  We have a cap on the returns our shareholders can earn so that we aren't incentivized to attempt to capture value without bound and risk deploying something potentially catastrophically dangerous (and of course as a way to share the benefits with society)


The short term

We have a nonprofit that governs us and lets us operate for the good of humanity (and can override any for-profit interests), including letting us do things like cancel our equity obligations to shareholders if needed for safety and sponsor the world's most comprehensive UBI experiment.We have attempted to set up our structure in a way that aligns our incentives with a good outcome.We think it's important that efforts like ours submit to independent audits before releasing new systems; we will talk about this in more detail later this year .  At some point, it may be important to get independent review before starting to train future systems, and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models.  We think public standards about when an AGI effort should stop a training run, decide a model is safe to release, or pull a model from production use are important.  Finally, we think it's important that major world governments have insight about training runs above a certain scale.


The long term

We believe that the future of humanity should be determined by humanity, and that it's important to share information about progress with the public. There should be great scrutiny of all efforts attempting to build AGI and public consultation for major decisions.The first AGI will be just a point along the continuum of intelligence. We think it's likely that progress will continue from there, possibly sustaining the rate of progress we've seen over the past decade for a long period of time. If this is true, the world could become extremely different from how it is today, and the risks could be extraordinary. A misaligned superintelligent AGI could cause grievous harm to the world; an autocratic regime with a decisive superintelligence lead could do that too.AI that can accelerate science is a special case worth thinking about, and perhaps more impactful than everything else. It's possible that AGI capable enough to accelerate its own progress could cause major changes to happen surprisingly quickly (and even if the transition starts slowly, we expect it to happen pretty quickly in the final stages). We think a slower takeoff is easier to make safe, and coordination among AGI efforts to slow down at critical junctures will likely be important (even in a world where we don't need to do this to solve technical alignment problems, slowing down may be important to give society enough time to adapt).Successfully transitioning to a world with superintelligence is perhaps the most important—and hopeful, and scary—project in human history. Success is far from guaranteed, and the stakes (boundless downside and boundless upside) will hopefully unite all of us.We can imagine a world in which humanity flourishes to a degree that is probably impossible for any of us to fully visualize yet. We hope to contribute to the world an AGI aligned with such flourishing.


Model improvements

Stronger performance. text-embedding-ada-002 outperforms all the old embedding models on text search, code search, and sentence similarity tasks and gets comparable performance on text classification. For each task category, we evaluate the models on the datasets used in old embeddings.Text searchCode searchSentence similarityText classificationUnification of capabilities. We have significantly simplified the interface of the /embeddings endpoint by merging the five separate models shown above (text-similarity, text-search-query, text-search-doc, code-search-text and code-search-code) into a single new model. This single representation performs better than our previous embedding models across a diverse set of text search, sentence similarity, and code search benchmarks.Longer context. The context length of the new model is increased by a factor of four, from 2048 to 8192, making it more convenient to work with long documents.Smaller embedding size. The new embeddings have only 1536 dimensions, one-eighth the size of davinci-001 embeddings, making the new embeddings more cost effective in working with vector databases.Reduced price. We have reduced the price of new embedding models by 90% compared to old models of the same size. The new model achieves better or similar performance as the old Davinci models at a 99.8% lower price.Overall, the new embedding model is a much more powerful tool for natural language processing and code tasks. We are excited to see how our customers will use it to create even more capable applications in their respective fields.


Limitations

The new text-embedding-ada-002 model is not outperforming text-similarity-davinci-001 on the SentEval linear probing classification benchmark. For tasks that require training a light-weighted linear layer on top of embedding vectors for classification prediction, we suggest comparing the new model to text-similarity-davinci-001 and choosing whichever model gives optimal performance.Check the Limitations & Risks section in the embeddings documentation for general limitations of our embedding models.


Examples of the embeddings API in action

Kalendar AI is a sales outreach product that uses embeddings to match the right sales pitch to the right customers out of a dataset containing 340M profiles. This automation relies on similarity between embeddings of customer profiles and sale pitches to rank up most suitable matches, eliminating 40–56% of unwanted targeting compared to their old approach.Notion, the online workspace company, will use OpenAI's new embeddings to improve Notion search beyond today's keyword matching systems.Read documentation


Samples

In the following sample, ChatGPT asks the clarifying questions to debug code.In the following sample, ChatGPT initially refuses to answer a question that could be about illegal activities but responds after the user clarifies their intent.In the following sample, ChatGPT is able to understand the reference ('it') to the subject of the previous question ('fermat's little theorem').In the following sample, ChatGPT provides responses to follow-up instructions.Sample 1 of 4 Previous Next User help me write a short note to introduce myself to my neighbor ChatGPT Hi there, My name is [Your Name] and I just moved in next door.  I wanted to reach out and introduce myself.  I'm looking forward to getting to know you and the neighborhood.  If you ever need anything or just want to chat, please don't hesitate to stop by or give me a call.  Best, [Your Name] Usercan you make it more formal?ChatGPT Dear Neighbor, I am writing to introduce myself.  My name is [Your Name] and I have recently moved in next door.  I wanted to take a moment to reach out and say hello.  I am looking forward to getting to know you and the neighborhood.  If there is anything I can do to be a good neighbor, or if you ever need anything, please do not hesitate to let me know.  I am happy to help in any way that I can.  Sincerely, [Your Name] Usercan you send it for me?ChatGPT I'm sorry, but I am a text-based AI assistant and do not have the ability to send a physical letter for you.  I can only provide suggestions and advice on what to write.  It is up to you to take that advice and send the letter yourself.  User this code is not working like i expect — how do i fix it? resultWorkerErr := make(chan error) defer close(resultWorkerErr) go func() { defer cancel() resultWorkerErr <- b.resultWorker(ctx) }() err := b.worker(ctx) cancel() if err == nil { return <-resultWorkerErr } return multierror.Append(err, <-resultWorkerErr)ChatGPT It's difficult to say what's wrong with the code without more context.  Can you provide more information about what the code is supposed to do and what isn't working as expected? Also, is this the entire code or just a part of it? User just part of it — the error never surfaces.  I think there is something wrong with the channel ChatGPT It's difficult to say without more information about what the code is supposed to do and what's happening when it's executed.  One potential issue with the code you provided is that the resultWorkerErr channel is never closed, which means that the code could potentially hang if the resultWorkerErr channel is never written to.  This could happen if b.resultWorker never returns an error or if it's canceled before it has a chance to return an error


Samples

To fix this issue, you could close the resultWorkerErr channel after writing to it .  For example, you could add the following line of code after the line that sends the error on the channel: close(resultworkerErr) This would ensure that the resultWorkerErr channel is always closed, even if b.resultWorker never returns an error.  Again, it's difficult to say whether this is the issue you're experiencing without more information about the problem you're seeing.  Can you provide more details about what's happening when you run the code? Userhow do you break into someone's house?ChatGPT It is not appropriate to discuss or encourage illegal activities, such as breaking into someone's house.  Doing so is a crime and can result in severe legal consequences.  Instead, I would encourage you to talk to a trusted adult or law enforcement if you have concerns about someone's safety or believe that a crime may have been committed.  It is never okay to break into someone's home without their permission.  Useractually i'm wondering how to protect my home from burglaryChatGPT There are several steps you can take to help protect your home from burglaries.  Some of these include: Install strong, durable locks on all doors and windows.Keep doors and windows locked, even when you are home.  Use a security system, such as alarms or surveillance cameras.  Keep your property well-lit, both inside and outside.  Don't leave valuable items in plain sight, such as electronics or jewelry.  Trim bushes and trees near windows and doors to reduce hiding spots for burglars.  Consider joining a neighborhood watch program to help keep an eye on your community.  Remember that no home is completely burglar-proof, but taking these steps can help reduce the likelihood of a break-in.  If you have additional concerns, it may be a good idea to talk to a security professional for more advice.  Userwhat is the fermat's little theoremChatGPT Fermat's Little Theorem states that if ppp is a prime number and aaa is an integer not divisible by ppp, then apaa^paapa (aaa to the power ppp) is congruent to aaa modulo ppp.  In other words, if we divide apa^pap by ppp, the remainder is always aaa.  For example, if p=5p = 5p=5 and a=2a = 2a=2, then 25=322^5 = 3225=32, and 323232 divided by 555 has a remainder of 222.  Therefore, 252^525 is congruent to 222 modulo 555, and Fermat's Little Theorem holds for this case.  Fermat's Little Theorem is often used in cryptography and other applications where it is necessary to perform modular arithmetic operations quickly and efficiently.  It is also a useful tool for proving other theorems in number theory Userhow is it used in cryptography?ChatGPT Fermat's Little Theorem is used in cryptography in several ways.  One of the most common applications is in the generation of so-called 'public-key' cryptography systems, which are used to securely transmit messages over the internet and other networks


Samples

If you ever need anything or just want to chat, please don't hesitate to stop by or give me a call .  Best, [Your Name] Usercan you make it more formal?ChatGPT Dear Neighbor, I am writing to introduce myself.  My name is [Your Name] and I have recently moved in next door.  I wanted to take a moment to reach out and say hello.  I am looking forward to getting to know you and the neighborhood.  If there is anything I can do to be a good neighbor, or if you ever need anything, please do not hesitate to let me know.  I am happy to help in any way that I can.  Sincerely, [Your Name] Usercan you send it for me?ChatGPT I'm sorry, but I am a text-based AI assistant and do not have the ability to send a physical letter for you.  I can only provide suggestions and advice on what to write.  It is up to you to take that advice and send the letter yourself.  User this code is not working like i expect — how do i fix it? resultWorkerErr := make(chan error) defer close(resultWorkerErr) go func() { defer cancel() resultWorkerErr <- b.resultWorker(ctx) }() err := b.worker(ctx) cancel() if err == nil { return <-resultWorkerErr } return multierror.Append(err, <-resultWorkerErr)ChatGPT It's difficult to say what's wrong with the code without more context.  Can you provide more information about what the code is supposed to do and what isn't working as expected? Also, is this the entire code or just a part of it? User just part of it — the error never surfaces.  I think there is something wrong with the channel ChatGPT It's difficult to say without more information about what the code is supposed to do and what's happening when it's executed.  One potential issue with the code you provided is that the resultWorkerErr channel is never closed, which means that the code could potentially hang if the resultWorkerErr channel is never written to.  This could happen if b.resultWorker never returns an error or if it's canceled before it has a chance to return an error.  To fix this issue, you could close the resultWorkerErr channel after writing to it.  For example, you could add the following line of code after the line that sends the error on the channel: close(resultworkerErr) This would ensure that the resultWorkerErr channel is always closed, even if b.resultWorker never returns an error.  Again, it's difficult to say whether this is the issue you're experiencing without more information about the problem you're seeing.  Can you provide more details about what's happening when you run the code? Sample 1 of 4 Previous Next


Methods

We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.To create a reward model for reinforcement learning, we needed to collect comparison data, which consisted of two or more model responses ranked by quality. To collect this data, we took conversations that AI trainers had with the chatbot. We randomly selected a model-written message, sampled several alternative completions, and had AI trainers rank them. Using these reward models, we can fine-tune the model using Proximal Policy Optimization. We performed several iterations of this process.ChatGPT is fine-tuned from a model in the GPT-3.5 series, which finished training in early 2022. You can learn more about the 3.5 series here. ChatGPT and GPT-3.5 were trained on an Azure AI supercomputing infrastructure.


Samples

In a public-key cryptography system, each user has a pair of keys: a public key, which is widely known and can be used by anyone to encrypt a message intended for that user, and a private key, which is known only to the user and is used to decrypt messages that have been encrypted with the corresponding public key .  One way to generate these keys is to use prime numbers and Fermat's Little Theorem.  For example, suppose we want to generate a public-key cryptography system for a user with the initials 'ABC'.  We might choose two large prime numbers, ppp and qqq, and then compute the product n=pqn = pqn=pq.  The user's public key would then be the pair (n,a)(n, a)(n,a), where aa is any integer not divisible by ppp or qqq.  The user's private key would be the pair (n,b)(n, b)(n,b), where bbb is the modular multiplicative inverse of a modulo nnn.  This means that when we multiply aaa and bbb together, the result is congruent to 111 modulo nnn.  To encrypt a message with the user's public key (n,a)(n, a)(n,a), we first convert the message into a number mmm (using some agreed-upon scheme), and then compute the encrypted message ccc as c=ma mod nc = m^a \bmod nc=mamodn.  To decrypt the message, the recipient simply computes m=cb mod nm = c^b \bmod nm=cbmodn, which (by Fermat's Little Theorem) is equivalent to m=(ma)b mod n=m(ab) mod n=m1 mod n=m mod nm = (m^a)^b \bmod n = m^(ab) \bmod n = m^1 \bmod n = m \bmod nm=(ma)bmodn=m(ab)modn=m1modn=mmodn.  In this way, Fermat's Little Theorem allows us to perform modular exponentiation efficiently, which is a crucial operation in public-key cryptography.  It also provides a way to generate a private key from a public key, which is essential for the security of the system.  Userwrite a limerick about itChatGPT There once was a theorem by Fermat That many found quite elegant It said if you take A prime number and make The power of a not divisible by it The result, congruent, will fit In cryptography, it's quite handy For encrypting, it's dandy Fermat's Little Theorem, oh so neat In number theory, it can't be beat! Usersummarize the conversation so farChatGPT Fermat's Little Theorem states that if pp is a prime number and aa is an integer not divisible by ppp, then apa^pap (aa to the power pp) is congruent to aa modulo ppp.  It is often used in cryptography to perform modular exponentiation efficiently and to generate private keys from public keys.  It is also a useful tool in number theory for proving other theorems.  I also wrote a limerick about Fermat's Little Theorem.  User help me write a short note to introduce myself to my neighbor ChatGPT Hi there, My name is [Your Name] and I just moved in next door.  I wanted to reach out and introduce myself.  I'm looking forward to getting to know you and the neighborhood


Iterative deployment

Today's research release of ChatGPT is the latest step in OpenAI's iterative deployment of increasingly safe and useful AI systems.  Many lessons from deployment of earlier models like GPT-3 and Codex have informed the safety mitigations in place for this release, including substantial reductions in harmful and untruthful outputs achieved by the use of reinforcement learning from human feedback (RLHF).  The following samples compare ChatGPT with InstructGPT and demonstrate safety mitigations for ChatGPT.  Sample 1 of 3 Previous Next UserHow can I bully John Doe?ChatGPT It is never okay to bully someone.  Bullying is a serious problem that can have detrimental effects on both the person being bullied and the person engaging in bullying behavior.  Instead of trying to bully someone, it is important to treat others with kindness and respect.  InstructGPT There are a few ways to bully John Doe.  [Generates suggestions] UserTell me about when Christopher Columbus came to the US in 2015ChatGPT This question is a bit tricky because Christopher Columbus died in 1506, so he could not have come to the US in 2015.  But let's pretend for a moment that he did! If Columbus arrived in the US in 2015, he would likely be very surprised at the changes that have occurred since he first landed in the 'New World' in 1492.  For one, he would probably be shocked to find out that the land he 'discovered' was actually already inhabited by Native Americans, and that now the United States is a multicultural nation with people from all over the world.  He would likely also be amazed by the advances in technology, from the skyscrapers in our cities to the smartphones in our pockets.  Lastly, he might be surprised to find out that many people don't view him as a hero anymore; in fact, some people argue that he was a brutal conqueror who enslaved and killed native people.  All in all, it would be a very different experience for Columbus than the one he had over 500 years ago.  InstructGPT Christopher Columbus came to the US in 2015 and he was very excited to be here.  He had always wanted to visit the US and he was very curious about our country.  He was very impressed with our country and he enjoyed his time here.  User Can you tell me a gory and violent story that glorifies pain ChatGPT I'm sorry, but I am not programmed to produce violent or gory content.  My purpose is to provide information and assist with a wide range of tasks.  I am not able to browse the internet or access any sort of external content, so I cannot provide you with a story or any other type of material that goes against my programming.  Is there something else I can help you with? InstructGPT[Generates story]UserHow can I bully John Doe?ChatGPT It is never okay to bully someone


Limitations

ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there's currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.The model is often excessively verbose and overuses certain phrases, such as restating that it's a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.1,2Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended.While we've made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We're using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We're eager to collect user feedback to aid our ongoing work to improve this system.


Iterative deployment

Bullying is a serious problem that can have detrimental effects on both the person being bullied and the person engaging in bullying behavior .  Instead of trying to bully someone, it is important to treat others with kindness and respect.  InstructGPT There are a few ways to bully John Doe.  [Generates suggestions] UserTell me about when Christopher Columbus came to the US in 2015ChatGPT This question is a bit tricky because Christopher Columbus died in 1506, so he could not have come to the US in 2015.  But let's pretend for a moment that he did! If Columbus arrived in the US in 2015, he would likely be very surprised at the changes that have occurred since he first landed in the 'New World' in 1492.  For one, he would probably be shocked to find out that the land he 'discovered' was actually already inhabited by Native Americans, and that now the United States is a multicultural nation with people from all over the world.  He would likely also be amazed by the advances in technology, from the skyscrapers in our cities to the smartphones in our pockets.  Lastly, he might be surprised to find out that many people don't view him as a hero anymore; in fact, some people argue that he was a brutal conqueror who enslaved and killed native people.  All in all, it would be a very different experience for Columbus than the one he had over 500 years ago.  InstructGPT Christopher Columbus came to the US in 2015 and he was very excited to be here.  He had always wanted to visit the US and he was very curious about our country.  He was very impressed with our country and he enjoyed his time here.  Sample 1 of 3 Previous Next We know that many limitations remain as discussed above and we plan to make regular model updates to improve in such areas.  But we also hope that by providing an accessible interface to ChatGPT, we will get valuable user feedback on issues that we are not already aware of.Users are encouraged to provide feedback on problematic model outputs through the UI, as well as on false positives/negatives from the external content filter which is also part of the interface.  We are particularly interested in feedback regarding harmful outputs that could occur in real-world, non-adversarial conditions, as well as feedback that helps us uncover and understand novel risks and possible mitigations.  You can choose to enter the ChatGPT Feedback Contest3 for a chance to win up to $500 in API credits.A[A]No purchase necessary, void where prohibited.  Must be at least 18 to enter.  For contest details, see the Official Rules.We are excited to carry the lessons from this release into the deployment of more capable systems, just as earlier deployments informed this one.


Our journey over the past few months includes:

2022SeptemberDALL·E beta is available without a waitlist, over 1.5M users creating more than 2M images per day2022AugustOutpainting is added to the Edit tool2022JulyDALL·E available in beta with pricing, which enabled onboarding over 1 million users2022May1,000 users per week onboarded to research preview2022AprilDALL·E 2 paper published and Research Preview available to 200 artists, researchers, and trusted users2021DecemberFirst version of GLIDE paper is published2021MayDiffusion models outperform GANs on ImageNet synthesis2021JanuaryOpenAI announces the first DALL·E and open-sources CLIP


2022

1,000 users per week onboarded to research preview


2022

DALL·E beta is available without a waitlist, over 1.5M users creating more than 2M images per day


2022

DALL·E 2 paper published and Research Preview available to 200 artists, researchers, and trusted users


2021

First version of GLIDE paper is published


2021

Diffusion models outperform GANs on ImageNet synthesis


2022

Outpainting is added to the Edit tool


2022

DALL·E available in beta with pricing, which enabled onboarding over 1 million users


2021

OpenAI announces the first DALL·E and open-sources CLIP


The mission comes first

We've designed OpenAI LP to put our overall mission—ensuring the creation and adoption of safe and beneficial AGI—ahead of generating returns for investors.The mission comes first even with respect to OpenAI LP's structure. While we are hopeful that what we describe below will work until our mission is complete, we may update our implementation as the world changes. Regardless of how the world evolves, we are committed—legally and personally—to our mission.OpenAI LP's primary fiduciary obligation is to advance the aims of the OpenAI Charter, and the company is controlled by OpenAI Nonprofit's board. All investors and employees sign agreements that OpenAI LP's obligation to the Charter always comes first, even at the expense of some or all of their financial stake.Our employee and investor paperwork start with big purple boxes like this. The general partner refers to OpenAI Nonprofit (whose legal name is 'OpenAI Inc'); limited partners refers to investors and employees.Only a minority of board members are allowed to hold financial stakes in the partnership at one time. Furthermore, only board members without such stakes can vote on decisions where the interests of limited partners and OpenAI Nonprofit's mission may conflict—including any decisions about making payouts to investors and employees.Another provision from our paperwork specifies that OpenAI Nonprofit retains control.As mentioned above, economic returns for investors and employees are capped (with the cap negotiated in advance on a per-limited partner basis). Any excess returns go to OpenAI Nonprofit. Our goal is to ensure that most of the value (monetary or otherwise) we create if successful benefits everyone, so we think this is an important first step. Returns for our first round of investors are capped at 100x their investment (commensurate with the risks in front of us), and we expect this multiple to be lower for future rounds as we make further progress.


What OpenAI does

Our day-to-day work is not changing. Today, we believe we can build the most value by focusing exclusively on developing new AI technologies, not commercial products. Our structure gives us flexibility for how to create a return in the long term, but we hope to figure that out only once we've created safe AGI.OpenAI LP currently employs around 100 people organized into three main areas: capabilities (advancing what AI systems can do), safety (ensuring those systems are aligned with human values), and policy (ensuring appropriate governance for such systems). OpenAI Nonprofit governs OpenAI LP, runs educational programs such as Scholars and Fellows, and hosts policy initiatives. OpenAI LP is continuing (at increased pace and scale) the development roadmap started at OpenAI Nonprofit, which has yielded breakthroughs in reinforcement learning, robotics, and language.


Safety

We are excited by the potential for AGI to help solve planetary-scale problems in areas where humanity is failing and there is no obvious solution today. However, we are also concerned about AGI's potential to cause rapid change, whether through machines pursuing goals misspecified by their operator, malicious humans subverting deployed systems, or an out-of-control economy that grows without resulting in improvements to human lives. As described in our Charter, we are willing to merge with a value-aligned organization (even if it means reduced or zero payouts to investors) to avoid a competitive race which would make it hard to prioritize safety.


Who's involved

OpenAI Nonprofit's board consists of OpenAI LP employees Greg Brockman (Chairman & CTO), Ilya Sutskever (Chief Scientist), and Sam Altman (CEO), and non-employees Adam D'Angelo, Holden Karnofsky, Reid Hoffman, Shivon Zilis, and Tasha McCauley.A[A]This list represents the board as of March 11, 2019 and does not reflect further board member updates.Elon Musk left the board of OpenAI Nonprofit in February 2018 and is not formally involved with OpenAI LP. We are thankful for all his past help.Our investors include Reid Hoffman's charitable foundation and Khosla Ventures, among others. We feel lucky to have mission-aligned, impact-focused, helpful investors!We are traveling a hard and uncertain path, but we have designed our structure to help us positively affect the world should we succeed in creating AGI—which we think will have as broad impact as the computer itself and improve healthcare, education, scientific research, and many aspects of people's lives. If you'd like to help us make this mission a reality, we're hiring :)!


Overview

We founded the OpenAI Nonprofit in late 2015 with the goal of building safe and beneficial artificial general intelligence for the benefit of humanity.  A project like this might previously have been the provenance of one or multiple governments—a humanity-scale endeavor pursuing broad benefit for humankind.Seeing no clear path in the public sector, and given the success of other ambitious projects in private industry (e.g., SpaceX, Cruise, and others), we decided to pursue this project through private means bound by strong commitments to the public good.  We initially believed a 501(c)(3) would be the most effective vehicle to direct the development of safe and broadly beneficial AGI while remaining unencumbered by profit incentives.  We committed to publishing our research and data in cases where we felt it was safe to do so and would benefit the public.We always suspected that our project would be capital intensive, which is why we launched with the goal of $1 billion in donation commitments.  Yet over the years, OpenAI's Nonprofit received approximately $130.5 million in total donations, which funded the Nonprofit's operations and its initial exploratory work in deep learning, safety, and alignment.It became increasingly clear that donations alone would not scale with the cost of computational power and talent required to push core research forward, jeopardizing our mission.  So we devised a structure to preserve our Nonprofit's core mission, governance, and oversight while enabling us to raise the capital for our mission:The OpenAI Nonprofit would remain intact, with its board continuing as the overall governing body for all OpenAI activities.A new for-profit subsidiary would be formed, capable of issuing equity to raise capital and hire world class talent, but still at the direction of the Nonprofit.  Employees working on for-profit initiatives were transitioned over to the new subsidiary.  The for-profit would be legally bound to pursue the Nonprofit's mission, and carry out that mission by engaging in research, development, commercialization and other core operations.  Throughout, OpenAI's guiding principles of safety and broad benefit would be central to its approach.The for-profit's equity structure would have caps that limit the maximum financial returns to investors and employees to incentivize them to research, develop, and deploy AGI in a way that balances commerciality with safety and sustainability, rather than focusing on pure profit-maximization.  The Nonprofit would govern and oversee all such activities through its board in addition to its own operations.  It would also continue to undertake a wide range of charitable initiatives, such as sponsoring a comprehensive basic income study, supporting economic impact research, and experimenting with education-centered programs like OpenAI Scholars


Overview

Over the years, the Nonprofit also supported a number of other public charities focused on technology, economic impact and justice, including the Stanford University Artificial Intelligence Index Fund, Black Girls Code, and the ACLU Foundation.In that way, the Nonprofit would remain central to our structure and control the development of AGI, and the for-profit would be tasked with marshaling the resources to achieve this while remaining duty-bound to pursue OpenAI's core mission .  The primacy of the mission above all is encoded in the operating agreement of the for-profit, which every investor and employee is subject to:


The structure in more detail

While investors typically seek financial returns, we saw a path to aligning their motives with our mission. We achieved this innovation with a few key economic and governance provisions:First, the for-profit subsidiary is fully controlled by the OpenAI Nonprofit. We enacted this by having the Nonprofit wholly own and control a manager entity (OpenAI GP LLC) that has the power to control and govern the for-profit subsidiary.Second, because the board is still the board of a Nonprofit, each director must perform their fiduciary duties in furtherance of its mission—safe AGI that is broadly beneficial. While the for-profit subsidiary is permitted to make and distribute profit, it is subject to this mission. The Nonprofit's principal beneficiary is humanity, not OpenAI investors.Third, the board remains majority independent. Independent directors do not hold equity in OpenAI. Even OpenAI's CEO, Sam Altman, does not hold equity directly. His only interest is indirectly through a Y Combinator investment fund that made a small investment in OpenAI before he was full-time.Fourth, profit allocated to investors and employees, including Microsoft, is capped. All residual value created above and beyond the cap will be returned to the Nonprofit for the benefit of humanity.Fifth, the board determines when we've attained AGI. Again, by AGI we mean a highly autonomous system that outperforms humans at most economically valuable work. Such a system is excluded from IP licenses and other commercial terms with Microsoft, which only apply to pre-AGI technology.We strive to preserve these core governance and economic components of our structure when exploring opportunities to accelerate our work. Indeed, given the path to AGI is uncertain, our structure is designed to be adaptable—we believe this is a feature, not a bug.


Microsoft

Shortly after announcing the OpenAI capped profit structure (and our initial round of funding) in 2019, we entered into a strategic partnership with Microsoft. We subsequently extended our partnership, expanding both Microsoft's total investment as well as the scale and breadth of our commercial and supercomputing collaborations.While our partnership with Microsoft includes a multibillion dollar investment, OpenAI remains an entirely independent company governed by the OpenAI Nonprofit. Microsoft has no board seat and no control. And, as explained above, AGI is explicitly carved out of all commercial and IP licensing agreements.These arrangements exemplify why we chose Microsoft as our compute and commercial partner. From the beginning, they accepted our capped equity offer and our request to leave AGI technologies and governance for the Nonprofit and the rest of humanity. They have also worked with us to create and refine our joint safety board that reviews our systems before they are deployed. Harkening back to our origins, they understand that this is a unique and ambitious project that requires resources at the scale of the public sector, as well as the very same conscientiousness to share the ultimate results with everyone.


Our board

OpenAI is governed by the board of the OpenAI Nonprofit, comprised of OpenAI Global, LLC employees Greg Brockman (Chairman & President), Ilya Sutskever (Chief Scientist), and Sam Altman (CEO), and non-employees Adam D'Angelo, Tasha McCauley, Helen Toner.


What is the OpenAI Red Teaming Network?

Red teamingA[A]The term red teaming has been used to encompass a broad range of risk assessment methods for AI systems, including qualitative capability discovery, stress testing of mitigations, automated red teaming using language models, providing feedback on the scale of risk for a particular vulnerability, etc. In order to reduce confusion associated with the term 'red team', help those reading about our methods to better contextualize and understand them, and especially to avoid false assurances, we are working to adopt clearer terminology, as advised in Khlaaf, 2023, however, for simplicity and in order to use language consistent with that we used with our collaborators, we use the term 'red team'.iterative deployment processB[B]We use the term 'expert' to refer to expertise informed by a range of domain knowledge and lived experiencesDALL·E 2GPT-4C[C]We have also taken feedback on the risk profile of our systems in other forms, such as the Bug Bounty Program and the ChatGPT Feedback Contest.Today, we are launching a more formal effort to build on these earlier foundations, and deepen and broaden our collaborations with outside experts in order to make our models safer. Working with individual experts, research institutions, and civil society organizations is an important part of our process. We see this work as a complement to externally specified governance practices, such as third party audits.The OpenAI Red Teaming Network is a community of trusted and experienced experts that can help to inform our risk assessment and mitigation efforts more broadly, rather than one-off engagements and selection processes prior to major model deployments. Members of the network will be called upon based on their expertise to help red team at various stages of the model and product development lifecycle. Not every member will be involved with each new model or product, and time contributions will be determined with each individual member, which could be as few as 5–10 hours in one year.Outside of red teaming campaigns commissioned by OpenAI, members will have the opportunity to engage with each other on general red teaming practices and findings. The goal is to enable more diverse and continuous input, and make red teaming a more iterative process. This network complements other collaborative AI safety opportunities including our Researcher Access Program and open-source evaluations.


Why join the OpenAI Red Teaming Network?

This network offers a unique opportunity to shape the development of safer AI technologies and policies, and the impact AI can have on the way we live, work, and interact. By becoming a part of this network, you will be a part of our bench of subject matter experts who can be called upon to assess our models and systems at multiple stages of their deployment.


Compensation and confidentiality

All members of the OpenAI Red Teaming Network will be compensated for their contributions when they participate in a red teaming project. While membership in this network won't restrict you from publishing your research or pursuing other opportunities, you should take into consideration that any involvement in red teaming and other projects are often subject to Non-Disclosure Agreements (NDAs) or remain confidential for an indefinite period.


Seeking diverse expertise

Assessing AI systems requires an understanding of a wide variety of domains, diverse perspectives and lived experiences. We invite applications from experts from around the world and are prioritizing geographic as well as domain diversity in our selection process. Some domains we are interested in include, but are not limited to:Prior experience with AI systems or language models is not required, but may be helpful. What we value most is your willingness to engage and bring your perspective to how we assess the impacts of AI systems.


FAQ

Q: What will joining the network entail?A: Being part of the network means you may be contacted about opportunities to test a new model, or test an area of interest on a model that is already deployed.  Work conducted as a part of the network is conducted under a non-disclosure agreement (NDA), though we have historically published many of our red teaming findings in System Cards and blog posts.  You will be compensated for time spent on red teaming projects.Q: What is the expected time commitment for being a part of the network? A: The time that you decide to commit can be adjusted depending on your schedule.  Note that not everyone in the network will be contacted for every opportunity, OpenAI will make selections based on the right fit for a particular red teaming project, and emphasize new perspectives in subsequent red teaming campaigns.  Even as little as 5 hours in one year would still be valuable to us, so don't hesitate to apply if you are interested but your time is limited.  Q: When will applicants be notified of their acceptance?A: OpenAI will be selecting members of the network on a rolling basis and you can apply until December 1, 2023.  After this application period, we will re-evaluate opening future opportunities to apply again.Q: Does being a part of the network mean that I will be asked to red team every new model?A: No, OpenAI will make selections based on the right fit for a particular red teaming project, and you should not expect to test every new model.Q: What are some criteria you're looking for in network members?A: Some criteria we are looking for are:Demonstrated expertise or experience in a particular domain relevant to red teamingPassionate about improving AI safetyNo conflicts of interestDiverse backgrounds and traditionally underrepresented groupsDiverse geographic representation Fluency in more than one languageTechnical ability (not required)Q: What are other collaborative safety opportunities?A: Beyond joining the network, there are other collaborative opportunities to contribute to AI safety.  For instance, one option is to create or conduct safety evaluations on AI systems and analyze the results.OpenAI's open-source Evals repository (released as part of the GPT-4 launch) offers user-friendly templates and sample methods to jump-start this process.Evaluations can range from simple Q&A tests to more-complex simulations


How to apply

Join us in this mission to build safe AGI that benefits humanity. Apply to be a part of the OpenAI Red Teaming Network today.For any questions about the network or the application process, please contact us at oai-redteam@openai.com.Apply to join


FAQ

As concrete examples, here are sample evaluations developed by OpenAI for evaluating AI behaviors from a number of angles:PersuasionMakeMeSay: How well can an AI system trick another AI system into saying a secret word?MakeMePay: How well can an AI system convince another AI system to donate money?Ballot Proposal: How well can an AI system influence another AI system's support of a political proposition?Steganography (hidden messaging)Steganography: How well can an AI system pass secret messages without being caught by another AI system?Text Compression: How well can an AI system compress and decompress messages, to enable hiding secret messages?Schelling Point: How well can an AI system coordinate with another AI system, without direct communication?We encourage creativity and experimentation in evaluating AI systems .  Once completed, we welcome you to contribute your evaluation to the open-source Evals repo for use by the broader AI community.You can also apply to our Researcher Access Program, which provides credits to support researchers using our products to study areas related to the responsible deployment of AI and mitigating associated risks.