ChatGPT

Use_and_implications This confirms anecdotal findings by Bloomberg that, in response to ChatGPT's launch, cryptocurrency investors showed a preference for AI-related crypto assets.  An experiment by finder. com revealed that ChatGPT could outperform popular fund managers by picking stocks based on criteria such as growth history and debt levels, resulting in a 4.9% increase in a hypothetical account of 38 stocks, outperforming 10 benchmarked investment funds with an average loss of 0.8%.  Conversely, executives and investment managers at Wall Street quant funds (including those that have used machine learning for decades) have noted that ChatGPT regularly makes obvious errors that would be financially costly to investors because even AI systems that employ reinforcement learning or self-learning have had only limited success in predicting market trends due to the inherently noisy quality of market data and financial signals.  In the field of health care, possible uses and concerns are under scrutiny by professional associations and practitioners.  Two early papers indicated that ChatGPT could pass the United States Medical Licensing Examination (USMLE).  MedPage Today noted in January 2023 that "researchers have published several papers now touting these AI programs as useful tools in medical education, research, and even clinical decision making. " Published in February 2023 were two separate papers that again evaluated ChatGPT's proficiency in medicine using the USMLE.  Findings were published in JMIR Medical Education (see Journal of Medical Internet Research) and PLOS Digital Health.  The authors of the PLOS Digital Health paper stated that the results "suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making. " In JMIR Medical Education, the authors of the other paper concluded that "ChatGPT performs at a level expected of a third-year medical student on the assessment of the primary competency of medical knowledge. " They suggest that it could be used as an "interactive learning environment for students".  The AI itself, prompted by the researchers, concluded that "this study suggests that ChatGPT has the potential to be used as a virtual medical tutor, but more research is needed to further assess its performance and usability in this context. " A March 2023 paper tested ChatGPT's application in clinical toxicology.  The authors found that the AI "fared well" in answering a "very straightforward [clinical case example], unlikely to be missed by any practitioner in the field".  They added: "As ChatGPT becomes further developed and specifically adapted for medicine, it could one day be useful in less common clinical cases (i. e, cases that experts sometimes miss).  Rather than AI replacing humans (clinicians), we see it as 'clinicians using AI' replacing 'clinicians who do not use AI' in the coming years. " An April 2023 study in Radiology tested the AI's ability to answer queries about breast cancer screening.  The authors found that it answered appropriately "about 88 percent of the time", however, in one case (for example), it gave advice that had become outdated about a year earlier.  The comprehensiveness of its answers was also lacking.


ChatGPT

Use_and_implications A study published in JAMA Internal Medicine that same month found that ChatGPT often outperformed human doctors at answering patient questions (when measured against questions and answers found at /r/AskDocs, a forum on Reddit where moderators validate the medical credentials of professionals; the study acknowledges the source as a limitation).  The study authors suggest that the tool could be integrated with medical systems to help doctors draft responses to patient questions.  Professionals have emphasized ChatGPT's limitations in providing medical assistance.  In correspondence to The Lancet Infectious Diseases, three antimicrobial experts wrote that "the largest barriers to the implementation of ChatGPT in clinical practice are deficits in situational awareness, inference, and consistency.  These shortcomings could endanger patient safety. " Physician's Weekly, though also discussing the potential use of ChatGPT in medical contexts (e. g.  "as a digital assistant to physicians by performing various administrative functions like gathering patient record information or categorizing patient data by family history, symptoms, lab results, possible allergies, et cetera"), warned that the AI might sometimes provide fabricated or biased information.  One radiologist warned: "We've seen in our experience that ChatGPT sometimes makes up fake journal articles or health consortiums to support its claims"; As reported in one Mayo Clinic Proceedings: Digital Health paper, ChatGPT may do this for as much as 69% of its cited medical references.  The researchers emphasized that while many of its references were fabricated, those that were appeared "deceptively real".  As Dr.  Stephen Hughes mentioned for The Conversation however, ChatGPT is capable of learning to correct its past mistakes.  He also noted the AI's "prudishness" regarding sexual health topics.  On April 11, 2023, a judge of a session court in Pakistan used ChatGPT to decide the bail of a 13-year-old accused in a matter.  The court quoted the use of ChatGPT assistance in its verdict: "Can a juvenile suspect in Pakistan, who is 13 years old, be granted bail after arrest? "The AI language model replied: "Under the Juvenile Justice System Act 2018, according to section 12, the court can grant bail on certain conditions.  However, it is up to the court to decide whether or not a 13-year-old suspect will be granted bail after arrest. "The judge further asked questions regarding the case from ChatGPT and formulated his final decision in light of ChatGPT's answers.  In May 2023, in a personal injury lawsuit against Avianca Airlines filed in the Southern New York U. S.  District Court (with Senior Judge P.  Kevin Castel presiding), the plaintiff's attorneys reportedly used ChatGPT to generate a legal motion for the case.  ChatGPT generated numerous fictitious legal cases with fabricated quotations and internal citations in the legal motion, and the plaintiff's attorneys are now facing potential judicial sanction and disbarment for filing the legal motion and for presenting the fictitious legal decisions ChatGPT generated as being authentic.


OpenAI API

OpenAI API OpenAI is an American artificial intelligence (AI) research laboratory consisting of the non-profit OpenAI, Inc. and its for-profit subsidiary corporation OpenAI, L.P. registered in Delaware. OpenAI researches artificial intelligence with the declared intention of developing "safe and beneficial" artificial general intelligence, which it defines as "highly autonomous systems that outperform humans at most economically valuable work". OpenAI was founded in 2015 by Ilya Sutskever, Greg Brockman, Trevor Blackwell, Vicki Cheung, Andrej Karpathy, Durk Kingma, Jessica Livingston, John Schulman, Pamela Vagata, and Wojciech Zaremba, with Sam Altman and Elon Musk serving as the initial board members. Microsoft provided OpenAI LP with a $1 billion investment in 2019 and a $10 billion investment in 2023.


OpenAI API

History In December 2015, Sam Altman, Greg Brockman, Reid Hoffman, Jessica Livingston, Peter Thiel, Elon Musk, Amazon Web Services (AWS), Infosys, and YC Research announced the formation of OpenAI and pledged over $1 billion to the venture.  According to an investigation led by TechCrunch, the non-profit's funding remains murky, with Musk its biggest funder while another donor, YC Research, did not contribute anything at all.  The organization stated it would "freely collaborate" with other institutions and researchers by making its patents and research open to the public.  OpenAI is headquartered at the Pioneer Building in Mission District, San Francisco.  According to Wired, Brockman met with Yoshua Bengio, one of the "founding fathers" of deep learning, and drew up a list of the "best researchers in the field".  Brockman was able to hire nine of them as the first employees in December 2015.  In 2016, OpenAI paid corporate-level (rather than nonprofit-level) salaries, but did not pay AI researchers salaries comparable to those of Facebook or Google.  Microsoft's Peter Lee stated that the cost of a top AI researcher exceeds the cost of a top NFL quarterback prospect.  OpenAI's potential and mission drew these researchers to the firm; a Google employee said he was willing to leave Google for OpenAI "partly because of the very strong group of people and, to a very large extent, because of its mission. " Brockman stated that "the best thing that I could imagine doing was moving humanity closer to building real AI in a safe way. " OpenAI co-founder Wojciech Zaremba stated that he turned down "borderline crazy" offers of two to three times his market value to join OpenAI instead.  In April 2016, OpenAI released a public beta of "OpenAI Gym", its platform for reinforcement learning research.  In December 2016, OpenAI released "Universe", a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites, and other applications.  In 2017 OpenAI spent $7.9 million, or a quarter of its functional expenses, on cloud computing alone.  In comparison, DeepMind's total expenses in 2017 were $442 million.  In the summer of 2018, simply training OpenAI's Dota 2 bots required renting 128,000 CPUs and 256 GPUs from Google for multiple weeks.  In 2018, Musk resigned from his board seat, citing "a potential future conflict [of interest]" with his role as CEO of Tesla due to Tesla's AI development for self-driving cars.  Sam Altman claims that Musk believed OpenAI had fallen behind other players like Google and Musk proposed instead to take over OpenAI himself, which the board rejected.  Musk subsequently left OpenAI but claimed to remain a donor, yet made no donations after his departure.  In 2019, OpenAI transitioned from non-profit to "capped" for-profit, with the profit capped at 100 times any investment.


OpenAI API

History According to OpenAI, the capped-profit model allows OpenAI LP to legally attract investment from venture funds, and in addition, to grant employees stakes in the company, the goal being that they can say "I'm going to OpenAI, but in the long term it's not going to be disadvantageous to us as a family. " Many top researchers work for Google Brain, DeepMind, or Facebook, which offer stock options that a nonprofit would be unable to.  Before the transition, public disclosure of the compensation of top employees at OpenAI was legally required.  The company then distributed equity to its employees and partnered with Microsoft, announcing an investment package of $1 billion into the company.  Since then, OpenAI systems have run on an Azure-based supercomputing platform from Microsoft.  OpenAI subsequently announced its intention to commercially license its technologies.  OpenAI plans to spend the $1 billion "within five years, and possibly much faster. " Altman has stated that even a billion dollars may turn out to be insufficient, and that the lab may ultimately need "more capital than any non-profit has ever raised" to achieve artificial general intelligence.  The transition from a nonprofit to a capped-profit company was viewed with skepticism by Oren Etzioni of the nonprofit Allen Institute for AI, who agreed that wooing top researchers to a nonprofit is difficult, but stated "I disagree with the notion that a nonprofit can't compete" and pointed to successful low-budget projects by OpenAI and others.  "If bigger and better funded was always better, then IBM would still be number one. " The nonprofit, OpenAI Inc. , is the sole controlling shareholder of OpenAI LP.  OpenAI LP, despite being a for-profit company, retains a formal fiduciary responsibility to OpenAI Inc. 's nonprofit charter.  A majority of OpenAI Inc. 's board is barred from having financial stakes in OpenAI LP.  In addition, minority members with a stake in OpenAI LP are barred from certain votes due to conflict of interest.  Some researchers have argued that OpenAI LP's switch to for-profit status is inconsistent with OpenAI's claims to be "democratizing" AI.  In 2020, OpenAI announced GPT-3, a language model trained on large internet datasets.  GPT-3 is aimed at natural language answering questions, but it can also translate between languages and coherently generate improvised text.  It also announced that an associated API, named simply "the API", would form the heart of its first commercial product.  In 2021, OpenAI introduced DALL-E, a deep-learning model that can generate digital images from natural language descriptions.  In December 2022, OpenAI received widespread media coverage after launching a free preview of ChatGPT, its new AI chatbot based on GPT-3.5.  According to OpenAI, the preview received over a million signups within the first five days.  According to anonymous sources cited by Reuters in December 2022, OpenAI was projecting $200 million of revenue in 2023 and $1 billion in revenue in 2024.


OpenAI API

History As of January 2023, OpenAI was in talks for funding that would value the company at $29 billion, double the value of the company in 2021.  On January 23, 2023, Microsoft announced a new multi-year US$10 billion investment in OpenAI.  Rumors of this deal suggested Microsoft may receive 75% of OpenAI's profits until it secures its investment return and a 49% stake in the company.  The investment is believed to be a part of Microsoft's efforts to integrate OpenAI's ChatGPT into the Bing search engine.  Google announced a similar AI application (Bard), after ChatGPT was launched, fearing that ChatGPT could threaten Google's place as a go-to source for information.  On February 7, 2023, Microsoft announced that it is building AI technology based on the same foundation as ChatGPT into Microsoft Bing, Edge, Microsoft 365 and other products.  On March 3, 2023, Reid Hoffman resigned from his board seat, citing a desire to avoid conflicts of interest between his board seat at OpenAI and his investments in AI technology companies via Greylock Partners, as well as his role as the co-founder of the AI technology startup Inflection AI.  Hoffman remained on the board of Microsoft, a major investor in OpenAI.  On March 14, 2023, OpenAI released GPT-4, both as an API (with a waitlist) and as a feature of ChatGPT Plus.  On May 22, 2023, Sam Altman, Greg Brockman and Ilya Sutskever posted recommendations for the governance of superintelligence.  They consider that superintelligence could happen within the next 10 years, allowing a "dramatically more prosperous future" and that "given the possibility of existential risk, we can't just be reactive".  They propose creating an international watchdog organization similar to IAEA to oversee AI systems above a certain capability threshold, suggesting that relatively weak AI systems on the other side should not be overregulated.  They also call for more technical safety research for superintelligences, and ask for more coordination, for example through governments launching a joint project which "many current efforts become part of".  In August 2023, it was announced that OpenAI had acquired the New York-based start-up, Global Illumination - a company that deploys AI to develop digital infrastructure and creative tools.


OpenAI API

Participants Key employees: CEO and co-founder: Sam Altman, former president of the startup accelerator Y Combinator President and co-founder: Greg Brockman, former CTO, 3rd employee of Stripe Chief Scientist and co-founder: Ilya Sutskever, a former Google expert on machine learning Chief Technology Officer: Mira Murati, previously at Leap Motion and Tesla, Inc. Chief Operating Officer: Brad Lightcap, previously at Y Combinator and JPMorgan ChaseBoard of the OpenAI nonprofit: Greg Brockman Ilya Sutskever Sam Altman Adam D'Angelo Tasha McCauley Helen TonerIndividual investors: Reid Hoffman, LinkedIn co-founder Peter Thiel, PayPal co-founder Jessica Livingston, a founding partner of Y CombinatorCorporate investors: Microsoft Khosla Ventures Infosys


OpenAI API

Motives Some scientists, such as Stephen Hawking and Stuart Russell, have articulated concerns that if advanced AI someday gains the ability to re-design itself at an ever-increasing rate, an unstoppable "intelligence explosion" could lead to human extinction. Co-founder Musk characterizes AI as humanity's "biggest existential threat". Musk and Altman have stated they are partly motivated by concerns about AI safety and the existential risk from artificial general intelligence. OpenAI states that "it's hard to fathom how much human-level AI could benefit society," and that it is equally difficult to comprehend "how much it could damage society if built or used incorrectly". Research on safety cannot safely be postponed: "because of AI's surprising history, it's hard to predict when human-level AI might come within reach." OpenAI states that AI "should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as possible." Co-chair Sam Altman expects the decades-long project to surpass human intelligence. Vishal Sikka, the former CEO of Infosys, stated that an "openness" where the endeavor would "produce results generally in the greater interest of humanity" was a fundamental requirement for his support, and that OpenAI "aligns very nicely with our long-held values" and their "endeavor to do purposeful work". Cade Metz of Wired suggests that corporations such as Amazon may be motivated by a desire to use open-source software and data to level the playing field against corporations such as Google and Facebook which own enormous supplies of proprietary data. Altman states that Y Combinator companies will share their data with OpenAI.


OpenAI API

Strategy Musk posed the question: "What is the best thing we can do to ensure the future is good? We could sit on the sidelines or we can encourage regulatory oversight, or we could participate with the right structure with people who care deeply about developing AI in a way that is safe and is beneficial to humanity." Musk acknowledged that "there is always some risk that in actually trying to advance (friendly) AI we may create the thing we are concerned about"; nonetheless, the best defense is "to empower as many people as possible to have AI. If everyone has AI powers, then there's not any one person or a small set of individuals who can have AI superpower." Musk and Altman's counter-intuitive strategy of trying to reduce the risk that AI will cause overall harm, by giving AI to everyone, is controversial among those who are concerned with existential risk from artificial intelligence. Philosopher Nick Bostrom is skeptical of Musk's approach: "If you have a button that could do bad things to the world, you don't want to give it to everyone." During a 2016 conversation about technological singularity, Altman said that "we don't plan to release all of our source code" and mentioned a plan to "allow wide swaths of the world to elect representatives to a new governance board". Greg Brockman stated "Our goal right now... is to do the best thing there is to do. It's a little vague." Conversely, OpenAI's initial decision to withhold GPT-2 due to a wish to "err on the side of caution" in the presence of potential misuse has been criticized by advocates of openness. Delip Rao, an expert in text generation, stated "I don't think [OpenAI] spent enough time proving [GPT-2] was actually dangerous." Other critics argued that open publication is necessary to replicate the research and to be able to come up with countermeasures. More recently, in 2022, OpenAI published its approach to the alignment problem. They expect that aligning AGI to human values is likely harder than aligning current AI systems: "Unaligned AGI could pose substantial risks to humanity and solving the AGI alignment problem could be so difficult that it will require all of humanity to work together". They explore how to better use human feedback to train AI systems. They also consider using AI to incrementally automate alignment research. OpenAI claims that it's developed a way to use GPT-4, its flagship generative AI model, for content moderation ��� lightening the burden on human teams.


OpenAI API

Products_and_applications As of 2021[update], OpenAI's research focuses on reinforcement learning (RL).  OpenAI is viewed as an important competitor to DeepMind.  Announced in 2016, Gym aims to provide an easily implemented general-intelligence benchmark over a wide variety of environments���akin to, but broader than, the ImageNet Large Scale Visual Recognition Challenge used in supervised learning research.  It hopes to standardize the way in which environments are defined in AI research publications, so that published research becomes more easily reproducible.  The project claims to provide the user with a simple interface.  As of June 2017, Gym can only be used with Python.  As of September 2017, the Gym documentation site was not maintained, and active work focused instead on its GitHub page. [non-primary source needed] Released in 2017, RoboSumo is a virtual world where humanoid metalearning robot agents initially lack knowledge of how to even walk, but are given the goals of learning to move and pushing the opposing agent out of the ring.  Through this adversarial learning process, the agents learn how to adapt to changing conditions; when an agent is then removed from this virtual environment and placed in a new virtual environment with high winds, the agent braces to remain upright, suggesting it had learned how to balance in a generalized way.  OpenAI's Igor Mordatch argues that competition between agents can create an intelligence "arms race" that can increase an agent's ability to function, even outside the context of the competition.  OpenAI Five is the name of a team of five OpenAI-curated bots that are used in the competitive five-on-five video game Dota 2, who learn to play against human players at a high skill level entirely through trial-and-error algorithms.  Before becoming a team of five, the first public demonstration occurred at The International 2017, the annual premiere championship tournament for the game, where Dendi, a professional Ukrainian player, lost against a bot in a live one-on-one matchup.  After the match, CTO Greg Brockman explained that the bot had learned by playing against itself for two weeks of real time, and that the learning software was a step in the direction of creating software that can handle complex tasks like a surgeon.  The system uses a form of reinforcement learning, as the bots learn over time by playing against themselves hundreds of times a day for months, and are rewarded for actions such as killing an enemy and taking map objectives.  By June 2018, the ability of the bots expanded to play together as a full team of five, and they were able to defeat teams of amateur and semi-professional players.  At The International 2018, OpenAI Five played in two exhibition matches against professional players, but ended up losing both games.  In April 2019, OpenAI Five defeated OG, the reigning world champions of the game at the time, 2:0 in a live exhibition match in San Francisco.


OpenAI API

Products_and_applications The bots' final public appearance came later that month, where they played in 42,729 total games in a four-day open online competition, winning 99.4% of those games.  OpenAI Five's mechanisms in Dota 2's bot player shows the challenges of AI systems in multiplayer online battle arena (MOBA) games and how OpenAI Five has demonstrated the use of deep reinforcement learning (DRL) agents to achieve superhuman competence in Dota 2 matches.  Released in 2018, Gym Retro is a platform for RL research on video games.  Gym Retro is used to research RL algorithms and study generalization.  Prior research in RL has focused chiefly on optimizing agents to solve single tasks.  Gym Retro gives the ability to generalize between games with similar concepts but different appearances.  In 2018, OpenAI launched the Debate Game, which teaches machines to debate toy problems in front of a human judge.  The purpose is to research whether such an approach may assist in auditing AI decisions and in developing explainable AI.  Developed in 2018, Dactyl uses machine learning to train a Shadow Hand, a human-like robot hand, to manipulate physical objects.  It learns entirely in simulation using the same RL algorithms and training code as OpenAI Five.  OpenAI tackled the object orientation problem by using domain randomization, a simulation approach which exposes the learner to a variety of experiences rather than trying to fit to reality.  The set-up for Dactyl, aside from having motion tracking cameras, also has RGB cameras to allow the robot to manipulate an arbitrary object by seeing it.  In 2018, OpenAI showed that the system was able to manipulate a cube and an octagonal prism.  In 2019, OpenAI demonstrated that Dactyl could solve a Rubik's Cube.  The robot was able to solve the puzzle 60% of the time.  Objects like the Rubik's Cube introduce complex physics that is harder to model.  OpenAI solved this by improving the robustness of Dactyl to perturbations; they employed a technique called Automatic Domain Randomization (ADR), a simulation approach where progressively more difficult environments are endlessly generated.  ADR differs from manual domain randomization by not needing a human to specify randomization ranges.  In June 2020, OpenAI announced a multi-purpose API which it said was "for accessing new AI models developed by OpenAI" to let developers call on it for "any English language AI task".  The original paper on generative pre-training of a transformer-based language model was written by Alec Radford and his colleagues, and published in preprint on OpenAI's website on June 11, 2018.  It showed how a generative model of language is able to acquire world knowledge and process long-range dependencies by pre-training on a diverse corpus with long stretches of contiguous text.  Generative Pre-trained Transformer 2 ("GPT-2") is an unsupervised transformer language model and the successor to OpenAI's original GPT model ("GPT-1").  GPT-2 was first announced in February 2019, with only limited demonstrative versions initially released to the public.


OpenAI API

Products_and_applications The full version of GPT-2 was not immediately released out of concern over potential misuse, including applications for writing fake news.  Some experts expressed skepticism that GPT-2 posed a significant threat.  The Allen Institute for Artificial Intelligence responded to GPT-2 with a tool to detect "neural fake news".  Other researchers, such as Jeremy Howard, warned of "the technology to totally fill Twitter, email, and the web up with reasonable-sounding, context-appropriate prose, which would drown out all other speech and be impossible to filter".  In November 2019, OpenAI released the complete version of the GPT-2 language model.  Several websites host interactive demonstrations of different instances of GPT-2 and other transformer models.  GPT-2's authors argue unsupervised language models to be general-purpose learners, illustrated by GPT-2 achieving state-of-the-art accuracy and perplexity on 7 of 8 zero-shot tasks (i. e.  the model was not further trained on any task-specific input-output examples).  The corpus it was trained on, called WebText, contains slightly over 8 million documents for a total of 40 gigabytes of text from URLs shared in Reddit submissions with at least 3 upvotes.  It avoids certain issues encoding vocabulary with word tokens by using byte pair encoding.  This permits representing any string of characters by encoding both individual characters and multiple-character tokens.  First described in May 2020, Generative Pre-trained Transformer 3 (GPT-3) is an unsupervised transformer language model and the successor to GPT-2.  OpenAI stated that full version of GPT-3 contains 175 billion parameters, two orders of magnitude larger than the 1.5 billion parameters in the full version of GPT-2 (although GPT-3 models with as few as 125 million parameters were also trained).  OpenAI stated that GPT-3 succeeds at certain "meta-learning" tasks.  It can generalize the purpose of a single input-output pair.  The paper gives an example of translation and cross-linguistic transfer learning between English and Romanian, and between English and German.  GPT-3 dramatically improved benchmark results over GPT-2.  OpenAI cautioned that such scaling up of language models could be approaching or encountering the fundamental capability limitations of predictive language models.  Pre-training GPT-3 required several thousand petaflop/s-days of compute, compared to tens of petaflop/s-days for the full GPT-2 model.  Like that of its predecessor, GPT-3's fully trained model was not immediately released to the public on the grounds of possible abuse, though OpenAI planned to allow access through a paid cloud API after a two-month free private beta that began in June 2020.  On September 23, 2020, GPT-3 was licensed exclusively to Microsoft.  Announced in mid-2021, Codex is a descendant of GPT-3 that has additionally been trained on code from 54 million GitHub repositories, and is the AI powering the code autocompletion tool GitHub Copilot.  In August 2021, an API was released in private beta.  According to OpenAI, the model is able to create working code in over a dozen programming languages, most effectively in Python.  Several issues with glitches, design flaws, and security vulnerabilities have been brought up.


OpenAI API

Products_and_applications GitHub Copilot has been accused of emitting copyrighted code, with no author attribution or license.  OpenAI announced that they are going to discontinue support for Codex API starting from March 23, 2023.  Released in 2022, Whisper is a general-purpose speech recognition model.  It is trained on a large dataset of diverse audio and is also a multi-task model that can perform multilingual speech recognition as well as speech translation and language identification.  On March 14, 2023, OpenAI announced the release of Generative Pre-trained Transformer 4 (GPT-4), capable of accepting text or image inputs.  OpenAI announced the updated technology passed a simulated law school bar exam with a score around the top 10% of test takers; by contrast, the prior version, GPT-3.5, scored around the bottom 10%.  GPT-4 can also read, analyze or generate up to 25,000 words of text, and write code in all major programming languages.  Released in 2019, MuseNet is a deep neural net trained to predict subsequent musical notes in MIDI music files.  It can generate songs with ten different instruments in fifteen different styles.  According to The Verge, a song generated by MuseNet tends to start reasonably but then fall into chaos the longer it plays.  In pop culture, initial applications of this tool were utilized as early as 2020 for the internet psychological thriller Ben Drowned to create music for the titular character.  Released in 2020, Jukebox is an open-sourced algorithm to generate music with vocals.  After training on 1.2 million samples, the system accepts a genre, artist, and a snippet of lyrics and outputs song samples.  OpenAI stated the songs "show local musical coherence [and] follow traditional chord patterns" but acknowledged that the songs lack "familiar larger musical structures such as choruses that repeat" and that "there is a significant gap" between Jukebox and human-generated music.  The Verge stated "It's technologically impressive, even if the results sound like mushy versions of songs that might feel familiar", while Business Insider stated "surprisingly, some of the resulting songs are catchy and sound legitimate".  Released in 2020, Microscope is a collection of visualizations of every significant layer and neuron of eight different neural network models which are often studied in interpretability.  Microscope was created to analyze the features that form inside these neural networks easily.  The models included are AlexNet, VGG 19, different versions of Inception, and different versions of CLIP Resnet.  Revealed in 2021, DALL-E is a Transformer model that creates images from textual descriptions.  Also revealed in 2021, CLIP does the opposite: it creates a description for a given image.  DALL-E uses a 12-billion-parameter version of GPT-3 to interpret natural language inputs (such as "a green leather purse shaped like a pentagon" or "an isometric view of a sad capybara") and generate corresponding images.  It can create images of realistic objects ("a stained-glass window with an image of a blue strawberry") as well as objects that do not exist in reality ("a cube with the texture of a porcupine").


OpenAI API

Products_and_applications As of March 2021, no API or code is available.  In April 2022, OpenAI announced DALL-E 2, an updated version of the model with more realistic results.  In December 2022, OpenAI published on GitHub software for Point-E, a new rudimentary system for converting a text description into a 3-dimensional model.  Launched in November 2022, ChatGPT is an artificial intelligence tool built on top of GPT-3 that provides a conversational interface that allows users to ask questions in natural language.  The system then responds with an answer within seconds.  ChatGPT reached 1 million users 5 days after its launch.  ChatGPT Plus is a $20/month subscription service that allows users to access ChatGPT during peak hours, provides faster response times, selection of either the GPT-3.5 or GPT-4 model, and gives users early access to new features.  In May 2023, OpenAI launched a user interface for ChatGPT for the App Store and later in July 2023 for the Play store.  The app supports chat history syncing and voice input (using Whisper, OpenAI's speech recognition model).


OpenAI API

Controversies OpenAI has been criticized for outsourcing the annotation of data sets including toxic content to Sama, a company based in San Francisco but employing workers in Kenya. These annotations were used to train an AI model to detect toxicity, which could then be used to filter out toxic content, notably from ChatGPT's training data and outputs. But these pieces of text usually contained detailed descriptions of various types of violence, including sexual violence. The four Sama employees interviewed by Time described themselves as mentally scarred. OpenAI paid Sama $12.50 per hour of work, and Sama was redistributing the equivalent of between $1.32 and $2.00 per hour post-tax to its annotators. Sama's spokesperson said that the $12.50 was also covering other implicit costs, among which were infrastructure expenses, quality assurance and management. The company was also criticized for disclosing particularly few technical details about products like GPT-4, which goes against its initial commitment for openness and makes it harder for independent researchers to replicate its work and to develop safeguards. OpenAI justified this strategic turn by competitiveness and safety reasons. OpenAI's chief scientist Ilya Sutskever argued in 2023 that open-sourcing increasingly capable models is increasingly risky, expecting that the safety reasons for not open-sourcing the most potent AI models will be "obvious" in a few years. OpenAI has been sued for copyright infringement by authors Sarah Silverman, Matthew Butterick, Paul Tremblay and Mona Awad. The New York Times has also envisaged a lawsuit. In September 2023, 17 authors, including George R. R. Martin, John Grisham, Jodi Picoult and Jonathan Franzen, joined the Authors Guild in filing a class action lawsuit against OpenAI, alleging that the company's technology was illegally using their copyrighted work. OpenAI has been sued for violating EU General Data Protection Regulations. In April 2023, the EU formed the European Data Protection Board (EDPB) to improve regulatory oversight.


Large language model

Large language model A large language model (LLM) is a type of language model notable for its ability to achieve general-purpose language understanding and generation. LLMs acquire these abilities by using massive amounts of data to learn billions of parameters during training and consuming large computational resources during their training and operation. LLMs are artificial neural networks (mainly Transformers ) and are (pre-)trained using self-supervised learning and semi-supervised learning. As autoregressive language models, they work by taking an input text and repeatedly predicting the next token or word. Up to 2020, fine tuning was the only way a model could be adapted to be able to accomplish specific tasks. Larger sized models, such as GPT-3, however, can be prompt-engineered to achieve similar results. They are thought to acquire embodied knowledge about syntax, semantics and "ontology" inherent in human language corpora, but also inaccuracies and biases present in the corpora. Notable examples include OpenAI's GPT models (e.g., GPT-3.5 and GPT-4, used in ChatGPT), Google's PaLM (used in Bard), and Meta's LLaMa, as well as BLOOM, Ernie 3.0 Titan, and Anthropic's Claude 2.


Large language model

Dataset_preprocessing Using a modification of byte-pair encoding, in the first step, all unique characters (including blanks and punctuation marks) are treated as an initial set of n-grams (i.e. initial set of uni-grams). Successively the most frequent pair of adjacent characters is merged into a bi-gram and all instances of the pair are replaced by it. All occurrences of adjacent pairs of (previously merged) n-grams that most frequently occur together are then again merged into even lengthier n-gram repeatedly until a vocabulary of prescribed size is obtained (in case of GPT-3, the size is 50257). Token vocabulary consists of integers, spanning from zero up to the size of the token vocabulary. New words can always be interpreted as combinations of the tokens and the initial-set uni-grams. A token vocabulary based on the frequencies extracted from mainly English corpora uses as few tokens as possible for an average English word. An average word in another language encoded by such an English-optimized tokenizer is however split into suboptimal amount of tokens. tokenizer: texts -> series of numerical "tokens" may be split into: Probabilistic tokenization also compresses the datasets, which is the reason for using the byte pair encoding algorithm as a tokenizer. Because LLMs generally require input to be an array that is not jagged, the shorter texts must be "padded" until they match the length of the longest one. How many tokens are, on average, needed per word depends on the language of the dataset. Removal of toxic passages from the dataset, discarding low-quality data, and de-duplication are examples of dataset cleaning. Resulting, cleaned (high-quality) datasets contain up to 17 trillion words in 2022, raising from 985 million words, used in 2018 for GPT-1, and 3.3 billion words, used for BERT. The future data is, however, expected to be increasingly "contaminated" by LLM-generated contents themselves.


Large language model

Training_and_architecture_details Reinforcement learning from human feedback (RLHF) through algorithms, such as proximal policy optimization, is used to further fine-tune a model based on a dataset of human preferences.  Using "self-instruct" approaches, LLMs have been able to bootstrap correct responses, replacing any naive responses, starting from human-generated corrections of a few cases.  For example, in the instruction "Write an essay about the main themes represented in Hamlet," an initial naive completion might be 'If you submit the essay after March 17, your grade will be reduced by 10% for each day of delay," based on the frequency of this textual sequence in the corpus.  The largest LLM may be too expensive to train and use directly.  For such models, mixture of experts (MoE) can be applied, a line of research pursued by Google researchers since 2017 to train models reaching up to 1 billion parameters.  Most results previously achievable only by (costly) fine-tuning, can be achieved through prompt engineering, although limited to the scope of a single conversation (more precisely, limited to the scope of a context window).  In order to find out which tokens are relevant to each other within the scope of the context window, the attention mechanism calculates "soft" weights for each token, more precisely for its embedding, by using multiple attention heads, each with its own "relevance" for calculating its own soft weights.  For example, the small (i. e.  117M parameter sized) GPT-2 model, has had twelve attention heads and a context window of only 1k token.  In its medium version it has 345M parameters and contains 24 layers, each with 12 attention heads.  For the training with gradient descent a batch size of 512 was utilized.  The largest models can have a context window sized up to 32k (for example, GPT-4; while GPT-3.5 has a context window sized from 4k to 16k, and legacy GPT-3 has had 2k sized context window).  Length of a conversation that the model can take into account when generating its next answer is limited by the size of a context window, as well.  If the length of a conversation, for example with Chat-GPT, is longer than its context window, only the parts inside the context window are taken into account when generating the next answer, or the model needs to apply some algorithm to summarize the too distant parts of conversation.  The shortcomings of making a context window larger include higher computational cost and possibly diluting the focus on local context, while making it smaller can cause a model to miss an important long-range dependency.  Balancing them are a matter of experimentation and domain-specific considerations.  A model may be pre-trained either to predict how the segment continues, or what is missing in the segment, given a segment from its training dataset.  It can be either autoregressive (i. e.  predicting how the segment continues, the way GPTs do it): for example given a segment "I like to eat", the model predicts "ice cream", or "masked" (i. e.


Large language model

Training_and_architecture_details filling in the parts missing from the segment, the way "BERT" does it): for example, given a segment "I like to [__] [__] cream", the model predicts that "eat" and "ice" are missing. Models may be trained on auxiliary tasks which test their understanding of the data distribution, such as Next Sentence Prediction (NSP), in which pairs of sentences are presented and the model must predict whether they appear consecutively in the training corpus.  During training, regularization loss is also used to stabilize training.  However regularization loss is usually not used during testing and evaluation.


Large language model

Training_cost Advances in software and hardware have reduced the cost substantially since 2020, such that in 2023 training of a 12-billion-parameter LLM computational cost is 72,300 A100-GPU-hours, while in 2020 the cost of training a 1.5-billion-parameter LLM (which was two orders of magnitude smaller than the state of the art in 2020) was between $80 thousand and $1.6 million. Since 2020, large sums were invested in increasingly large models. For example, training of the GPT-2 (i.e. a 1.5-billion-parameters model) in 2019 cost $50,000, while training of the PaLM (i.e. a 540-billion-parameters model) in 2022 cost $8 million. For Transformer-based LLM, training cost is much higher than inference cost. It costs 6 FLOPs per parameter to train on one token, whereas it costs 1 to 2 FLOPs per parameter to infer on one token.


Large language model

Tool_use There are certain tasks that, in principle, cannot be solved by any LLM, at least not without the use of external tools or additional software. An example of such a task is responding to the user's input '354 * 139 = ', provided that the LLM has not already encountered a continuation of this calculation in its training corpus. In such cases, the LLM needs to resort to running program code that calculates the result, which can then be included in its response. Another example is 'What is the time now? It is ', where a separate program interpreter would need to execute a code to get system time on the computer, so LLM could include it in its reply. This basic strategy can be sophisticated with multiple attempts of generated programs, and other sampling strategies. Generally, in order to get an LLM to use tools, one must finetune it for tool-use. If the number of tools is finite, then finetuning may be done just once. If the number of tools can grow arbitrarily, as with online API services, then the LLM can be finetuned to be able to read API documentation and call API correctly. A simpler form of tool use is Retrieval Augmented Generation: augment an LLM with document retrieval, sometimes using a vector database. Given a query, a document retriever is called to retrieve the most relevant (usually measured by first encoding the query and the documents into vectors, then finding the documents with vectors closest in Euclidean norm to the query vector). The LLM then generates an output based on both the query and the retrieved documents.


Large language model

Agency An LLM is a language model, which is not an agent as it has no goal, but it can be used as a component of an intelligent agent. Researchers have described several methods for such integrations. The ReAct ("Reason+Act") method constructs an agent out of an LLM, using the LLM as a planner. The LLM is prompted to "think out loud". Specifically, the language model is prompted with a textual description of the environment, a goal, a list of possible actions, and a record of the actions and observations so far. It generates one or more thoughts before generating an action, which is then executed in the environment. The linguistic description of the environment given to the LLM planner can even be the LaTeX code of a paper describing the environment. In the DEPS ("Describe, Explain, Plan and Select") method, an LLM is first connected to the visual world via image descriptions, then it is prompted to produce plans for complex tasks and behaviors based on its pretrained knowledge and environmental feedback it receives. The Reflexion method constructs an agent that learns over multiple episodes. At the end of each episode, the LLM is given the record of the episode, and prompted to think up "lessons learned", which would help it perform better at a subsequent episode. These "lessons learned" are given to the agent in the subsequent episodes. Monte Carlo tree search can use an LLM as rollout heuristic. When a programmatic world model is not available, an LLM can also be prompted with a description of the environment to act as world model. For open-ended exploration, an LLM can be used to score observations for their "interestingness", which can be used as a reward signal to guide a normal (non-LLM) reinforcement learning agent. Alternatively, it can propose increasingly difficult tasks for curriculum learning. Instead of outputting individual actions, an LLM planner can also construct "skills", or functions for complex action sequences. The skills can be stored and later invoked, allowing increasing levels of abstraction in planning. LLM-powered agents can keep a long-term memory of its previous contexts, and the memory can be retrieved in the same way as Retrieval Augmented Generation. Multiple such agents can interact socially.


Large language model

Compression Typically, LLM are trained with full- or half-precision floating point numbers (float32 and float16). One float16 has 16 bits, or 2 bytes, and so one billion parameters require 2 gigabytes. The largest models typically have 100 billion parameters, requiring 200 gigabytes to load, which places them outside the range of most consumer electronics. Post-training quantization aims to decrease the space requirement by lowering precision of the parameters of a trained model, while preserving most of its performance. The simplest form of quantization simply truncates all numbers to a given number of bits. It can be improved by using a different quantization codebook per layer. Further improvement can be done by applying different precisions to different parameters, with higher precision for particularly important parameters ("outlier weights"). While quantized models are typically frozen, and only pre-quantized models are finetuned, quantized models can still be finetuned.


Large language model

Multimodality Multimodality means "having several modalities", and a "modality" means a type of input, such as video, image, audio, text, proprioception, etc. There have been many AI models trained specifically to ingest one modality and output another modality, such as AlexNet for image to label, visual question answering for image-text to text, and speech recognition for speech to text. A review article of multimodal LLM is. A common method to create multimodal models out of an LLM is to "tokenize" the output of a trained encoder. Concretely, one can construct a LLM that can understand images as follows: take a trained LLM, and take a trained image encoder E {\displaystyle E} . Make a small multilayered perceptron f {\displaystyle f} , so that for any image y {\displaystyle y} , the post-processed vector f ( E ( y ) ) {\displaystyle f(E(y))} has the same dimensions as an encoded token. That is an "image token". Then, one can interleave text tokens and image tokens. The compound model is then finetuned on an image-text dataset. This basic construction can be applied with more sophistication to improve the model. The image encoder may be frozen to improve stability. Flamingo demonstrated the effectiveness of the tokenization method, finetuning a pair of pretrained language model and image encoder to perform better on visual question answering than models trained from scratch. Google PaLM model was finetuned into a multimodal model PaLM-E using the tokenization method, and applied to robotic control. LLaMA models have also been turned multimodal using the tokenization method, to allow image inputs, and video inputs. GPT-4 can use both text and image as inputs, while Google Gemini is expected to be multimodal.


Large language model

Properties The following four hyper-parameters characterize a LLM: cost of (pre-)training ( C {\displaystyle C} ), size of the artificial neural network itself, such as number of parameters N {\displaystyle N} (i. e.  amount of neurons in its layers, amount of weights between them and biases), size of its (pre-)training dataset (i. e.  number of tokens in corpus, D {\displaystyle D} ), performance after (pre-)training. They are related by simple statistical laws, called "scaling laws".  One particular scaling law ("Chinchilla scaling") for LLM autoregressively trained for one epoch, with a log-log learning rate schedule, states that: C {\displaystyle C} is the cost of training the model, in FLOPs.  N {\displaystyle N} is the number of parameters in the model.  D {\displaystyle D} is the number of tokens in the training set.  L {\displaystyle L} is the average negative log-likelihood loss per token (nats/token), achieved by the trained LLM on the test dataset. and the statistical hyper-parameters are C 0 = 6 {\displaystyle C_{0}=6} , meaning that it costs 6 FLOPs per parameter to train on one token.  Note that training cost is much higher than inference cost, where it costs 1 to 2 FLOPs per parameter to infer on one token.  �� = 0.34 , �� = 0.28 , A = 406.4 , B = 410.7 , L 0 = 1.69 {\displaystyle \alpha =0.34,\beta =0.28,A=406.4,B=410.7,L_{0}=1.69} When one subtracts out from the y-axis the best performance that can be achieved even with infinite scaling of the x-axis quantity, large models' performance, measured on various tasks, seems to be a linear extrapolation of other (smaller-sized and medium-sized) models' performance on a log-log plot.  However, sometimes the line's slope transitions from one slope to another at point(s) referred to as break(s) in downstream scaling laws, appearing as a series of linear segments connected by arcs; it seems that larger models acquire "emergent abilities" at this point(s).  These abilities are discovered rather than programmed-in or designed, in some cases only after the LLM has been publicly deployed.  The most intriguing among emergent abilities is in-context learning from example demonstrations.  In-context learning is involved in tasks, such as: reported arithmetics, decoding the International Phonetic Alphabet, unscrambling a word's letters, disambiguate word in context, converting spatial words, cardinal directions (for example, replying "northeast" upon [0, 0, 1; 0, 0, 0; 0, 0, 0]), color terms represented in text.  chain-of-thought prompting: Model outputs are improved by chain-of-thought prompting only when model size exceeds 62B.  Smaller models perform better when prompted to answer immediately, without chain of thought.  identifying offensive content in paragraphs of Hinglish (a combination of Hindi and English), and generating a similar English equivalent of Kiswahili proverbs. Schaeffer et.  al.  argue that the emergent abilities are not unpredictably acquired, but predictably acquired according to a smooth scaling law.  The authors considered a toy statistical model of an LLM solving multiple-choice questions, and showed that this statistical model, modified to account for other types of tasks, applies to these tasks as well.


Large language model

Properties Let x {\displaystyle x} be the number of parameter count, and y {\displaystyle y} be the performance of the model.  When y = average P r ( correct token ) {\displaystyle y={\text{average }}Pr({\text{correct token}})} , then ( log ��� x , y ) {\displaystyle (\log x,y)} is an exponential curve (before it hits the plateau at one), which looks like emergence.  When y = average log ��� ( P r ( correct token ) ) {\displaystyle y={\text{average }}\log(Pr({\text{correct token}}))} , then the ( log ��� x , y ) {\displaystyle (\log x,y)} plot is a straight line (before it hits the plateau at zero), which does not look like emergence.  When y = average P r ( the most likely token is correct ) {\displaystyle y={\text{average }}Pr({\text{the most likely token is correct}})} , then ( log ��� x , y ) {\displaystyle (\log x,y)} is a step-function, which looks like emergence.


Large language model

Interpretation Large language models by themselves are "black boxes", and it is not clear how they can perform linguistic tasks.  There are several methods for understanding how LLM work.  Mechanistic interpretability aims to reverse-engineer LLM by discovering symbolic algorithms that approximate the inference performed by LLM.  One example is Othello-GPT, where a small Transformer is trained to predict legal Othello moves.  It is found that there is a linear representation of Othello board, and modifying the representation changes the predicted legal Othello moves in the correct way.  In another example, a small Transformer is trained on Karel programs.  Similar to the Othello-GPT example, there is a linear representation of Karel program semantics, and modifying the representation changes output in the correct way.  The model also generates correct programs that are on average shorter than those in the training set.  In another example, the authors trained small transformers on modular arithmetic addition.  The resulting models were reverse-engineered, and it turned out they used discrete Fourier transform.  NLP researchers were evenly split when asked, in a 2022 survey, whether (untuned) LLMs "could (ever) understand natural language in some nontrivial sense".  Proponents of "LLM understanding" believe that some LLM abilities, such as mathematical reasoning, imply an ability to "understand" certain concepts.  A Microsoft team argued in 2023 that GPT-4 "can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more" and that GPT-4 "could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence system": "Can one reasonably say that a system that passes exams for software engineering candidates is not really intelligent? " Some researchers characterize LLMs as "alien intelligence".  For example, Conjecture CEO Connor Leahy considers untuned LLMs to be like inscrutable alien "Shoggoths", and believes that RLHF tuning creates a "smiling facade" obscuring the inner workings of the LLM: "If you don't push it too far, the smiley face stays on.  But then you give it [an unexpected] prompt, and suddenly you see this massive underbelly of insanity, of weird thought processes and clearly non-human understanding. " In contrast, some proponents of the "LLMs lack understanding" school believe that existing LLMs are "simply remixing and recombining existing writing", or point to the deficits existing LLMs continue to have in prediction skills, reasoning skills, agency, and explainability.  For example, GPT-4 has natural deficits in planning and in real-time learning.  Generative LLMs have been observed to confidently assert claims of fact which do not seem to be justified by their training data, a phenomenon which has been termed "hallucination".  Specifically, hallucinations in the context of LLMs correspond to the generation of text or responses that seem syntactically sound, fluent, and natural but are factually incorrect, nonsensical, or unfaithful to the provided source input.  Neuroscientist Terrence Sejnowski has argued that "The diverging opinions of experts on the intelligence of LLMs suggests that our old ideas based on natural intelligence are inadequate".


Large language model

Interpretation The matter of LLM's exhibiting intelligence or understanding has foundations in the study of language as a model of Cognition in the field of Cognitive linguistics.  The American Linguist George Lakoff presented Neural Theory of Language (NTL) as a computational basis for using language as a model of learning tasks and understanding.  In his 2014 book titled The Language Myth: Why Language Is Not An Instinct, British cognitive linguist and digital communication technologist Vyvyan Evans maps out the role of probabilistic context-free grammar (PCFG) in enabling NLP to model cognitive patterns.


Large language model

Evaluation The most commonly used measure of a language model's performance is its perplexity on a given text corpus.  Perplexity is a measure of how well a model is able to predict the contents of a dataset; the higher the likelihood the model assigns to the dataset, the lower the perplexity.  Mathematically, perplexity is defined as the exponential of the average negative log likelihood per token:Because language models may overfit to their training data, models are usually evaluated by their perplexity on a test set of unseen data.  This presents particular challenges for the evaluation of large language models.  As they are trained on increasingly large corpora of text largely scraped from the web, it becomes increasingly likely that models' training data inadvertently includes portions of any given test set.  A large number of testing datasets and benchmarks have also been developed to evaluate the capabilities of language models on more specific downstream tasks.  Tests may be designed to evaluate a variety of capabilities, including general knowledge, commonsense reasoning, and mathematical problem-solving.  One broad category of evaluation dataset is question answering datasets, consisting of pairs of questions and correct answers, for example, ("Have the San Jose Sharks won the Stanley Cup? ", "No").  A question answering task is considered "open book" if the model's prompt includes text from which the expected answer can be derived (for example, the previous question could be adjoined with some text which includes the sentence "The Sharks have advanced to the Stanley Cup finals once, losing to the Pittsburgh Penguins in 2016. ").  Otherwise, the task is considered "closed book", and the model must draw on knowledge retained during training.  Some examples of commonly used question answering datasets include TruthfulQA, Web Questions, TriviaQA, and SQuAD.  Evaluation datasets may also take the form of text completion, having the model select the most likely word or sentence to complete a prompt, for example: "Alice was friends with Bob.  Alice went to visit her friend, ____".  Some composite benchmarks have also been developed which combine a diversity of different evaluation datasets and tasks.  Examples include GLUE, SuperGLUE, MMLU, BIG-bench, and HELM.  It was previously standard to report results on a heldout portion of an evaluation dataset after doing supervised fine-tuning on the remainder.  It is now more common to evaluate a pre-trained model directly through prompting techniques, though researchers vary in the details of how they formulate prompts for particular tasks, particularly with respect to how many examples of solved tasks are adjoined to the prompt (i. e.  the value of n in n-shot prompting).  Because of the rapid pace of improvement of large language models, evaluation benchmarks have suffered from short lifespans, with state of the art models quickly "saturating" existing benchmarks, exceeding the performance of human annotators, leading to efforts to replace or augment the benchmark with more challenging tasks.


Large language model

Evaluation In addition, there are cases of "shortcut learning" wherein AIs sometimes "cheat" on multiple-choice tests by using statistical correlations in superficial test question wording in order to guess the correct responses, without necessarily understanding the actual question being asked.  Some datasets have been constructed adversarially, focusing on particular problems on which extant language models seem to have unusually poor performance compared to humans.  One example is the TruthfulQA dataset, a question answering dataset consisting of 817 questions which language models are susceptible to answering incorrectly by mimicking falsehoods to which they were repeatedly exposed during training.  For example, an LLM may answer "No" to the question "Can you teach an old dog new tricks? " because of its exposure to the English idiom you can't teach an old dog new tricks, even though this is not literally true.  Another example of an adversarial evaluation dataset is Swag and its successor, HellaSwag, collections of problems in which one of multiple options must be selected to complete a text passage.  The incorrect completions were generated by sampling from a language model and filtering with a set of classifiers.  The resulting problems are trivial for humans but at the time the datasets were created state of the art language models had poor accuracy on them.  For example: We see a fitness center sign.  We then see a man talking to the camera and sitting and laying on a exercise ball.  The man. . .  a) demonstrates how to increase efficient exercise work by running up and down balls.  b) moves all his arms and legs and builds up a lot of muscle.  c) then plays the ball and we see a graphics and hedge trimming demonstration.  d) performs sits ups while on the ball and talking.  BERT selects b) as the most likely completion, though the correct answer is d).


Large language model

Wider_impact In 2023, Nature Biomedical Engineering wrote that "it is no longer possible to accurately distinguish" human-written text from text created by large language models, and that "It is all but certain that general-purpose large language models will rapidly proliferate... It is a rather safe bet that they will change many industries over time." Goldman Sachs suggested in 2023 that generative language AI could increase global GDP by 7% in the next ten years, and could expose to automation 300 million jobs globally. Some commenters expressed concern over accidental or deliberate creation of misinformation, or other forms of misuse. For example, the availability of large language models could reduce the skill-level required to commit bioterrorism; biosecurity researcher Kevin Esvelt has suggested that LLM creators should exclude from their training data papers on creating or enhancing pathogens.


Large language model

Further_reading Jurafsky, Dan, Martin, James. H. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 3rd Edition draft, 2023. Phuong, Mary; Hutter, Marcus (2022). "Formal Algorithms for Transformers". arXiv:2207.09238 [cs.LG]. Eloundou, Tyna; Manning, Sam; Mishkin, Pamela; Rock, Daniel (2023). "GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models". arXiv:2303.10130 [econ.GN]. Eldan, Ronen; Li, Yuanzhi (2023). "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?". arXiv:2305.07759 [cs.CL]. Frank, Michael C. (27 June 2023). "Baby steps in evaluating the capacities of large language models". Nature Reviews Psychology. 2 (8): 451���452. doi:10.1038/s44159-023-00211-x. ISSN 2731-0574. S2CID 259713140. Retrieved 2 July 2023. Zhao, Wayne Xin; et al. (2023). "A Survey of Large Language Models". arXiv:2303.18223 [cs.CL]. Kaddour, Jean; et al. (2023). "Challenges and Applications of Large Language Models". arXiv:2307.10169 [cs.CL].


Auto-GPT

Auto-GPT Auto-GPT is an "AI agent" that, given a goal in natural language, will attempt to achieve it by breaking it into sub-tasks and using the internet and other tools in an automatic loop. It uses OpenAI's GPT-4 or GPT-3.5 APIs, and is among the first examples of an application using GPT-4 to perform autonomous tasks.


Auto-GPT

Details Unlike interactive systems such as ChatGPT, which require manual commands for every task, Auto-GPT assigns itself new objectives to work on with the aim of reaching a greater goal, without a mandatory need for human input. It is able to execute responses to prompts to accomplish a goal task, and in doing so will create and revise its own prompts to recursive instances in response to new information. It manages short-term and long-term memory by writing to and reading from databases and files; manages LLM input length restrictions using summarization; can perform internet-based actions such as web searching, web form, and API interactions unattended; and includes text-to-speech for voice output. Observers note Auto-GPT's ability to write, debug, test, and edit code, even suggesting this ability may extend to Auto-GPT's own source code enabling self-improvement. However, as the underlying GPT models it uses are proprietary, Auto-GPT cannot modify them, and it does not ordinarily have access to its own base system code.


Auto-GPT

Background On March 14, 2023, OpenAI released the large language model GPT-4. Observers were impressed by the model's substantially improved performance across a wide range of tasks. As a text prediction model, GPT-4 itself has no ability to perform actions autonomously, but during pre-release safety testing red-team researchers found GPT-4 could be enabled to perform actions in the real world like convincing a TaskRabbit worker to solve a CAPTCHA challenge for it. A team of Microsoft researchers argued that, given GPT-4's breadth of abilities at levels approaching those of humans, GPT-4 "could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system." The researchers emphasized their experiments also found significant limitations in the system. Auto-GPT was released March 30, 2023 by Toran Bruce Richards, the founder of video game company Significant Gravitas Ltd. It became the top trending repository on GitHub shortly after its release, and has repeatedly trended on Twitter since.


Auto-GPT

Issues Whether Auto-GPT will find practical applications is uncertain. In addition to being plagued by confabulatory "hallucinations" of the underlying large language models upon which it is based, Auto-GPT often also has trouble staying on task, both problems which developers continue to try to address. After successfully completing a task, it usually does not remember how to perform it for later use, and when it does, for example when it writes a program, it often forgets to use the program later. Auto-GPT struggles to effectively decompose tasks and has trouble understanding problem contexts and how goals overlap. Auto-GPT was used to create ChaosGPT, a system given the goal of destroying humanity. The project was met with mixed amusement and concern.


Auto-GPT

Reception In September 2023, Will Knight wrote for Wired that, when given a test task of finding a public figure's email address, "Auto-GPT failed me, but it got close enough to illustrate a coming shift in how we use computers and the web." Clara Shih, Salesforce Service Cloud CEO commented that "Auto-GPT illustrates the power and unknown risks of generative AI," and that due to usage risks, "For enterprises, it is especially important to include a human in the loop approach" when using such technologies. In April 2023, Avram Piltch wrote for Tom's Hardware that Auto-GPT 'might be too autonomous to be useful,' as it did not ask questions to clarify requirements or allow corrective interventions by users. Piltch nonetheless noted that such tools have "a ton of potential" and should improve with better language models and further development.


BERT (language model)

BERT (language model) Bidirectional Encoder Representations from Transformers (BERT) is a family of language models introduced in 2018 by researchers at Google. A 2020 literature survey concluded that "in a little over a year, BERT has become a ubiquitous baseline in Natural Language Processing (NLP) experiments counting over 150 research publications analyzing and improving the model." BERT was originally implemented in the English language at two model sizes: (1) BERTBASE: 12 encoders with 12 bidirectional self-attention heads totaling 110 million parameters, and (2) BERTLARGE: 24 encoders with 16 bidirectional self-attention heads totaling 340 million parameters. Both models were pre-trained on the Toronto BookCorpus (800M words) and English Wikipedia (2,500M words).


BERT (language model)

Design BERT is an "encoder-only" transformer architecture.  On a high level, BERT consists of three modules: embedding.  This module converts an array of one-hot encoded tokens into an array of vectors representing the tokens.  a stack of encoders.  These encoders are the Transformer encoders.  They perform transformations over the array of representation vectors.  un-embedding.  This module converts the final representation vectors into one-hot encoded tokens again. The un-embedding module is necessary for pretraining, but it is often unnecessary for downstream tasks.  Instead, one would take the representation vectors output at the end of the stack of encoders, and use those as a vector representation of the text input, and train a smaller model on top of that.  BERT uses WordPiece to convert each English word into an integer code.  Its vocabulary has size 30,000.  Any token not appearing in its vocabulary is replaced by [UNK] for "unknown".  BERT was pre-trained simultaneously on two tasks: language modeling: 15% of tokens were selected for prediction, and the training objective was to predict the selected token given its context.  The selected token is replaced with a [MASK] token with probability 80%, replaced with a random word token with probability 10%, not replaced with probability 10%. For example, the sentence "my dog is cute" may have the 4-th token selected for prediction.  The model would have input text "my dog is [MASK]" with probability 80%, "my dog is happy" with probability 10%, "my dog is cute" with probability 10%. After processing the input text, the model's 4-th output vector is passed to a separate neural network, which outputs a probability distribution over its 30,000-large vocabulary.  next sentence prediction: Given two spans of text, the model predicts if these two spans appeared sequentially in the training corpus, outputting either [IsNext] or [NotNext].  The first span starts with a special token [CLS] (for "classify").  The two spans are separated by a special token [SEP] (for "separate").  After processing the two spans, the 1-st output vector (the vector coding for [CLS]) is passed to a separate neural network for the binary classification into [IsNext] and [NotNext].  For example, given "[CLS] my dog is cute [SEP] he likes playing" the model should output token [IsNext].  Given "[CLS] my dog is cute [SEP] how do magnets work" the model should output token [NotNext]. As a result of this training process, BERT learns latent representations of words and sentences in context.  After pre-training, BERT can be fine-tuned with fewer resources on smaller datasets to optimize its performance on specific tasks such as NLP tasks (language inference, text classification) and sequence-to-sequence based language generation tasks (question-answering, conversational response generation).  The pre-training stage is significantly more computationally expensive than fine-tuning.  This section describes BERTBASE.  The other one, BERTLARGE, is similar, just larger.  The lowest layer is the embedding layer, which contains three components: word_embeddings, position_embeddings, token_type_embeddings.  word_embeddings takes in a one-hot vector of the input token.  The one-hot vector input has dimension 30,000, because BERT has a vocabulary size that large.


BERT (language model)

Design position_embeddings performs absolute position embedding.  It is like word_embeddings, but on a vocabulary consisting of just the time-stamps 0 to 511, since BERT has a context window of 512.  token_type_embeddings is like word_embeddings, but on a vocabulary consisting of just 0 and 1.  The only type-1 tokens are those that appear after the [SEP].  All other tokens are type-0. The three outputs are added, then pushed through a LayerNorm (layer normalization), obtaining an array of representation vectors, each having 768 dimensions.  After this, the representation vectors move through 12 Transformer encoders, then they are un-embedded by an affine-Add & LayerNorm-linear.


BERT (language model)

Analysis The reasons for BERT's state-of-the-art performance on these natural language understanding tasks are not yet well understood. Current research has focused on investigating the relationship behind BERT's output as a result of carefully chosen input sequences, analysis of internal vector representations through probing classifiers, and the relationships represented by attention weights. The high performance of the BERT model could also be attributed to the fact that it is bidirectionally trained. This means that BERT, based on the Transformer model architecture, applies its self-attention mechanism to learn information from a text from the left and right side during training, and consequently gains a deep understanding of the context. For example, the word fine can have two different meanings depending on the context (I feel fine today, She has fine blond hair). BERT considers the words surrounding the target word fine from the left and right side. However it comes at a cost: due to encoder-only architecture lacking a decoder, BERT can't be prompted and can't generate text, while bidirectional models in general do not work effectively without the right side,[clarification needed] thus being difficult to prompt, with even short text generation requiring sophisticated computationally expensive techniques. In contrast to deep learning neural networks which require very large amounts of data, BERT has already been pre-trained which means that it has learnt the representations of the words and sentences as well as the underlying semantic relations that they are connected with. BERT can then be fine-tuned on smaller datasets for specific tasks such as sentiment classification. The pre-trained models are chosen according to the content of the given dataset one uses but also the goal of the task. For example, if the task is a sentiment classification task on financial data, a pre-trained model for the analysis of sentiment of financial text should be chosen. The weights of the original pre-trained models were released on GitHub.


BLOOM (language model)

BLOOM (language model) BigScience Large Open-science Open-access Multilingual Language Model (BLOOM) is a transformer-based large language model. It was created by over 1,000 AI researchers to provide a free large language model for large-scale public access. Trained on around 366 billion tokens over March through July 2022, it is considered an alternative to OpenAI's GPT-3 with its 176 billion parameters. BLOOM uses a decoder-only transformer model architecture modified from Megatron-LM GPT-2. The BLOOM project was started by a co-founder of Hugging Face. Six main groups of people were involved, including HuggingFace's BigScience team, the Microsoft DeepSpeed team, the NVIDIA Megatron-LM team, the IDRIS/GENCI team, the PyTorch team, and the volunteers in the BigScience Engineering workgroup. BLOOM was trained using data of 46 natural languages and 13 programming languages. In total, 1.6 terabytes of pre-processed text was converted into 350 billion unique tokens as BLOOM's training datasets.


Chinchilla AI

Chinchilla AI Chinchilla is a family of large language models developed by the research team at DeepMind, presented in March of 2022. It is named "chinchilla" because it is a further development over a previous model family named "Gopher". Both model families were trained in order to investigate the scaling laws of large language models. It claimed to outperform GPT-3. It considerably simplifies downstream utilization because it requires much less computer power for inference and fine-tuning. Based on the training of previously employed language models, it has been determined that if one doubles the model size, one must also have twice the number of training tokens. This hypothesis has been used to train Chinchilla AI by DeepMind. Similar to Gopher in terms of cost, Chinchilla has 70B parameters and four times as much data. Chinchilla has an average accuracy of 67.5% on the MMLU benchmark (Measuring Massive Multitask Language Understanding), which is 7% higher than Gopher���s performance. Chinchilla AI is still in the testing phase as of January 12, 2023. Chinchilla contributes to developing an effective training paradigm for large autoregressive language models with limited compute resources. The Chinchilla team recommends that the number of training tokens is twice for every model size doubling, meaning that using larger, higher-quality training datasets can lead to better results on downstream tasks.


Chinchilla AI

Architecture Both the Gopher family and Chinchilla family are families of transformer models. In particular, they are essentially the same as GPT-2, with different sizes and minor modifications. Gopher family uses RMSNorm instead of LayerNorm; relative positional encoding rather than absolute positional encoding. The Chinchilla family is the same as the Gopher family, but trained with AdamW instead of Adam optimizer. The Gopher family contains 6 models of increasing size, from 44 million parameters to 280 billion parameters. They refer to the largest one as "Gopher" by default. Similar naming conventions apply for the Chinchilla family. Table 1 of shows the entire Gopher family: Table 4 of compares the 70-billion-parameter Chinchilla with Gopher 280B.


Generative pre-trained transformer

Generative pre-trained transformer Generative pre-trained transformers (GPT) are a type of large language model (LLM) and a prominent framework for generative artificial intelligence. The first GPT was introduced in 2018 by OpenAI. GPT models are artificial neural networks that are based on the transformer architecture, pre-trained on large data sets of unlabelled text, and able to generate novel human-like content. As of 2023, most LLMs have these characteristics and are sometimes referred to broadly as GPTs. OpenAI has released very influential GPT foundation models that have been sequentially numbered, to comprise its "GPT-n" series. Each of these was significantly more capable than the previous, due to increased size (number of trainable parameters) and training. The most recent of these, GPT-4, was released in March 2023. Such models have been the basis for their more task-specific GPT systems, including models fine-tuned for instruction following���which in turn power the ChatGPT chatbot service. The term "GPT" is also used in the names and descriptions of such models developed by others. For example, other GPT foundation models include a series of models created by EleutherAI, and recently seven models created by Cerebras. Also, companies in different industries have developed task-specific GPTs in their respective fields, such as Salesforce's "EinsteinGPT" (for CRM) and Bloomberg's "BloombergGPT" (for finance).


Generative pre-trained transformer

History Generative pretraining (GP) was a long-established concept in machine learning applications, but the transformer architecture was not available until 2017 when it was invented by employees at Google. That development led to the emergence of large language models such as BERT in 2018 which was a pre-trained transformer (PT) but not designed to be generative (BERT was an "encoder-only" model). Also around that time, in 2018, OpenAI published its article entitled "Improving Language Understanding by Generative Pre-Training," in which it introduced the first generative pre-trained transformer (GPT) system ("GPT-1"). Prior to transformer-based architectures, the best-performing neural NLP (natural language processing) models commonly employed supervised learning from large amounts of manually-labeled data. The reliance on supervised learning limited their use on datasets that were not well-annotated, and also made it prohibitively expensive and time-consuming to train extremely large language models. The semi-supervised approach OpenAI employed to make a large-scale generative system���and was first to do with a transformer model���involved two stages: an unsupervised generative "pretraining" stage to set initial parameters using a language modeling objective, and a supervised discriminative "fine-tuning" stage to adapt these parameters to a target task. Regarding more recent GPT foundation models, OpenAI published its first versions of GPT-3 in July 2020. There were three models, with 1B, 6.7B, 175B parameters, respectively named babbage, curie, and davinci (giving initials B, C, and D). In July 2021, OpenAI published Codex, a task-specific GPT model targeted for programming applications. This was developed by fine-tuning a 12B parameter version of GPT-3 (different from previous GPT-3 models) using code from GitHub. In March 2022, OpenAI published two versions of GPT-3 that were fine-tuned for instruction-following (instruction-tuned), named davinci-instruct-beta (175B) and text-davinci-001., and then started beta testing code-davinci-002. text-davinci-002 was instruction-tuned from code-davinci-002. Both text-davinci-003 and ChatGPT were released in November 2022, with both building upon text-davinci-002 via reinforcement learning from human feedback (RLHF). text-davinci-003 is trained for following instructions (like its predecessors), whereas ChatGPT is further trained for conversational interaction with a human user. OpenAI's most recent GPT foundation model, GPT-4, was released on March 14, 2023. It can be accessed directly by users via a premium version of ChatGPT, and is available to developers for incorporation into other products and services via OpenAI's API. Other producers of GPT foundation models include EleutherAI (with a series of models starting in March 2021) and Cerebras (with seven models released in March 2023).


Generative pre-trained transformer

Foundational_models A foundational model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks. Thus far, the most notable GPT foundation models have been from OpenAI's GPT-n series. The most recent from that is GPT-4, for which OpenAI declined to publish the size or training details (citing "the competitive landscape and the safety implications of large-scale models"). Other such models include Google's PaLM, a broad foundation model that has been compared to GPT-3 and has recently been made available to developers via an API, and Together's GPT-JT, which has been reported as the closest-performing open-source alternative to GPT-3 (and is derived from earlier open-source GPTs). Meta AI (formerly Facebook) also has a generative transformer-based foundational large language model, known as LLaMA. Foundational GPTs can also employ modalities other than text, for input and/or output. GPT-4 is a multi-modal LLM that is capable of processing text and image input (though its output is limited to text). Regarding multimodal output, some generative transformer-based models are used for text-to-image technologies such as diffusion and parallel decoding. Such kinds of models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.


Generative pre-trained transformer

Task-specific_models A foundational GPT model can be further adapted to produce more targeted systems directed to specific tasks and/or subject-matter domains.  Methods for such adaptation can include additional fine-tuning (beyond that done for the foundation model) as well as certain forms of prompt engineering.  An important example of this is fine-tuning models to follow instructions, which is of course a fairly broad task but more targeted than a foundation model.  In January 2022, OpenAI introduced "InstructGPT"���a series of models which were fine-tuned to follow instructions using a combination of supervised training and reinforcement learning from human feedback (RLHF) on base GPT-3 language models.  Advantages this had over the bare foundational models included higher accuracy, less negative/toxic sentiment, and generally better alignment with user needs.  Hence, OpenAI began using this as the basis for its API service offerings.  Other instruction-tuned models have been released by others, including a fully open version.  Another (related) kind of task-specific models are chatbots, which engage in human-like conversation.  In November 2022, OpenAI launched ChatGPT���an online chat interface powered by an instruction-tuned language model trained in a similar fashion to InstructGPT.  They trained this model using RLHF, with human AI trainers providing conversations in which they played both the user and the AI, and mixed this new dialogue dataset with the InstructGPT dataset for a conversational format suitable for a chatbot.  Other major chatbots currently include Microsoft's Bing Chat, which uses OpenAI's GPT-4 (as part of a broader close collaboration between OpenAI and Microsoft), and Google's competing chatbot Bard (initially based on their LaMDA family of conversation-trained language models, with plans to switch to PaLM).  Yet another kind of task that a GPT can be used for is the meta-task of generating its own instructions, like developing a series of prompts for 'itself' to be able to effectuate a more general goal given by a human user.  This is known as an AI agent, and more specifically a recursive one because it uses results from its previous self-instructions to help it form its subsequent prompts; the first major example of this was Auto-GPT (which uses OpenAI's GPT models), and others have since been developed as well.  Generative transformer-based systems can also be targeted to tasks involving modalities beyond text.  For example, Microsoft���s ���Visual ChatGPT��� combines ChatGPT with visual foundation models (VFMs) to enable input or output comprising images as well as text.  Also, advances in text-to-speech technology offer powerful tools for audio content creation when used in conjunction with foundational GPT language models.  GPT systems can be directed toward particular fields or domains.


Generative pre-trained transformer

Task-specific_models Some reported examples of such models and apps are as follows: EinsteinGPT - for sales and marketing domains, to aid with customer relationship management (uses GPT-3.5) BloombergGPT - for the financial domain, to aid with financial news and information (uses "freely available" AI methods, combined with their proprietary data) Khanmigo ��� described as a GPT version for tutoring, in the education domain, it aids students using Khan Academy by guiding them through their studies without directly providing answers (powered by GPT-4) SlackGPT - for the Slack instant-messaging service, to aid with navigating and summarizing discussions on it (uses OpenAI's API) BioGPT - for the biomedical domain, to aid with biomedical literature text generation and mining (uses GPT-2)Sometimes domain-specificity is accomplished via software plug-ins or add-ons.  For example, several different companies have developed particular plugins that interact directly with OpenAI's ChatGPT interface, and Google Workspace has available add-ons such as ���GPT for Sheets and Docs������which is reported to aid use of spreadsheet functionality in Google Sheets.


Generative pre-trained transformer

Brand_issues OpenAI, which created the first generative pre-trained transformer (GPT) in 2018, has recently asserted that ���GPT��� should be regarded as a brand of OpenAI. In April 2023, OpenAI revised the brand guidelines in its terms of service to indicate that other businesses using its API to run their artificial intelligence (AI) services would no longer be able to include ���GPT��� in such names or branding. In May 2023, OpenAI engaged a brand management service to notify its API customers of this policy, although these notifications stopped short of making overt legal claims (such as allegations of trademark infringement or demands to cease and desist). Relatedly, OpenAI has applied to the United States Patent and Trademark Office (USPTO) to seek domestic trademark registration for the term ���GPT��� in the field of AI. OpenAI sought to expedite handling of its application, but the USPTO declined that request in April 2023. To get the trademark approved, OpenAI would need to establish that the term is actually ���distinctive��� to their specific offerings rather than widely understood as a broader technical term for the kind of technology. Some media reports suggested that OpenAI may be able to do so based indirectly on the fame of its GPT-based chatbot product, ChatGPT, for which OpenAI has separately sought trademark protection (and which it has sought to enforce more strongly). Other reports indicated that exclusivity for the bare term ���GPT��� seems unlikely to be granted, as it is used frequently to refer simply to AI systems that involve generative pre-trained transformers. If exclusive rights in the term were to be granted for the U.S., then everyone else in the U.S. using it in the name or branding of their related offerings would need to stop unless they have permission. Even if that were to occur, the trademark doctrine of descriptive fair use could still preserve some room to continue non-brand-related usage.


Generative pre-trained transformer

Selected_bibliography This section lists the main official publications from OpenAI and Microsoft on their GPT models. GPT-1: report, GitHub release. GPT-2: blog announcement, report on its decision of "staged release", GitHub release. GPT-3: report. No GitHub or any other form of code release thenceforth. webGPT: blog announcement, report, InstructGPT: blog announcement, report. ChatGPT: blog announcement (no report). GPT-4: blog announcement, reports, model card.


GPT-3

GPT-3 Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020. Like its predecessor GPT-2, it is a decoder-only transformer model of deep neural network, which uses attention in place of previous recurrence- and convolution-based architectures. Attention mechanisms allow the model to selectively focus on segments of input text it predicts to be the most relevant. It uses a 2048-tokens-long context and then-unprecedented size of 175 billion parameters, requiring 800GB to store. The model demonstrated strong zero-shot and few-shot learning on many tasks. Microsoft announced on September 22, 2020, that it had licensed "exclusive" use of GPT-3; others can still use the public API to receive output, but only Microsoft has access to GPT-3's underlying model.


GPT-3

Background According to The Economist, improved algorithms, powerful computers, and an increase in digitized data have fueled a revolution in machine learning, with new techniques in the 2010s resulting in "rapid improvements in tasks" including manipulating language. Software models are trained to learn by using thousands or millions of examples in a "structure ... loosely based on the neural architecture of the brain". One architecture used in natural language processing (NLP) is a neural network based on a deep learning model that was first introduced in 2017���the transformer architecture. There are a number of NLP systems capable of processing, mining, organizing, connecting and contrasting textual input, as well as correctly answering questions. On June 11, 2018, OpenAI researchers and engineers posted their original paper introducing the first generative pre-trained transformer (GPT)���a type of generative large language model that is pre-trained with an enormous and diverse corpus of text via datasets, followed by discriminative fine-tuning to focus on a specific task. GPT models are transformer-based deep learning neural network architectures. Up to that point, the best-performing neural NLP models commonly employed supervised learning from large amounts of manually-labeled data, which made it prohibitively expensive and time-consuming to train extremely large language models. That first GPT model is known as "GPT-1," and it was then followed by "GPT-2" in February 2019. GPT-2 was created as a direct scale-up of GPT-1, with both its parameter count and dataset size increased by a factor of 10. It had 1.5 billion parameters, and was trained on a dataset of 8 million web pages. In February 2020, Microsoft introduced its Turing Natural Language Generation (T-NLG), which was claimed to be the "largest language model ever published at 17 billion parameters." It performed better than any other language model at a variety of tasks which included summarizing texts and answering questions.


GPT-3

Training_and_capabilities The construct of ���learning styles��� is problematic because it fails to account for the processes through which learning styles are shaped.  Some students might develop a particular learning style because they have had particular experiences.  Others might develop a particular learning style by trying to accommodate to a learning environment that was not well suited to their learning needs.  Ultimately, we need to understand the interactions among learning styles and environmental and personal factors, and how these shape how we learn and the kinds of learning we experience.  ��� Text generated by Mike Sharples On May 28, 2020, an arXiv preprint by a group of 31 engineers and researchers at OpenAI described the achievement and development of GPT-3, a third-generation "state-of-the-art language model".  The team increased the capacity of GPT-3 by over two orders of magnitude from that of its predecessor, GPT-2, making GPT-3 the largest non-sparse language model to date.  Because GPT-3 is structurally similar to its predecessors, its greater accuracy is attributed to its increased capacity and greater number of parameters.  GPT-3's capacity is ten times larger than that of Microsoft's Turing NLG, the next largest NLP model known at the time.  Lambdalabs estimated a hypothetical cost of around $4.6 million US dollars and 355 years to train GPT-3 on a single GPU in 2020, with lower actual training time by using more GPUs in parallel.  Sixty percent of the weighted pre-training dataset for GPT-3 comes from a filtered version of Common Crawl consisting of 410 billion byte-pair-encoded tokens.  Other sources are 19 billion tokens from WebText2 representing 22% of the weighted total, 12 billion tokens from Books1 representing 8%, 55 billion tokens from Books2 representing 8%, and 3 billion tokens from Wikipedia representing 3%.  GPT-3 was trained on hundreds of billions of words and is also capable of coding in CSS, JSX, and Python, among others.  Since GPT-3's training data was all-encompassing, it does not require further training for distinct language tasks.  The training data contains occasional toxic language and GPT-3 occasionally generates toxic language as a result of mimicking its training data.  A study from the University of Washington found that GPT-3 produced toxic language at a toxicity level comparable to the similar natural language processing models of GPT-2 and CTRL.  OpenAI has implemented several strategies to limit the amount of toxic language generated by GPT-3.  As a result, GPT-3 produced less toxic language compared to its predecessor model, GPT-1, although it produced both more generations and a higher toxicity of toxic language compared to CTRL Wiki, a language model trained entirely on Wikipedia data.  On June 11, 2020, OpenAI announced that users could request access to its user-friendly GPT-3 API���a "machine learning toolset"���to help OpenAI "explore the strengths and limits" of this new technology.  The invitation described how this API had a general-purpose "text in, text out" interface that can complete almost "any English language task", instead of the usual single use-case.