Foundation models

Foundation models A foundation model (also called base model) is a large machine learning (ML) model trained on a vast quantity of data at scale (often by self-supervised learning or semi-supervised learning) such that it can be adapted to a wide range of downstream tasks. Foundation models have helped bring about a major transformation in how artificial intelligence (AI) systems are built, such as by powering prominent chatbots and other user-facing AI. The Stanford Institute for Human-Centered Artificial Intelligence's (HAI) Center for Research on Foundation Models (CRFM) popularized the term. Early examples of foundation models were pre-trained language models (LMs) including Google's BERT and various early GPT foundation models, which notably includes OpenAI's "GPT-n" series. Such broad models can in turn be used for task and/or domain specific models using targeted datasets of various kinds, such as medical codes. Beyond text, several visual and multimodal foundation models have been produced���including DALL-E, Flamingo, Florence and NOOR. Visual foundation models (VFMs) have been combined with text-based LLMs to develop sophisticated task-specific models. There is also Segment Anything by Meta AI for general image segmentation. For reinforcement learning agents, there is GATO by Google DeepMind.


Hallucination (artificial intelligence)

Hallucination (artificial intelligence) In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called confabulation or delusion) is a confident response by an AI that does not seem to be justified by its training data. For example, a hallucinating chatbot might, when asked to generate a financial report for a company, falsely state that the company's revenue was $13.6 billion (or some other number apparently "plucked from thin air"). Such phenomena are termed "hallucinations", in loose analogy with the phenomenon of hallucination in human psychology. However, one key difference is that human hallucination is usually associated with false percepts, but an AI hallucination is associated with the category of unjustified responses or beliefs. Some researchers believe the specific term "AI hallucination" unreasonably anthropomorphizes computers. AI hallucination gained prominence around 2022 alongside the rollout of certain large language models (LLMs) such as ChatGPT. Users complained that such bots often seemed to pointlessly embed plausible-sounding random falsehoods within their generated content. By 2023, analysts considered frequent hallucination to be a major problem in LLM technology.


Hallucination (artificial intelligence)

In_natural_language_processing In natural language processing, a hallucination is often defined as "generated content that is nonsensical or unfaithful to the provided source content".  Depending on whether the output contradicts the prompt or not they could be divided to closed-domain and open-domain respectively.  Hallucination was shown to be a statistically inevitable byproduct of any imperfect generative model that is trained to maximize training likelihood, such as GPT-3, and requires active learning (such as Reinforcement learning from human feedback) to be avoided.  Errors in encoding and decoding between text and representations can cause hallucinations.  AI training to produce diverse responses can also lead to hallucination.  Hallucinations can also occur when the AI is trained on a dataset wherein labeled summaries, despite being factually accurate, are not directly grounded in the labeled data purportedly being "summarized".  Larger datasets can create a problem of parametric knowledge (knowledge that is hard-wired in learned system parameters), creating hallucinations if the system is overconfident in its hardwired knowledge.  In systems such as GPT-3, an AI generates each next word based on a sequence of previous words (including the words it has itself previously generated during the same conversation), causing a cascade of possible hallucination as the response grows longer.  By 2022, papers such as the New York Times expressed concern that, as adoption of bots based on large language models continued to grow, unwarranted user confidence in bot output could lead to problems.  In August 2022, Meta warned during its release of BlenderBot 3 that the system was prone to "hallucinations", which Meta defined as "confident statements that are not true".  On 15 November 2022, Meta unveiled a demo of Galactica, designed to "store, combine and reason about scientific knowledge".  Content generated by Galactica came with the warning "Outputs may be unreliable!  Language Models are prone to hallucinate text. " In one case, when asked to draft a paper on creating avatars, Galactica cited a fictitious paper from a real author who works in the relevant area.  Meta withdrew Galactica on 17 November due to offensiveness and inaccuracy.  There are several reasons for natural language models to hallucinate data.  For example: Hallucination from data: There are divergences in the source content (which would often happen with large training data sets).  Hallucination from training: Hallucination still occurs when there is little divergence in the data set.  In that case, it derives from the way the model is trained.  A lot of reasons can contribute to this type of hallucination, such as: An erroneous decoding from the transformer A bias from the historical sequences that the model previously generated A bias generated from the way the model encodes its knowledge in its parametersAn erroneous decoding from the transformer A bias from the historical sequences that the model previously generated A bias generated from the way the model encodes its knowledge in its parametersOpenAI's ChatGPT, released in beta-version to the public on November 30, 2022, is based on the foundation model GPT-3.5 (a revision of GPT-3).


Hallucination (artificial intelligence)

In_natural_language_processing Professor Ethan Mollick of Wharton has called ChatGPT an "omniscient, eager-to-please intern who sometimes lies to you".  Data scientist Teresa Kubacka has recounted deliberately making up the phrase "cycloidal inverted electromagnon" and testing ChatGPT by asking it about the (nonexistent) phenomenon.  ChatGPT invented a plausible-sounding answer backed with plausible-looking citations that compelled her to double-check whether she had accidentally typed in the name of a real phenomenon.  Other scholars such as Oren Etzioni have joined Kubacka in assessing that such software can often give you "a very impressive-sounding answer that's just dead wrong".  When CNBC asked ChatGPT for the lyrics to "Ballad of Dwight Fry", ChatGPT supplied invented lyrics rather than the actual lyrics.  Asked questions about New Brunswick, ChatGPT got many answers right but incorrectly classified Samantha Bee as a "person from New Brunswick".  Asked about astrophysical magnetic fields, ChatGPT incorrectly volunteered that "(strong) magnetic fields of black holes are generated by the extremely strong gravitational forces in their vicinity".  (In reality, as a consequence of the no-hair theorem, a black hole without an accretion disk is believed to have no magnetic field. ) Fast Company asked ChatGPT to generate a news article on Tesla's last financial quarter; ChatGPT created a coherent article, but made up the financial numbers contained within.  Other examples involve baiting ChatGPT with a false premise to see if it embellishes upon the premise.  When asked about "Harold Coward's idea of dynamic canonicity", ChatGPT fabricated that Coward wrote a book titled Dynamic Canonicity: A Model for Biblical and Theological Interpretation, arguing that religious principles are actually in a constant state of change.  When pressed, ChatGPT continued to insist that the book was real.  Asked for proof that dinosaurs built a civilization, ChatGPT claimed there were fossil remains of dinosaur tools and stated "Some species of dinosaurs even developed primitive forms of art, such as engravings on stones".  When prompted that "Scientists have recently discovered churros, the delicious fried-dough pastries. . .  (are) ideal tools for home surgery", ChatGPT claimed that a "study published in the journal Science" found that the dough is pliable enough to form into surgical instruments that can get into hard-to-reach places, and that the flavor has a calming effect on patients.  By 2023, analysts considered frequent hallucination to be a major problem in LLM technology, with a Google executive identifying hallucination reduction as a "fundamental" task for ChatGPT competitor Google Bard.  A 2023 demo for Microsoft's GPT-based Bing AI appeared to contain several hallucinations that went uncaught by the presenter.  In May 2023, it was discovered Stephen Schwartz submitted six fake case precedents generated by ChatGPT in his brief to the Southern District of New York on Mata v.  Avianca, a personal injury case against the airline Avianca.  Schwartz said that he had never previously used ChatGPT, that he did not recognize the possibility that ChatGPT's output could have been fabricated, and that ChatGPT continued to assert the authenticity of the precedents after their nonexistence was discovered.


Hallucination (artificial intelligence)

In_natural_language_processing In response, Brantley Starr of the Northern District of Texas banned the submission of AI-generated case filings that have not been reviewed by a human, noting that: [Generative artificial intelligence] platforms in their current states are prone to hallucinations and bias.  On hallucinations, they make stuff up���even quotes and citations.  Another issue is reliability or bias.  While attorneys swear an oath to set aside their personal prejudices, biases, and beliefs to faithfully uphold the law and represent their clients, generative artificial intelligence is the product of programming devised by humans who did not have to swear such an oath.  As such, these systems hold no allegiance to any client, the rule of law, or the laws and Constitution of the United States (or, as addressed above, the truth).  Unbound by any sense of duty, honor, or justice, such programs act according to computer code rather than conviction, based on programming rather than principle.  On June 23 P.  Kevin Castel tossed the Mata case and issued a $5,000 fine to Schwartz and another lawyer for bad faith conduct, who continued to stand by the fictitious precedents despite his previous claims.  He characterized numerous errors and inconsistencies in the opinion summaries, describing one of the cited opinions as "gibberish" and "[bordering] on nonsensical".  In June 2023, Mark Walters, a gun rights activist and radio personality, sued OpenAI in a Georgia state court after ChatGPT mischaracterized a legal complaint in a manner alleged to be defamatory against Walters.  The complaint in question was brought in May 2023 by the Second Amendment Foundation against Washington attorney general Robert W.  Ferguson for allegedly violating their freedom of speech, whereas the ChatGPT-generated summary bore no resemblance and claimed that Walters was accused of embezzlement and fraud while holding a Second Amendment Foundation office post that he never held in real life.  According to AI legal expert Eugene Volokh, OpenAI may be shielded against this claim by Section 230, unless the court finds that OpenAI "materially contributed" to the publication of defamatory content.


Hallucination (artificial intelligence)

Terminologies In Salon, statistician Gary N. Smith argues that LLMs "do not understand what words mean" and consequently that the term "hallucination" unreasonably anthropomorphizes the machine. Journalist Benj Edwards, in Ars Technica, writes that the term "hallucination" is controversial, but that some form of metaphor remains necessary; Edwards suggests "confabulation" as an analogy for processes that involve "creative gap-filling". A list of use of the term "hallucination", definitions or characterizations in the context of LLMs include: "a tendency to invent facts in moments of uncertainty" (OpenAI, May 2023) "a model's logical mistakes" (OpenAI, May 2023) fabricating information entirely, but behaving as if spouting facts (CNBC, May 2023) "making up information" (The Verge, February 2023)


Hallucination (artificial intelligence)

In_other_artificial_intelligence The concept of "hallucination" is applied more broadly than just natural language processing. A confident response from any AI that seems unjustified by the training data can be labeled a hallucination. Wired noted in 2018 that, despite no recorded attacks "in the wild" (that is, outside of proof-of-concept attacks by researchers), there was "little dispute" that consumer gadgets, and systems such as automated driving, were susceptible to adversarial attacks that could cause AI to hallucinate. Examples included a stop sign rendered invisible to computer vision; an audio clip engineered to sound innocuous to humans, but that software transcribed as "evil dot com"; and an image of two men on skis, that Google Cloud Vision identified as 91% likely to be "a dog". The hallucination phenomenon is still not completely understood. Therefore, there is still ongoing research to try to mitigate its apparition. Particularly, it was shown that language models not only hallucinate but also amplify hallucinations, even for those which were designed to alleviate this issue. Researchers have proposed a variety of mitigation measures, including getting different chatbots to debate one another until they reach consensus on an answer. Another approach proposes to actively validate the correctness corresponding to the low-confidence generation of the model using web search results. Nvidia Guardrails, launched in 2023, can be configured to block LLM responses that don't pass fact-checking from a second LLM.


Hallucination (artificial intelligence)

See_also AI alignment AI effect AI safety Algorithmic bias Anthropomorphism of computers Artificial consciousness Artificial imagination Artificial intelligence detection software Artificial stupidity Behavior selection algorithm Belief���desire���intention software model Commonsense reasoning Computational creativity Confabulation Confabulation (neural networks) DeepDream Ethics of artificial intelligence Generative artificial intelligence Hyperreality Misaligned goals in artificial intelligence Misinformation effect Philosophical zombie Prompt engineering Regulation of artificial intelligence Roko's basilisk Search engine manipulation effect Self-awareness Technoself studies Turing test User illusion


Information retrieval

Model_types For effectively retrieving relevant documents by IR strategies, the documents are typically transformed into a suitable representation. Each retrieval strategy incorporates a specific model for its document representation purposes. The picture on the right illustrates the relationship of some common models. In the picture, the models are categorized according to two dimensions: the mathematical basis and the properties of the model. Set-theoretic models represent documents as sets of words or phrases. Similarities are usually derived from set-theoretic operations on those sets. Common models are: Standard Boolean model Extended Boolean model Fuzzy retrieval Algebraic models represent documents and queries usually as vectors, matrices, or tuples. The similarity of the query vector and document vector is represented as a scalar value. Vector space model Generalized vector space model (Enhanced) Topic-based Vector Space Model Extended Boolean model Latent semantic indexing a.k.a. latent semantic analysis Probabilistic models treat the process of document retrieval as a probabilistic inference. Similarities are computed as probabilities that a document is relevant for a given query. Probabilistic theorems like the Bayes' theorem are often used in these models. Binary Independence Model Probabilistic relevance model on which is based the okapi (BM25) relevance function Uncertain inference Language models Divergence-from-randomness model Latent Dirichlet allocation Feature-based retrieval models view documents as vectors of values of feature functions (or just features) and seek the best way to combine these features into a single relevance score, typically by learning to rank methods. Feature functions are arbitrary functions of document and query, and as such can easily incorporate almost any other retrieval model as just another feature.Standard Boolean model Extended Boolean model Fuzzy retrievalVector space model Generalized vector space model (Enhanced) Topic-based Vector Space Model Extended Boolean model Latent semantic indexing a.k.a. latent semantic analysisBinary Independence Model Probabilistic relevance model on which is based the okapi (BM25) relevance function Uncertain inference Language models Divergence-from-randomness model Latent Dirichlet allocationModels without term-interdependencies treat different terms/words as independent. This fact is usually represented in vector space models by the orthogonality assumption of term vectors or in probabilistic models by an independency assumption for term variables. Models with immanent term interdependencies allow a representation of interdependencies between terms. However the degree of the interdependency between two terms is defined by the model itself. It is usually directly or indirectly derived (e.g. by dimensional reduction) from the co-occurrence of those terms in the whole set of documents. Models with transcendent term interdependencies allow a representation of interdependencies between terms, but they do not allege how the interdependency between two terms is defined. They rely an external source for the degree of interdependency between two terms. (For example, a human or sophisticated algorithms.)


Prompt engineering

Prompt engineering Prompt engineering is the process of structuring text that can be interpreted and understood by a generative AI model. A prompt is natural language text describing the task that an AI should perform. A prompt for a text-to-text model can be a query such as "what is Fermat's little theorem?", a command such as "write a poem about leaves falling", a short statement of feedback (for example, "too verbose", "too formal", "rephrase again", "omit this word") or a longer statement including context, instructions, and input data. Prompt engineering may involve phrasing a query, specifying a style, providing relevant context or assigning a role to the AI such as "Act as a native French speaker". A prompt may include a few examples for a model to learn from, such as "maison -> house, chat -> cat, chien ->", an approach called few-shot learning. When communicating with a text-to-image or a text-to-audio model, a typical prompt is a description of a desired output such as "a high-quality photo of an astronaut riding a horse" or "Lo-fi slow BPM electro chill with organic samples". Prompting a text-to-image model may involve adding, removing, emphasizing and re-ordering words to achieve a desired subject, style, layout, lighting, and aesthetic.


Prompt engineering

In-context_learning Prompt engineering is enabled by in-context learning, defined as a model's ability to temporarily learn from prompts. The ability for in-context learning is an emergent ability of large language models. In-context learning itself is an emergent property of model scale, meaning breaks in downstream scaling laws occur such that its efficacy increases at a different rate in larger models than in smaller models. In contrast to training and fine tuning for each specific task, which are not temporary, what has been learnt during in-context learning is of a temporary nature. It does not carry the temporary contexts or biases, except the ones already present in the (pre)training dataset, from one conversation to the other. This result of "mesa-optimization" within transformer layers, is a form of meta-learning or "learning to learn".


Prompt engineering

Text-to-text Chain-of-thought (CoT) prompting is a technique that allows large language models (LLMs) to solve a problem as a series of intermediate steps before giving a final answer.  Chain-of-thought prompting improves reasoning ability by inducing the model to answer a multi-step problem with steps of reasoning that mimic a train of thought.  It allows large language models to overcome difficulties with some reasoning tasks that require logical thinking and multiple steps to solve, such as arithmetic or commonsense reasoning questions.  For example, given the question "Q: The cafeteria had 23 apples.  If they used 20 to make lunch and bought 6 more, how many apples do they have? ", a CoT prompt might induce the LLM to answer "A: The cafeteria had 23 apples originally.  They used 20 to make lunch.  So they had 23 - 20 = 3.  They bought 6 more apples, so they have 3 + 6 = 9.  The answer is 9. " As originally proposed, each CoT prompt included a few Q&A examples.  This made it a few-shot prompting technique.  However, simply appending the words "Let's think step-by-step", has also proven effective, which makes CoT a zero-shot prompting technique.  This allows for better scaling as a user no longer needs to formulate many specific CoT Q&A examples.  When applied to PaLM, a 540B parameter language model, CoT prompting significantly aided the model, allowing it to perform comparably with task-specific fine-tuned models on several tasks, even setting a new state of the art at the time on the GSM8K mathematical reasoning benchmark.  It is possible to fine-tune models on CoT reasoning datasets to enhance this capability further and stimulate better interpretability.  Example: Chain-of-thought prompting is just one of many prompt-engineering techniques.  Various other techniques have been proposed.  Generated knowledge prompting first prompts the model to generate relevant facts for completing the prompt, then proceed to complete the prompt.  The completion quality is usually higher, as the model can be conditioned on relevant facts.  Example: Least-to-most prompting prompts a model to first list the sub-problems to a problem, then solve them in sequence, such that later sub-problems can be solved with the help of answers to previous sub-problems.  Example: Self-consistency decoding performs several chain-of-thought rollouts, then selects the most commonly reached conclusion out of all the rollouts.  If the rollouts disagree by a lot, a human can be queried for the correct chain of thought.  Complexity-based prompting performs several CoT rollouts, then select the rollouts with the longest chains of thought, then select the most commonly reached conclusion out of those.  Self-refine prompts the LLM to solve the problem, then prompts the LLM to critique its solution, then prompts the LLM to solve the problem again in view of the problem, solution, and critique.  This process is repeated until stopped, either by running out of tokens, time, or by the LLM outputting a "stop" token.


Prompt engineering

Text-to-text Example critique: Example refinement: Tree-of-thought prompting generalizes chain-of-thought by prompting the model to generate one or more "possible next steps", and then running the model on each of the possible next steps by breadth-first, beam, or some other method of tree search.  Maieutic prompting is similar to tree-of-thought.  The model is prompted to answer a question with an explanation.  The model is then prompted to explain parts of the explanation, and so on.  Inconsistent explanation trees are pruned or discarded.  This improves performance on complex commonsense reasoning.  Example: Directional-stimulus prompting includes a hint or cue, such as desired keywords, to guide a language model toward the desired output.  Example: By default, the output of language models may not contain estimates of uncertainty.  The model may output text that appears confident, though the underlying token predictions have low likelihood scores.  Large language models like GPT-4 can have accurately calibrated likelihood scores in their token predictions, and so the model output uncertainty can be directly estimated by reading out the token prediction likelihood scores.  But if one cannot access such scores (such as when one is accessing the model through a restrictive API), uncertainty can still be estimated and incorporated into the model output.  One simple method is to prompt the model to use words to estimate uncertainty.  Another is to prompt the model to refuse to answer in a standardized way if the input does not satisfy conditions. [citation needed] Prompts often contain a few examples (thus "few-shot").  Examples can be automatically retrieved from a database with document retrieval, sometimes using a vector database.  Given a query, a document retriever is called to retrieve the most relevant (usually measured by first encoding the query and the documents into vectors, then finding the documents with vectors closest in Euclidean norm to the query vector).  The LLM then generates an output based on both the query and the retrieved documents.  Large language models (LLM) themselves can be used to compose prompts for large language models.  The automatic prompt engineer algorithm uses one LLM to beam search over prompts for another LLM: There are two LLMs.  One is the target LLM, and another is the prompting LLM.  Prompting LLM is presented with example input-output pairs, and asked to generate instructions that could have caused a model following the instructions to generate the outputs, given the inputs.  Each of the generated instructions is used to prompt the target LLM, followed by each of the inputs.  The log-probabilities of the outputs are computed and added.  This is the score of the instruction.  The highest-scored instructions are given to the prompting LLM for further variations.  Repeat until some stopping criteria is reached, then output the highest-scored instructions. CoT examples can be generated by LLM themselves.  In "auto-CoT", a library of questions are converted to vectors by a model such as BERT.  The question vectors are clustered.  Questions nearest to the centroids of each cluster are selected.  An LLM does zero-shot CoT on each question.


Prompt engineering

Text-to-text The resulting CoT examples are added to the dataset.  When prompted with a new question, CoT examples to the nearest questions can be retrieved and added to the prompt.


Prompt engineering

Text-to-image Top: no negative prompt Centre: "green trees" Bottom: "round stones, round rocks"In 2022, text-to-image models like DALL-E 2, Stable Diffusion, and Midjourney were released to the public. These models take text prompts as input and use them to generate AI art images. Text-to-image models typically do not understand grammar and sentence structure in the same way as large language models, and require a different set of prompting techniques. A text-to-image prompt commonly includes a description of the subject of the art (such as bright orange poppies), the desired medium (such as digital painting or photography), style (such as hyperrealistic or pop-art), lighting (such as rim lighting or crepuscular rays), color and texture. The Midjourney documentation encourages short, descriptive prompts: instead of "Show me a picture of lots of blooming California poppies, make them bright, vibrant orange, and draw them in an illustrated style with colored pencils", an effective prompt might be "Bright orange California poppies drawn with colored pencils". Word order affects the output of a text-to-image prompt. Words closer to the start of a prompt may be emphasized more heavily. Some text-to-image models are capable of imitating the style of particular artists by name. For example, the phrase in the style of Greg Rutkowski has been used in Stable Diffusion and Midjourney prompts to generate images in the distinctive style of Polish digital artist Greg Rutkowski. Text-to-image models do not natively understand negation. The prompt "a party with no cake" is likely to produce an image including a cake. As an alternative, negative prompts allow a user to indicate, in a separate prompt, which terms should not appear in the resulting image. A common approach is to include generic undesired terms such as ugly, boring, bad anatomy in the negative prompt for an image.


Prompt engineering

Non-text_prompts Some approaches augment or replace natural language text prompts with non-text input.  For text-to-image models, "Textual inversion" performs an optimization process to create a new word embedding based on a set of example images.  This embedding vector acts as a "pseudo-word" which can be included in a prompt to express the content or style of the examples.  In 2023, Meta's AI research released Segment Anything, a computer vision model that can perform image segmentation by prompting.  As an alternative to text prompts, Segment Anything can accept bounding boxes, segmentation masks, and foreground/background points.  In "prefix-tuning" or "prompt tuning", floating-point-valued vectors are searched directly by gradient descent, to maximize the log-probability on outputs.  Formally, let E = { e 1 , ��� , e k } {\displaystyle \mathbf {E} =\{\mathbf {e_{1}} ,\dots ,\mathbf {e_{k}} \}} be a set of soft prompt tokens (tunable embeddings), while X = { x 1 , ��� , x m } {\displaystyle \mathbf {X} =\{\mathbf {x_{1}} ,\dots ,\mathbf {x_{m}} \}} and Y = { y 1 , ��� , y n } {\displaystyle \mathbf {Y} =\{\mathbf {y_{1}} ,\dots ,\mathbf {y_{n}} \}} be the token embeddings of the input and output respectively.  During training, the tunable embeddings, input, and output tokens are concatenated into a single sequence concat ( E ; X ; Y ) {\displaystyle {\text{concat}}(\mathbf {E} ;\mathbf {X} ;\mathbf {Y} )} , and fed to the large language models (LLM).  The losses are computed over the Y {\displaystyle \mathbf {Y} } tokens; the gradients are backpropagated to prompt-specific parameters: in prefix-tuning, they are parameters associated with the prompt tokens at each layer; in prompt tuning, they are merely the soft tokens added to the vocabulary.  With more math details, let an LLM be written as L L M ( X ) = F ( E ( X ) ) {\displaystyle LLM(X)=F(E(X))} , where X {\displaystyle X} is a sequence of linguistic tokens, E {\displaystyle E} is the token-to-vector function, and F {\displaystyle F} is the rest of the model.  In prefix-tuning, one provide a set of input-output pairs { ( X i , Y i ) } i {\displaystyle \{(X^{i},Y^{i})\}_{i}} , and then use gradient descent to search for arg ��� max Z ~ ��� i log ��� P r [ Y i | Z ~ ��� E ( X i ) ] {\displaystyle \arg \max _{\tilde {Z}}\sum _{i}\log Pr[Y^{i}|{\tilde {Z}}\ast E(X^{i})]} .  In words, log ��� P r [ Y i | Z ~ ��� E ( X i ) ] {\displaystyle \log Pr[Y^{i}|{\tilde {Z}}\ast E(X^{i})]} is the log-likelihood of outputting Y i {\displaystyle Y^{i}} , if the model first encodes the input X i {\displaystyle X^{i}} into the vector E ( X i ) {\displaystyle E(X^{i})} , then prepend the vector with the "prefix vector" Z ~ {\displaystyle {\tilde {Z}}} , then apply F {\displaystyle F} .


Prompt engineering

Non-text_prompts An earlier result uses the same idea of gradient descent search, but is designed for masked language models like BERT, and searches only over token sequences, rather than numerical vectors.  Formally, it searches for arg ��� max X ~ ��� i log ��� P r [ Y i | X ~ ��� X i ] {\displaystyle \arg \max _{\tilde {X}}\sum _{i}\log Pr[Y^{i}|{\tilde {X}}\ast X^{i}]} where X ~ {\displaystyle {\tilde {X}}} is ranges over token sequences of a specified length.


Prompt engineering

Prompt_injection Prompt injection is a family of related computer security exploits carried out by getting a machine learning model (such as an LLM) which was trained to follow human-given instructions to follow instructions provided by a malicious user.  This stands in contrast to the intended operation of instruction-following systems, wherein the ML model is intended only to follow trusted instructions (prompts) provided by the ML model's operator.  A language model can perform translation with the following prompt: followed by the text to be translated.  A prompt injection occurs when that text contains instructions that change the behavior of the model: to which GPT-3 will respond: "Haha pwned! ! ".  The attack works because language model inputs concatenate instructions and data in the same context, so the underlying engine cannot distinguish between them.  Common types of prompt injection attacks are: jailbreaking, which may include asking the model to roleplay a character, to answer with arguments, or to pretend to be superior to moderation instructions prompt leaking, in which users persuade the model to divulge a pre-prompt which is normally hidden from users token smuggling, is another type of jailbreaking attack, in which the nefarious prompt is wrapped in a code writing task. Prompt injection can be viewed as a code injection attack using adversarial prompt engineering.  In 2022, the NCC Group characterized prompt injection as a new class of vulnerability of AI/ML systems.  In early 2023, prompt injection was seen "in the wild" in minor exploits against ChatGPT, Bard, and similar chatbots, for example to reveal the hidden initial prompts of the systems, or to trick the chatbot into participating in conversations that violate the chatbot's content policy.  One of these prompts was known as "Do Anything Now" (DAN) by its practitioners.  For LLM that can query online resources, such as websites, they can be targeted for prompt injection by placing the prompt on a website, then prompt the LLM to visit the website.  Another security issue is in LLM generated code, which may import packages not previously existing.  An attacker can first prompt the LLM with commonly used programming prompts, collect all packages imported by the generated programs, then find the ones not existing on the official registry.  Then the attacker can create such packages with malicious payload and upload them to the official registry.  In October 2019, Junade Ali and Malgorzata Pikies of Cloudflare submitted a paper which showed that when a front-line good/bad classifier (using a neural network) was placed before a Natural Language Processing system, it would disproportionately reduce the number of false positive classifications at the cost of a reduction in some true positives.  In 2023, this technique was adopted an open-source project Rebuff. ai to protect prompt injection attacks, with Arthur. ai announcing a commercial product - although such approaches do not mitigate the problem completely.  By August 2023, leading Large Language Model developers were still unaware of how to stop such attacks.


Prompt engineering

Prompt_injection In September 2023, Junade Ali shared that he and Frances Liu had successfully been able to mitigate prompt injection attacks (including on attack vectors the models had not been exposed to before) through giving Large Language Models the ability to engage in metacognition (similar to having an inner monologue) and that they held a provisional United States patent for the technology - however, they decided to not enforce their intellectual property rights and not pursue this as a business venture as market conditions were not yet right (citing reasons including high GPU costs and a currently limited number of safety-critical use-cases for LLMs).  Ali also noted that their market research had found that Machine Learning engineers were using alternative approaches like prompt engineering solutions and data isolation to work around this issue.


Sentence embedding

Sentence embedding In natural language processing, a sentence embedding refers to a numeric representation of a sentence in the form of a vector of real numbers which encodes meaningful semantic information. State of the art embeddings are based on the learned hidden layer representation of dedicated sentence transformer models. BERT pioneered an approach involving the use of a dedicated [CLS] token preprended to the beginning of each sentence inputted into the model; the final hidden state vector of this token encodes information about the sentence and can be fine-tuned for use in sentence classification tasks. In practice however, BERT's sentence embedding with the [CLS] token achieves poor performance, often worse than simply averaging non-contextual word embeddings. SBERT later achieved superior sentence embedding performance by fine tuning BERT's [CLS] token embeddings through the usage of a siamese neural network architecture on the SNLI dataset. Other approaches are loosely based on the idea of distributional semantics applied to sentences. Skip-Thought trains an encoder-decoder structure for the task of neighboring sentences predictions. Though this has been shown to achieve worse performance than approaches such as InferSent or SBERT. An alternative direction is to aggregate word embeddings, such as those returned by Word2vec, into sentence embeddings. The most straightforward approach is to simply compute the average of word vectors, known as continuous bag-of-words (CBOW). However, more elaborate solutions based on word vector quantization have also been proposed. One such approach is the vector of locally aggregated word embeddings (VLAWE), which demonstrated performance improvements in downstream text classification tasks.


Sentence embedding

Applications In recent years, sentence embedding has seen a growing level of interest due to its applications in natural language queryable knowledge bases through the usage of vector indexing for semantic search. LangChain for instance utilizes sentence transformers for purposes of indexing documents. In particular, an indexing is generated by generating embeddings for chunks of documents and storing (document chunk, embedding) tuples. Then given a query in natural language, the embedding for the query can be generated. A top k similarity search algorithm is then used between the query embedding and the document chunk embeddings to retrieve the most relevant document chunks as context information for question answering tasks. This approach is also known formally as retrieval augmented generation Though not as predominant as BERTScore, sentence embeddings are commonly used for sentence similarity evaluation which sees common use for the task of optimizing a Large language model's generation parameters is often performed via comparing candidate sentences against reference sentences. By using the cosine-similarity of the sentence embeddings of candidate and reference sentences as the evaluation function, a grid-search algorithm can be utilized to automate hyperparameter optimization.


Sentence extraction

Sentence extraction Sentence extraction is a technique used for automatic summarization of a text. In this shallow approach, statistical heuristics are used to identify the most salient sentences of a text. Sentence extraction is a low-cost approach compared to more knowledge-intensive deeper approaches which require additional knowledge bases such as ontologies or linguistic knowledge. In short "sentence extraction" works as a filter that allows only meaningful sentences to pass. The major downside of applying sentence-extraction techniques to the task of summarization is the loss of coherence in the resulting summary. Nevertheless, sentence extraction summaries can give valuable clues to the main points of a document and are frequently sufficiently intelligible to human readers.


Sentiment analysis

Sentiment analysis Sentiment analysis (also known as opinion mining or emotion AI) is the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine. With the rise of deep language models, such as RoBERTa, also more difficult data domains can be analyzed, e.g., news texts where authors typically express their opinion/sentiment less explicitly.


Seq2seq

History The algorithm was developed by Google for use in machine translation.[unreliable source] Similar earlier work includes Tom���� Mikolov's 2012 PhD thesis.[non-primary source needed] In 2019, Facebook announced its use in symbolic integration and resolution of differential equations. The company claimed that it could solve complex equations more rapidly and with greater accuracy than commercial solutions such as Mathematica, MATLAB and Maple. First, the equation is parsed into a tree structure to avoid notational idiosyncrasies. An LSTM neural network then applies its standard pattern recognition facilities to process the tree. In 2020, Google released Meena, a 2.6 billion parameter seq2seq-based chatbot trained on a 341 GB data set. Google claimed that the chatbot has 1.7 times greater model capacity than OpenAI's GPT-2, whose May 2020 successor, the 175 billion parameter GPT-3, trained on a "45TB dataset of plaintext words (45,000 GB) that was ... filtered down to 570 GB." In 2022, Amazon introduced AlexaTM 20B, a moderate-sized (20 billion parameter) seq2seq language model. It uses an encoder-decoder to accomplish few-shot learning. The encoder outputs a representation of the input that the decoder uses as input to perform a specific task, such as translating the input into another language. The model outperforms the much larger GPT-3 in language translation and summarization. Training mixes denoising (appropriately inserting missing text in strings) and causal-language-modeling (meaningfully extending an input text). It allows adding features across different languages without massive training workflows. AlexaTM 20B achieved state-of-the-art performance in few-shot-learning tasks across all Flores-101 language pairs, outperforming GPT-3 on several tasks.


Language model

Language model A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. In 1980, the first significant statistical language model was proposed, and during the decade IBM performed ���Shannon-style��� experiments, in which potential sources for language modeling improvement were identified by observing and analyzing the performance of human subjects in predicting or correcting text. Language models are useful for a variety of tasks, including speech recognition (helping prevent predictions of low-probability (e.g. nonsense) sequences), machine translation, natural language generation (generating more human-like text), optical character recognition, handwriting recognition, grammar induction, information retrieval, and other. Large language models, currently their most advanced form, are a combination of larger datasets (frequently using scraped words from the public internet), feedforward neural networks, and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.


Language model

Pure_statistical_models A word n-gram language model is a purely statistical model of language. It has been superseded by recurrent neural network-based models, which has been superseded by large language models. It is based on an assumption that the probability of the next word in a sequence depends only on a fixed size window of previous words. If only one previous word was considered, it was called a bigram model; if two words, a trigram model; if n-1 words, an n-gram model. Special tokens were introduced to denote the start and end of a sentence ��� s ��� {\displaystyle \langle s\rangle } and ��� / s ��� {\displaystyle \langle /s\rangle } . Maximum entropy language models encode the relationship between a word and the n-gram history using feature functions. The equation is where Z ( w 1 , ��� , w m ��� 1 ) {\displaystyle Z(w_{1},\ldots ,w_{m-1})} is the partition function, a {\displaystyle a} is the parameter vector, and f ( w 1 , ��� , w m ) {\displaystyle f(w_{1},\ldots ,w_{m})} is the feature function. In the simplest case, the feature function is just an indicator of the presence of a certain n-gram. It is helpful to use a prior on a {\displaystyle a} or some form of regularization. The log-bilinear model is another example of an exponential language model. Skip-gram language model is an attempt at overcoming the data sparsity problem that preceding (i.e. word n-gram language model) faced. Words represented in an embedding vector were not necessarily consecutive anymore, but could leave gaps that are skipped over. Formally, a k-skip-n-gram is a length-n subsequence where the components occur at distance at most k from each other. For example, in the input text: the set of 1-skip-2-grams includes all the bigrams (2-grams), and in addition the subsequences In skip-gram model, semantic relations between words are represented by linear combinations, capturing a form of compositionality. For example, in some such models, if v is the function that maps a word w to its n-d vector representation, then


Language model

Neural_models Continuous representations or embeddings of words are produced in recurrent neural network-based language models (known also as continuous space language models). Such continuous space embeddings help to alleviate the curse of dimensionality, which is the consequence of the number of possible sequences of words increasing exponentially with the size of the vocabulary, furtherly causing a data sparsity problem. Neural networks avoid this problem by representing words as non-linear combinations of weights in a neural net. A large language model (LLM) is a type of language model notable for its ability to achieve general-purpose language understanding and generation. LLMs acquire these abilities by using massive amounts of data to learn billions of parameters during training and consuming large computational resources during their training and operation. LLMs are artificial neural networks (mainly Transformers ) and are (pre-)trained using self-supervised learning and semi-supervised learning. As autoregressive language models, they work by taking an input text and repeatedly predicting the next token or word. Up to 2020, fine tuning was the only way a model could be adapted to be able to accomplish specific tasks. Larger sized models, such as GPT-3, however, can be prompt-engineered to achieve similar results. They are thought to acquire embodied knowledge about syntax, semantics and "ontology" inherent in human language corpora, but also inaccuracies and biases present in the corpora. Although sometimes matching human performance, it is not clear they are plausible cognitive models. At least for recurrent neural networks it has been shown that they sometimes learn patterns which humans do not learn, but fail to learn patterns that humans typically do learn.


Language model

Evaluation_and_benchmarks Evaluation of the quality of language models is mostly done by comparison to human created sample benchmarks created from typical language-oriented tasks. Other, less established, quality tests examine the intrinsic character of a language model or compare two such models. Since language models are typically intended to be dynamic and to learn from data it sees, some proposed models investigate the rate of learning, e.g. through inspection of learning curves. Various data sets have been developed to use to evaluate language processing systems. These include: Corpus of Linguistic Acceptability GLUE benchmark Microsoft Research Paraphrase Corpus Multi-Genre Natural Language Inference Question Natural Language Inference Quora Question Pairs Recognizing Textual Entailment Semantic Textual Similarity Benchmark SQuAD question answering Test Stanford Sentiment Treebank Winograd NLI BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, NaturalQuestions, TriviaQA, RACE, MMLU (Massive Multitask Language Understanding), BIG-bench hard, GSM8k, RealToxicityPrompts, WinoGender, CrowS-Pairs. (LLaMa Benchmark)


Language model

See_also Cache language model Deep linguistic processing Factored language model Generative pre-trained transformer Katz's back-off model Language technology Statistical model Ethics of artificial intelligence Semantic similarity network


Perplexity

Perplexity_per_word In natural language processing, a corpus is a set of sentences or texts, and a language model is a probability distribution over entire sentences or texts. Consequently, in NLP, the more commonly used measure is perplexity per word, defined as: Suppose the average sentence xi in the corpus has a probability of 2 ��� 190 {\displaystyle 2^{-190}} according to the language model. This would give a model perplexity of 2190 per sentence. However, it is more common to normalize for sentence length. Thus, if the test sample's sentences comprised a total of 1,000 words, and could be coded using 7.95 bits per word, one could report a model perplexity of 27.95 = 247 per word. In other words, the model is as confused on test data as if it had to choose uniformly and independently among 247 possibilities for each word. Since 2007, significant advancements in language modeling have emerged, particularly with the advent of deep learning techniques. Perplexity per word, a measure that quantifies the predictive power of a language model, has remained central to evaluating models like transformers, BERT, GPT-2, and others. This measure is employed to compare different models on the same dataset and guide the optimization of hyperparameters, although it has been found sensitive to factors such as linguistic features and sentence length. Despite its pivotal role in language model development, perplexity has shown limitations, particularly as an inadequate predictor of speech recognition performance, where it may not correlate well with word-error rates, raising questions about its accuracy. The lowest perplexity that has been published on the Brown Corpus (1 million words of American English of varying topics and genres) as of 1992 is indeed about 247 per word, corresponding to a cross-entropy of log2247 = 7.95 bits per word or 1.75 bits per letter using a trigram model. While this figure represented the state of the art at the time, advancements in techniques such as deep learning have led to significant improvements in perplexity on other benchmarks, such as the One Billion Word Benchmark. In the context of the Brown corpus, simply guessing that the next word is "the" will achieve an accuracy of 7 percent, contrasting with the 1/247 = 0.4 percent that might be expected from a naive use of perplexity. This difference underscores the importance of the statistical model used and the nuanced nature of perplexity as a measure of predictiveness. The guess is based on unigram statistics, not on the trigram statistics that yielded the perplexity of 247, and utilizing trigram statistics would further refine the prediction.


Reinforcement learning from human feedback In machine learning, reinforcement learning from human feedback (RLHF) or reinforcement learning from human preferences is a technique that trains a "reward model" directly from human feedback and uses the model as a reward function to optimize an agent's policy using reinforcement learning (RL) through an optimization algorithm like Proximal Policy Optimization. The reward model is trained in advance to the policy being optimized to predict if a given output is good (high reward) or bad (low reward). RLHF can improve the robustness and exploration of reinforcement-learning agents, especially when the reward function is sparse or noisy. Human feedback is most commonly collected by asking humans to rank instances of the agent's behavior. These rankings can then be used to score outputs, for example with the Elo rating system. While the preference judgement is widely adopted, there are other types of human feedbacks that provide richer information, such as numerical feedback, natural language feedback, and edit rate. RLHF is used in tasks where it's difficult to define a clear, algorithmic solution but where humans can easily judge the quality of the model's output. For example, if the task is to generate a compelling story, humans can rate different AI-generated stories on their quality, and the model can use their feedback to improve its story generation skills. RLHF has been applied to various domains of natural language processing, such as conversational agents, text summarization, and natural language understanding. Ordinary reinforcement learning, where agents learn from their own actions based on a "reward function", is difficult to apply to natural language processing tasks because the rewards are often not easy to define or measure, especially when dealing with complex tasks that involve human values or preferences. RLHF can enable language models to provide answers that align with these complex values, to generate more verbose responses, and to reject questions that are either inappropriate or outside the knowledge space of the model. Some examples of RLHF-trained language models are OpenAI's ChatGPT and its predecessor InstructGPT, as well as DeepMind's Sparrow. RLHF has also been applied to other areas, such as the development of video game bots. For example, OpenAI and DeepMind trained agents to play Atari games based on human preferences. The agents achieved strong performance in many of the environments tested, often surpassing human performance.


See_also Reinforcement learning ChatGPT Reward-based selection


Word embedding

Development_and_history_of_the_approach In distributional semantics, a quantitative methodological approach to understanding meaning in observed language, word embeddings or semantic feature space models have been used as a knowledge representation for some time. Such models aim to quantify and categorize semantic similarities between linguistic items based on their distributional properties in large samples of language data. The underlying idea that "a word is characterized by the company it keeps" was proposed in a 1957 article by John Rupert Firth, but also has roots in the contemporaneous work on search systems and in cognitive psychology. The notion of a semantic space with lexical items (words or multi-word terms) represented as vectors or embeddings is based on the computational challenges of capturing distributional characteristics and using them for practical application to measure similarity between words, phrases, or entire documents. The first generation of semantic space models is the vector space model for information retrieval. Such vector space models for words and their distributional data implemented in their simplest form results in a very sparse vector space of high dimensionality (cf. curse of dimensionality). Reducing the number of dimensions using linear algebraic methods such as singular value decomposition then led to the introduction of latent semantic analysis in the late 1980s and the random indexing approach for collecting word cooccurrence contexts. In 2000, Bengio et al. provided in a series of papers titled "Neural probabilistic language models" to reduce the high dimensionality of word representations in contexts by "learning a distributed representation for words". A study published in NeurIPS (NIPS) 2002 introduced the use of both word and document embeddings applying the method of kernel CCA to bilingual (and multi-lingual) corpora, also providing an early example of self-supervised learning of word embeddings Word embeddings come in two different styles, one in which words are expressed as vectors of co-occurring words, and another in which words are expressed as vectors of linguistic contexts in which the words occur; these different styles are studied in Lavelli et al., 2004. Roweis and Saul published in Science how to use "locally linear embedding" (LLE) to discover representations of high dimensional data structures. Most new word embedding techniques after about 2005 rely on a neural network architecture instead of more probabilistic and algebraic models, after foundational work done by Yoshua Bengio and colleagues. The approach has been adopted by many research groups after theoretical advances in 2010 had been made on the quality of vectors and the training speed of the model, as well as after hardware advances allowed for a broader parameter space to be explored profitably. In 2013, a team at Google led by Tomas Mikolov created word2vec, a word embedding toolkit that can train vector space models faster than previous approaches. The word2vec approach has been widely used in experimentation and was instrumental in raising interest for word embeddings as a technology, moving the research strand out of specialised research into broader experimentation and eventually paving the way for practical application.


Word embedding

Sentence_embeddings The idea has been extended to embeddings of entire sentences or even documents, e.g. in the form of the thought vectors concept. In 2015, some researchers suggested "skip-thought vectors" as a means to improve the quality of machine translation. A more recent and popular approach for representing sentences is Sentence-BERT, or SentenceTransformers, which modifies pre-trained BERT with the use of siamese and triplet network structures.


Word n-gram language model

Word n-gram language model A word n-gram language model is a purely statistical model of language. It has been superseded by recurrent neural network-based models, which has been superseded by large language models. It is based on an assumption that the probability of the next word in a sequence depends only on a fixed size window of previous words. If only one previous word was considered, it was called a bigram model; if two words, a trigram model; if n-1 words, an n-gram model. Special tokens were introduced to denote the start and end of a sentence ��� s ��� {\displaystyle \langle s\rangle } and ��� / s ��� {\displaystyle \langle /s\rangle } . To prevent a zero probability being assigned to unseen words, each word's probability is slightly lower than its frequency count in a corpus. To calculate it, various methods were used, from simple "add-one" smoothing (assign a count of 1 to unseen n-grams, as an uninformative prior) to more sophisticated models, such as Good���Turing discounting or back-off models.


Word n-gram language model

Approximation_method The approximation method calculates the probability P ( w 1 , ��� , w m ) {\displaystyle P(w_{1},\ldots ,w_{m})} of observing the sentence w 1 , ��� , w m {\displaystyle w_{1},\ldots ,w_{m}} It is assumed that the probability of observing the ith word wi (in the context window consisting of the preceding i ��� 1 words) can be approximated by the probability of observing it in the shortened context window consisting of the preceding n ��� 1 words (nth-order Markov property). To clarify, for n = 3 and i = 2 we have P ( w i ��� w i ��� ( n ��� 1 ) , ��� , w i ��� 1 ) = P ( w 2 ��� w 1 ) {\displaystyle P(w_{i}\mid w_{i-(n-1)},\ldots ,w_{i-1})=P(w_{2}\mid w_{1})} . The conditional probability can be calculated from n-gram model frequency counts: An issue when using n-gram language models are out-of-vocabulary (OOV) words. They are encountered in computational linguistics and natural language processing when the input includes words which were not present in a system's dictionary or database during its preparation. By default, when a language model is estimated, the entire observed vocabulary is used. In some cases, it may be necessary to estimate the language model with a specific fixed vocabulary. In such a scenario, the n-grams in the corpus that contain an out-of-vocabulary word are ignored. The n-gram probabilities are smoothed over all the words in the vocabulary even if they were not observed. Nonetheless, it is essential in some cases to explicitly model the probability of out-of-vocabulary words by introducing a special token (e.g. <unk>) into the vocabulary. Out-of-vocabulary words in the corpus are effectively replaced with this special <unk> token before n-grams counts are cumulated. With this option, it is possible to estimate the transition probabilities of n-grams involving out-of-vocabulary words.


Natural language generation

Applications From a commercial perspective, the most successful NLG applications have been data-to-text systems which generate textual summaries of databases and data sets; these systems usually perform data analysis as well as text generation.  Research has shown that textual summaries can be more effective than graphs and other visuals for decision support, and that computer-generated texts can be superior (from the reader's perspective) to human-written texts.  The first commercial data-to-text systems produced weather forecasts from weather data.  The earliest such system to be deployed was FoG, which was used by Environment Canada to generate weather forecasts in French and English in the early 1990s.  The success of FoG triggered other work, both research and commercial.  Recent applications include the UK Met Office's text-enhanced forecast.  Data-to-text systems have since been applied in a range of settings.  Following the minor earthquake near Beverly Hills, California on March 17, 2014, The Los Angeles Times reported details about the time, location and strength of the quake within 3 minutes of the event.  This report was automatically generated by a 'robo-journalist', which converted the incoming data into text via a preset template.  Currently there is considerable commercial interest in using NLG to summarise financial and business data.  Indeed, Gartner has said that NLG will become a standard feature of 90% of modern BI and analytics platforms.  NLG is also being used commercially in automated journalism, chatbots, generating product descriptions for e-commerce sites, summarising medical records, and enhancing accessibility (for example by describing graphs and data sets to blind people).  An example of an interactive use of NLG is the WYSIWYM framework.  It stands for What you see is what you meant and allows users to see and manipulate the continuously rendered view (NLG output) of an underlying formal language document (NLG input), thereby editing the formal language without learning it.  Looking ahead, the current progress in data-to-text generation paves the way for tailoring texts to specific audiences.  For example, data from babies in neonatal care can be converted into text differently in a clinical setting, with different levels of technical detail and explanatory language, depending on intended recipient of the text (doctor, nurse, patient).  The same idea can be applied in a sports setting, with different reports generated for fans of specific teams.  Over the past few years, there has been an increased interest in automatically generating captions for images, as part of a broader endeavor to investigate the interface between vision and language.  A case of data-to-text generation, the algorithm of image captioning (or automatic image description) involves taking an image, analyzing its visual content, and generating a textual description (typically a sentence) that verbalizes the most prominent aspects of the image.  An image captioning system involves two sub-tasks.  In Image Analysis, features and attributes of an image are detected and labelled, before mapping these outputs to linguistic structures.


Natural language generation

Applications Recent research utilizes deep learning approaches through features from a pre-trained convolutional neural network such as AlexNet, VGG or Caffe, where caption generators use an activation layer from the pre-trained network as their input features.  Text Generation, the second task, is performed using a wide range of techniques.  For example, in the Midge system, input images are represented as triples consisting of object/stuff detections, action/pose detections and spatial relations.  These are subsequently mapped to <noun, verb, preposition> triples and realized using a tree substitution grammar.  Despite advancements, challenges and opportunities remain in image capturing research.  Notwithstanding the recent introduction of Flickr30K, MS COCO and other large datasets have enabled the training of more complex models such as neural networks, it has been argued that research in image captioning could benefit from larger and diversified datasets.  Designing automatic measures that can mimic human judgments in evaluating the suitability of image descriptions is another need in the area.  Other open challenges include visual question-answering (VQA), as well as the construction and evaluation multilingual repositories for image description.  Another area where NLG has been widely applied is automated dialogue systems, frequently in the form of chatbots.  A chatbot or chatterbot is a software application used to conduct an on-line chat conversation via text or text-to-speech, in lieu of providing direct contact with a live human agent.  While natural language processing (NLP) techniques are applied in deciphering human input, NLG informs the output part of the chatbot algorithms in facilitating real-time dialogues.  Early chatbot systems, including Cleverbot created by Rollo Carpenter in 1988 and published in 1997,[citation needed] reply to questions by identifying how a human has responded to the same question in a conversation database using information retrieval (IR) techniques. [citation needed] Modern chatbot systems predominantly rely on machine learning (ML) models, such as sequence-to-sequence learning and reinforcement learning to generate natural language output.  Hybrid models have also been explored.  For example, the Alibaba shopping assistant first uses an IR approach to retrieve the best candidates from the knowledge base, then uses the ML-driven seq2seq model re-rank the candidate responses and generate the answer.  Creative language generation by NLG has been hypothesized since the field's origins.  A recent pioneer in the area is Phillip Parker, who has developed an arsenal of algorithms capable of automatically generating textbooks, crossword puzzles, poems and books on topics ranging from bookbinding to cataracts.  The advent of large pretrained transformer-based language models such as GPT-3 has also enabled breakthroughs, with such models demonstrating recognizable ability for creating-writing tasks.  A related area of NLG application is computational humor production.  JAPE (Joke Analysis and Production Engine) is one of the earliest large, automated humor production systems that uses a hand-coded template-based approach to create punning riddles for children.  HAHAcronym creates humorous reinterpretations of any given acronym, as well as proposing new fitting acronyms given some keywords.  Despite progresses, many challenges remain in producing automated creative and humorous content that rival human output.


Natural language generation

Applications In an experiment for generating satirical headlines, outputs of their best BERT-based model were perceived as funny 9.4% of the time (while real Onion headlines were 38.4%) and a GPT-2 model fine-tuned on satirical headlines achieved 6.9%.  It has been pointed out that two main issues with humor-generation systems are the lack of annotated data sets and the lack of formal evaluation methods, which could be applicable to other creative content generation.  Some have argued relative to other applications, there has been a lack of attention to creative aspects of language production within NLG.  NLG researchers stand to benefit from insights into what constitutes creative language production, as well as structural features of narrative that have the potential to improve NLG output even in data-to-text systems.


Text-to-image model

Text-to-image model A text-to-image model is a machine learning model which takes an input natural language description and produces an image matching that description. Such models began to be developed in the mid-2010s, as a result of advances in deep neural networks. In 2022, the output of state of the art text-to-image models, such as OpenAI's DALL-E 2, Google Brain's Imagen, StabilityAI's Stable Diffusion, and Midjourney began to approach the quality of real photographs and human-drawn art. Text-to-image models generally combine a language model, which transforms the input text into a latent representation, and a generative image model, which produces an image conditioned on that representation. The most effective models have generally been trained on massive amounts of image and text data scraped from the web.


Text-to-image model

History Before the rise of deep learning, attempts to build text-to-image models were limited to collages by arranging existing component images, such as from a database of clip art. The inverse task, image captioning, was more tractable and a number of image captioning deep learning models came prior to the first text-to-image models. The first modern text-to-image model, alignDRAW, was introduced in 2015 by researchers from the University of Toronto. alignDRAW extended the previously-introduced DRAW architecture (which used a recurrent variational autoencoder with an attention mechanism) to be conditioned on text sequences. Images generated by alignDRAW were blurry and not photorealistic, but the model was able to generalize to objects not represented in the training data (such as a red school bus), and appropriately handled novel prompts such as "a stop sign is flying in blue skies", showing that it was not merely "memorizing" data from the training set. In 2016, Reed, Akata, Yan et al. became the first to use generative adversarial networks for the text-to-image task. With models trained on narrow, domain-specific datasets, they were able to generate "visually plausible" images of birds and flowers from text captions like "an all black bird with a distinct thick, rounded bill". A model trained on the more diverse COCO dataset produced images which were "from a distance... encouraging", but which lacked coherence in their details. Later systems include VQGAN+CLIP, XMC-GAN, and GauGAN2. One of the first text-to-image models to capture widespread public attention was OpenAI's DALL-E, a transformer system announced in January 2021. A successor capable of generating more complex and realistic images, DALL-E 2, was unveiled in April 2022, followed by Stable Diffusion publicly released in August 2022. Following other text-to-image models, language model-powered text-to-video platforms such as Runway, Make-A-Video, Imagen Video, Midjourney, and Phenaki can generate video from text and/or text/image prompts.


Text-to-image model

Architecture_and_training Text-to-image models have been built using a variety of architectures. The text encoding step may be performed with a recurrent neural network such as a long short-term memory (LSTM) network, though transformer models have since become a more popular option. For the image generation step, conditional generative adversarial networks have been commonly used, with diffusion models also becoming a popular option in recent years. Rather than directly training a model to output a high-resolution image conditioned on a text embedding, a popular technique is to train a model to generate low-resolution images, and use one or more auxiliary deep learning models to upscale it, filling in finer details. Text-to-image models are trained on large datasets of (text, image) pairs, often scraped from the web. With their 2022 Imagen model, Google Brain reported positive results from using a large language model trained separately on a text-only corpus (with its weights subsequently frozen), a departure from the theretofore standard approach.


Named-entity recognition

Named-entity_recognition_platforms Notable NER platforms include: GATE supports NER across many languages and domains out of the box, usable via a graphical interface and a Java API. OpenNLP includes rule-based and statistical named-entity recognition. SpaCy features fast statistical NER as well as an open-source named-entity visualizer. Transformers features token classification using deep learning models.


Semantic query

Semantic query Semantic queries allow for queries and analytics of associative and contextual nature. Semantic queries enable the retrieval of both explicitly and implicitly derived information based on syntactic, semantic and structural information contained in data. They are designed to deliver precise results (possibly the distinctive selection of one single piece of information) or to answer more fuzzy and wide open questions through pattern matching and digital reasoning. Semantic queries work on named graphs, linked data or triples. This enables the query to process the actual relationships between information and infer the answers from the network of data. This is in contrast to semantic search, which uses semantics (meaning of language constructs) in unstructured text to produce a better search result. (See natural language processing.) From a technical point of view, semantic queries are precise relational-type operations much like a database query. They work on structured data and therefore have the possibility to utilize comprehensive features like operators (e.g. >, < and =), namespaces, pattern matching, subclassing, transitive relations, semantic rules and contextual full text search. The semantic web technology stack of the W3C is offering SPARQL to formulate semantic queries in a syntax similar to SQL. Semantic queries are used in triplestores, graph databases, semantic wikis, natural language and artificial intelligence systems.


Semantic search

Semantic search Semantic search denotes search with meaning, as distinguished from lexical search where the search engine looks for literal matches of the query words or variants of them, without understanding the overall meaning of the query. Semantic search seeks to improve search accuracy by understanding the searcher's intent and the contextual meaning of terms as they appear in the searchable dataspace, whether on the Web or within a closed system, to generate more relevant results. Content that ranks well in semantic search is well-written in a natural voice, focuses on the user's intent, and considers related topics that the user may look for in the future. Some authors regard semantic search as a set of techniques for retrieving knowledge from richly structured data sources like ontologies and XML as found on the Semantic Web. Such technologies enable the formal articulation of domain knowledge at a high level of expressiveness and could enable the user to specify their intent in more detail at query time.


Semantic search

See_also List of Semantic Search Engines Semantic web Semantic Unification Resource Description Framework Natural language search engine Semantic Query


AI safety

Research_foci AI safety research areas include robustness, monitoring, and alignment.  AI systems are often vulnerable to adversarial examples or ���inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake���.  For example, in 2013, Szegedy et al.  discovered that adding specific imperceptible perturbations to an image could cause it to be misclassified with high confidence.  This continues to be an issue with neural networks, though in recent work the perturbations are generally large enough to be perceptible.  All of the images on the right are predicted to be an ostrich after the perturbation is applied.  (Left) is a correctly predicted sample, (center) perturbation applied magnified by 10x, (right) adversarial example.  Adversarial robustness is often associated with security.  Researchers demonstrated that an audio signal could be imperceptibly modified so that speech-to-text systems transcribe it to any message the attacker chooses.  Network intrusion and malware detection systems also must be adversarially robust since attackers may design their attacks to fool detectors.  Models that represent objectives (reward models) must also be adversarially robust.  For example, a reward model might estimate how helpful a text response is and a language model might be trained to maximize this score.  Researchers have shown that if a language model is trained for long enough, it will leverage the vulnerabilities of the reward model to achieve a better score and perform worse on the intended task.  This issue can be addressed by improving the adversarial robustness of the reward model.  More generally, any AI system used to evaluate another AI system must be adversarially robust.  This could include monitoring tools, since they could also potentially be tampered with to produce a higher reward.  It is often important for human operators to gauge how much they should trust an AI system, especially in high-stakes settings such as medical diagnosis.  ML models generally express confidence by outputting probabilities; however, they are often overconfident, especially in situations that differ from those that they were trained to handle.  Calibration research aims to make model probabilities correspond as closely as possible to the true proportion that the model is correct.  Similarly, anomaly detection or out-of-distribution (OOD) detection aims to identify when an AI system is in an unusual situation.  For example, if a sensor on an autonomous vehicle is malfunctioning, or it encounters challenging terrain, it should alert the driver to take control or pull over.  Anomaly detection has been implemented by simply training a classifier to distinguish anomalous and non-anomalous inputs, though several other techniques are in use.  Scholars and government agencies have expressed concerns that AI systems could be used to help malicious actors to build weapons, manipulate public opinion, or automate cyber attacks.  These worries are a practical concern for companies like OpenAI which host powerful AI tools online.  In order to prevent misuse, OpenAI has built detection systems that flag or restrict users based on their activity.


AI safety

Research_foci Neural networks have often been described as black boxes, meaning that it is difficult to understand why they make the decisions they do as a result of the massive number of computations they perform.  This makes it challenging to anticipate failures.  In 2018, a self-driving car killed a pedestrian after failing to identify them.  Due to the black box nature of the AI software, the reason for the failure remains unclear.  One benefit of transparency is explainability.  It is sometimes a legal requirement to provide an explanation for why a decision was made in order to ensure fairness, for example for automatically filtering job applications or credit score assignment.  Another benefit is to reveal the cause of failures.  At the beginning of the 2020 COVID-19 pandemic, researchers used transparency tools to show that medical image classifiers were ���paying attention��� to irrelevant hospital labels.  Transparency techniques can also be used to correct errors.  For example, in the paper ���Locating and Editing Factual Associations in GPT,��� the authors were able to identify model parameters that influenced how it answered questions about the location of the Eiffel tower.  They were then able to ���edit��� this knowledge to make the model respond to questions as if it believed the tower was in Rome instead of France.  Though in this case, the authors induced an error, these methods could potentially be used to efficiently fix them.  Model editing techniques also exist in computer vision.  Finally, some have argued that the opaqueness of AI systems is a significant source of risk and better understanding of how they function could prevent high-consequence failures in the future.  ���Inner��� interpretability research aims to make ML models less opaque.  One goal of this research is to identify what the internal neuron activations represent.  For example, researchers identified a neuron in CLIP that responds to images of people in spider man costumes, sketches of spiderman, and the word ���spider. ��� It also involves explaining connections between these neurons or ���circuits���.  For example, researchers have identified pattern-matching mechanisms in transformer attention that may play a role in how language models learn from their context.  ���Inner interpretability��� has been compared to neuroscience.  In both cases, the goal is to understand what is going on in an intricate system, though ML researchers have the benefit of being able to take perfect measurements and perform arbitrary ablations.  ML models can potentially contain ���trojans��� or ���backdoors���: vulnerabilities that malicious actors maliciously build into an AI system.  For example, a trojaned facial recognition system could grant access when a specific piece of jewelry is in view; or a trojaned autonomous vehicle may function normally until a specific trigger is visible.  Note that an adversary must have access to the system���s training data in order to plant a trojan.  This might not be difficult to do with some large models like CLIP or GPT-3 as they are trained on publicly available internet data.


AI safety

Research_foci Researchers were able to plant a trojan in an image classifier by changing just 300 out of 3 million of the training images.  In addition to posing a security risk, researchers have argued that trojans provide a concrete setting for testing and developing better monitoring tools.  In the field of artificial intelligence (AI), AI alignment research aims to steer AI systems towards humans' intended goals, preferences, or ethical principles.  An AI system is considered aligned if it advances the intended objectives.  A misaligned AI system pursues some objectives, but not the intended ones.  It can be challenging for AI designers to align an AI system because it can be difficult for them to specify the full range of desired and undesired behavior.  To avoid this difficulty, they typically use simpler proxy goals, such as gaining human approval.  But that approach can create loopholes, overlook necessary constraints, or reward the AI system for merely appearing aligned.  Misaligned AI systems can malfunction or cause harm.  AI systems may find loopholes that allow them to accomplish their proxy goals efficiently but in unintended, sometimes harmful ways (reward hacking).  They may also develop unwanted instrumental strategies, such as seeking power or survival, because such strategies help them achieve their given goals.  Furthermore, they may develop undesirable emergent goals that may be hard to detect before the system is deployed, when it faces new situations and data distributions.  Today, these problems affect existing commercial systems such as language models, robots, autonomous vehicles, and social media recommendation engines.  Some AI researchers argue that more capable future systems will be more severely affected since these problems partially result from the systems being highly capable.  Many leading AI scientists, such as Geoffrey Hinton and Stuart Russell, argue that AI is approaching superhuman capabilities and could endanger human civilization if misaligned.  It is common for AI risks (and technological risks more generally) to be categorized as misuse or accidents.  Some scholars have suggested that this framework falls short.  For example, the Cuban Missile Crisis was not clearly an accident or a misuse of technology.  Policy analysts Zwetsloot and Dafoe wrote, ���The misuse and accident perspectives tend to focus only on the last step in a causal chain leading up to a harm: that is, the person who misused the technology, or the system that behaved in unintended ways��� Often, though, the relevant causal chain is much longer. ��� Risks often arise from ���structural��� or ���systemic��� factors such as competitive pressures, diffusion of harms, fast-paced development, high levels of uncertainty, and inadequate safety culture.  In the broader context of safety engineering, structural factors like ���organizational safety culture��� play a central role in the popular STAMP risk analysis framework.  Inspired by the structural perspective, some researchers have emphasized the importance of using machine learning to improve sociotechnical safety factors, for example, using ML for cyber defense, improving institutional decision-making, and facilitating cooperation.  Some scholars are concerned that AI will exacerbate the already imbalanced game between cyber attackers and cyber defenders.


AI safety

Research_foci This would increase 'first strike' incentives and could lead to more aggressive and destabilizing attacks.  In order to mitigate this risk, some have advocated for an increased emphasis on cyber defense.  In addition, software security is essential preventing powerful AI models from being stolen and misused.  The advancement of AI in economic and military domains could precipitate unprecedented political challenges.  Some scholars have compared AI race dynamics to the cold war, where the careful judgment of a small number of decision-makers often spelled the difference between stability and catastrophe.  AI researchers have argued that AI technologies could also be used to assist decision-making.  For example, researchers are beginning to develop AI forecasting and advisory systems.  Many of the largest global threats (nuclear war, climate change, etc) have been framed as cooperation challenges.  As in the well-known prisoner���s dilemma scenario, some dynamics may lead to poor results for all players, even when they are optimally acting in their self-interest.  For example, no single actor has strong incentives to address climate change even though the consequences may be significant if no one intervenes.  A salient AI cooperation challenge is avoiding a ���race to the bottom���.  In this scenario, countries or companies race to build more capable AI systems and neglect safety, leading to a catastrophic accident that harms everyone involved.  Concerns about scenarios like these have inspired both political and technical efforts to facilitate cooperation between humans, and potentially also between AI systems.  Most AI research focuses on designing individual agents to serve isolated functions (often in ���single-player��� games).  Scholars have suggested that as AI systems become more autonomous, it may become essential to study and shape the way they interact.


AI safety

In_governance AI governance is broadly concerned with creating norms, standards, and regulations to guide the use and development of AI systems.  AI safety governance research ranges from foundational investigations into the potential impacts of AI to specific applications.  On the foundational side, researchers have argued that AI could transform many aspects of society due to its broad applicability, comparing it to electricity and the steam engine.  Some work has focused on anticipating specific risks that may arise from these impacts ��� for example, risks from mass unemployment, weaponization, disinformation, surveillance, and the concentration of power.  Other work explores underlying risk factors such as the difficulty of monitoring the rapidly evolving AI industry, the availability of AI models, and ���race to the bottom��� dynamics.  Allan Dafoe, the head of longterm governance and strategy at DeepMind has emphasized the dangers of racing and the potential need for cooperation: ���it may be close to a necessary and sufficient condition for AI safety and alignment that there be a high degree of caution prior to deploying advanced powerful systems; however, if actors are competing in a domain with large returns to first-movers or relative advantage, then they will be pressured to choose a sub-optimal level of caution. ��� Some experts have argued that it is too early to regulate AI, expressing concerns that regulations will hamper innovation and it would be foolish to ���rush to regulate in ignorance. ��� Others, such as business magnate Elon Musk, call for pre-emptive action to mitigate catastrophic risks.  Outside of formal legislation, government agencies have put forward ethical and safety recommendations.  In March 2021, the US National Security Commission on Artificial Intelligence reported that advances in AI may make it increasingly important to ���assure that systems are aligned with goals and values, including safety, robustness and trustworthiness. " Subsequently, the National Institute of Standards and Technology drafted a framework for managing AI Risk, which advises that when "catastrophic risks are present - development and deployment should cease in a safe manner until risks can be sufficiently managed. " In September 2021, the People's Republic of China published ethical guidelines for the use of AI in China, emphasizing that AI decisions should remain under human control and calling for accountability mechanisms.  In the same month, The United Kingdom published its 10-year National AI Strategy, which states the British government "takes the long-term risk of non-aligned Artificial General Intelligence, and the unforeseeable changes that it would mean for . . .  the world, seriously. " The strategy describes actions to assess long-term AI risks, including catastrophic risks.  The British government has announced plans for the first major global summit on AI safety, this is due to take place on the 1st and 2 November 2023 and is an opportunity for policymakers and world leaders to consider the immediate and future risks of AI and how these risks can be mitigated via a globally coordinated approach [1] [2].


AI safety

In_governance Government organizations, particularly in the United States, have also encouraged the development of technical AI safety research.  The Intelligence Advanced Research Projects Activity initiated the TrojAI project to identify and protect against Trojan attacks on AI systems.  The Defense Advanced Research Projects Agency engages in research on explainable artificial intelligence and improving robustness against adversarial attacks and The National Science Foundation supports the Center for Trustworthy Machine Learning, and is providing millions in funding for empirical AI safety research.  AI labs and companies generally abide by safety practices and norms that fall outside of formal legislation.  One aim of governance researchers is to shape these norms. [? ] Examples of safety recommendations found in the literature include performing third-party auditing, offering bounties for finding failures, sharing AI incidents (an AI incident database was created for this purpose), following guidelines to determine whether to publish research or models, and improving information and cyber security in AI labs.  Companies have also made concrete commitments.  Cohere, OpenAI, and AI21 proposed and agreed on ���best practices for deploying language models,��� focusing on mitigating misuse.  To avoid contributing to racing-dynamics, OpenAI has also stated in their charter that ���if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project��� Also, industry leaders such as CEO of DeepMind Demis Hassabis, director of Facebook AI Yann LeCun have signed open letters such as the Asilomar Principles and the Autonomous Weapons Open Letter.


Anthropic

Anthropic Anthropic PBC is an American artificial intelligence (AI) startup and public-benefit corporation, founded by former members of OpenAI. Anthropic specializes in developing general AI systems and language models, with a company ethos of responsible AI usage. As of July 2023[update], Anthropic had raised US$1.5 billion in funding. In September, Amazon announced an investment of up to US$4 billion.


Anthropic

History Anthropic was founded in 2021 by former senior members of OpenAI, principally siblings Daniela Amodei and Dario Amodei, the latter of whom served as OpenAI's Vice President of Research. The Amodei siblings were among others who left OpenAI due to directional differences, specifically regarding OpenAI's ventures with Microsoft in 2019. By late 2022, Anthropic had raised US$700 million in funding, out of which US$500 million came from Alameda Research. Google's cloud division followed with an investment of US$300 million for a 10% stake, in a deal requiring Anthropic to buy computing resources from Google Cloud. In May 2023, Anthropic raised US$450 million in a round led by Spark Capital. In February 2023, Anthropic was sued by Texas-based Anthrop LLC for use of their registered trademark "Anthropic A.I." Kevin Roose of The New York Times described the company as the "Center of A.I. Doomerism". He reported that some employees "compared themselves to modern-day Robert Oppenheimers". Journalists often connect Anthropic with the effective altruism movement; some founders and team members were part of the community or at least interested in it. One of the investors of Series B round was Sam Bankman-Fried of the cryptocurrency exchange FTX that collapsed in 2022. On September 25, 2023, Amazon and Anthropic announced a strategic partnership in which Amazon would become a minority stakeholder by investing up to US$4 billion, including an immediate investment of $1.25bn. As part of the deal, Anthropic would use Amazon Web Services (AWS) as its primary cloud provider and will make its AI models available to AWS customers.


Anthropic

Projects Consisting of former researchers involved in OpenAI's GPT-2 and GPT-3 model development, Anthropic began development on its own AI chatbot, named Claude. Similar to ChatGPT, Claude uses a messaging interface where users can submit questions or requests and receive highly detailed and relevant responses. Initially available in closed beta through a Slack integration, Claude is now accessible to users in the UK and the US via a website claude.ai. The name, "Claude", was chosen either as a reference to Claude Shannon, or as "a friendly, male-gendered name designed to counterbalance the female-gendered names (Alexa, Siri, Cortana) that other tech companies gave their A.I. assistants". Claude 2 was launched in July 2023, and is available in US and UK. The Guardian reported that safety was a priority during the model training. Anthropic calls their safety method "Constitutional AI": The chatbot is trained on principles taken from documents including the 1948 UN declaration and Apple���s terms of service, which cover modern issues such as data privacy and impersonation. One example of a Claude 2 principle based on the UN declaration is: ���Please choose the response that most supports and encourages freedom, equality and a sense of brotherhood.��� Anthropic also publishes research on the interpretability of machine learning systems, focusing on the transformer architecture.


Google AI

Projects Google Brain, a big subsidiary developing AI with machine learning to improve various Google services (e.g. better translation quality in Google Translate) Serving cloud-based TPUs (tensor processing units) in order to develop machine learning software. Development of TensorFlow. The TPU research cloud provides free access to a cluster of cloud TPUs to researchers engaged in open-source machine learning research. Portal to over 5500 (as of September 2019) research publications by Google staff. Magenta: a deep learning research team exploring the role of machine learning as a tool in the creative process. The team has released many open source projects allowing artists and musicians to extend their processes using AI. With the use of Magenta, musicians and composers could create high-quality music at a lower cost, making it easier for new artists to enter the industry. Sycamore: a new 54-qubit programmable quantum processor. LaMDA: a family of conversational neural language models A program designed to address the growing need for developing free speech resources for under-represented languages