ReAct framework and ChatGPT plugins

# ReAct framework and ChatGPT plugins
## ChatGPT Plugins
If you are subscribed to ChatGPT Plus, you can easily augment LLMs with tools for personal or professional use without using the Open AI API. They made available the use of third-party [Plugins](https://openai.com/blog/chatgpt-plugins) through their chat interface. It’s not disclosed how OpenAI implement this, but they may be using a form of prompt engineering similar to ReAct to abilitate the plugins. These plugins or tools can give their language models access to more recent, personal, or specific information. Here are some of the most popular use cases plugins can help with. ![Plugins available to ChatGPT Plus subscribers](ReAct%20framework%20and%20ChatGPT%20plugins%20806d77f2ff6c4a16be58c011998b358e/Untitled%201.png) Plugins available to ChatGPT Plus subscribers With these third-party plugins, you can directly upload your documents, such as PDFs, and ask questions about the information in those documents. You can provide a link to a GitHub repository and ask questions or let the language model explain the code to you. There are also plugins to create diagrams, flow charts, or graphs.


ReAct framework and ChatGPT plugins

# ReAct framework and ChatGPT plugins
## Conclusion
In this module, we explore various ways to enhance the capabilities of current language models. These new methods also reduce the risks of [hallucinations](https://www.notion.so/Understanding-Hallucinations-and-Bias-e417eb8cce7849a58b2c8a4b0ef0f6f1?pvs=21). We learn about the ReAct framework, a tool that empowers language models to act independently. We discuss the diverse features available via OpenAI services, including function calling and third-party plugins. Function calling assists developers by allowing the use of custom functions during user interaction with the model. Concurrently, plugins available through the chat interface enable users to utilize enhanced OpenAI language models without coding.


What is LLMOps

# What is LLMOps
## Introduction
As LLMs continue to revolutionize various applications, managing their lifecycle has become important. In this lesson, we will explore the concept of LLMOps, its origins, and its significance in today's AI industry. We will also discuss the steps involved in building an LLM-powered application, the differences between LLMOps and MLOps, and the challenges and solutions associated with each step.


What is LLMOps

# What is LLMOps
## The Emergence of LLMOps
In recent years, the world of AI has witnessed the rise of large language models. These models have billions of parameters and are trained on billions of words, hence the term "large.” The advent of LLMs has led to the emergence of a new term, LLMOps, which stands for Large Language Model Operations. This lesson aims to comprehensively understand LLMOps, its origins, and its significance in the AI industry. LLMOps is essentially a set of tools and best practices designed to manage the GenAI lifecycle, from development and deployment to maintenance. LLMOps have gained traction with the rise of LLMs, particularly after the release of OpenAI's ChatGPT, which led to a surge in LLM-powered applications, such as chatbots, writing assistants, and programming assistants. However, the process of building production-ready LLM-powered applications presents unique challenges that differ from those encountered when building AI products with traditional machine learning models. This has necessitated the development of new tools and practices, giving birth to the term "LLMOps.”


What is LLMOps

# What is LLMOps
## Steps Involved in LLMOps and Differences with MLOps
While LLMOps can be considered a subset of MLOps (Machine Learning Operations), there are key differences between the two, primarily due to the differences in building AI products with classical ML models and LLMs. The process of building an LLM-powered application involves several key steps. ### 1. **Selection of a Foundation Model** Foundation models are pre-trained LLMs that can be adapted for various downstream tasks. Training these models from scratch is complex, time-consuming, and costly. Hence, developers usually opt for either proprietary models owned by large companies or open-source models hosted on community platforms like Hugging Face. This differs from standard MLOps, where a model is typically trained from scratch with a smaller architectures or on different data, especially for tabular classification and regression tasks (except for computer vision, where most applications start with a model trained on general datasets like [Imagenet](https://www.image-net.org/) or [COCO](https://cocodataset.org/#home)). Typically a dataset is splitted into training and evaluation sets, where 70% of the data go into the training set, or other evaluation techniques like [crossvalidation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) are used. When working with LLMs, this is not possible due to the high costs involved in the pretraining. MLOps models are data-hungry and require a lot (thousands at least) of labeled data to be trained on. Consequently, choosing the suitable foundation model in LLMOps is very important, as crucial as choosing a proprietary or open-source foundation model. Proprietary LLMs are usually bigger and more performant than open-source alternatives (thanks to the money investments that large corporations can make) and may also be more cost-effective for the final user as there’s no need to set up an expensive infrastructure to host the model (which organizations can do efficiently, as they have many customers and amortize the costs). On the contrary, open-source models are generally more customizable and can be improved by anyone from the open-source community, indeed they soon matched the quality of many proprietary LLMs. Another aspect to consider is the knowledge cutoff of LLMs: the date of the last published document on which the model was trained. For example, the model used in ChatGPT is currently limited to data up until September 2021. Consequently, the model can easily talk about everything that happened before that date but finds it hard to talk about later stuff. For example, ChatGPT doesn’t know about the latest startups or products released. Therefore, he may hallucinate when talking about them. ### 2. **Adaptation to Downstream Tasks** After selecting a foundation model, it can be customized for specific tasks through techniques such as prompt engineering. This involves adjusting the input to produce the desired output. It's important to keep track of the prompts used when using prompt engineering since they will likely be improved over time and can impact performance on specific tasks. By doing this, if a new prompt in production works worse than the previous one in some aspects and If we want to revert, it can be done easily. Additionally, fine-tuning can be utilized to enhance the model's performance on a specific task, requiring a high-quality dataset for it (thus, involving a data collection step). In the case of fine-tuning, there are different approaches such as fine-tuning the model, fine-tuning the instructions, or using [soft prompts](https://learnprompting.org/docs/trainable/soft_prompting). There are challenges with fine-tuning due to the large size of the model. Additionally, deploying the newly finetuned model on a new infrastructure can be difficult. To solve this problem, today, there are finetuning techniques that improve only a small subset of additional parameters to add to the existing foundational model, such as [LoRA](https://arxiv.org/abs/2106.09685). Using LoRA, it’s possible to keep the same foundation model always


What is LLMOps

# What is LLMOps
## Steps Involved in LLMOps and Differences with MLOps
deployed on the infrastructure while adding the additional finetuned parameters when needed. Recently, popular proprietary models like GPT3.5 and PaLM can now be finetuned easily directly on the company platform. When fine-tuning a model, it's essential to keep track of the dataset used and the metrics achieved. It can be helpful to use a tool like [Weights and Biases](https://wandb.ai/site), which tracks experiments and provides a dashboard where you can monitor the metrics of your fine-tuned model on an evaluation set as it is trained. This provides insights into whether the training is progressing well or not. See [this page](https://wandb.ai/site/experiment-tracking) to learn more about how W&B experiment tracking works. It will be used in the following lessons to train and fine-tune language models. ### 3. **Evaluation** Evaluating the performance of an LLM is more complex than evaluating traditional ML models. The main reason for this is that the output of an LLM is usually free text, and it’s harder to devise metrics that can be computed via code and that work well on free text. For example, try thinking about how you could evaluate the quality of an answer given by an LLM assistant whose job is to summarize YouTube videos, for which you don’t have reference summaries written by humans. Currently, organizations often resort to A/B testing to assess the effectiveness of their models, checking whether the user’s satisfaction is the same or better after the change in production. Another aspect to consider is hallucinations. How can we measure, with a metric implemented in code, whether the answer of our LLM assistant contains hallucinations? This is another open challenge where organizations mainly rely on A/B testing. ### 4. **Deployment and Monitoring** Deploying and monitoring LLMs is very important as their completions can change significantly between releases. Tools for monitoring LLMs are emerging to address this need. Another concern about LLMOps is the latency of the model. Indeed, since the model is autoregressive (i.e., produces the output one token at a time), it may take some time to output a complete paragraph. This is in contrast with the most popular applications of LLMs, which want them as assistants, which, therefore, should be able to output text at a throughput similar to a user's reading speed. One of the emerging tools in the LLMOps landscape is [W&B Prompts](https://docs.wandb.ai/guides/prompts), a suite designed specifically for the development of LLM-powered applications. W&B Prompts offers a comprehensive set of features that allow developers to visualize and inspect the execution flow of LLMs, analyze the inputs and outputs, view intermediate results, and securely manage prompts and LLM chain configurations. A key component of W&B Prompts is [Trace](https://github.com/wandb/wandb), a tool that tracks and visualizes the inputs, outputs, execution flow, and model architecture of LLM chains. Trace is particularly useful for LLM chaining, plug-in, or pipelining use cases. It provides a Trace Table for an overview of the inputs and outputs of a chain, a Trace Timeline that displays the execution flow of the chain color-coded according to component types, and a Model Architecture view that provides details about the structure of the chain and the parameters used to initialize each component. LLMOps is a rapidly evolving field, and it's hard to predict its future trajectory. However, it's clear that as LLMs become more prevalent, so will the tools and practices associated with LLMOps. The rise of LLMs and LLMOps signifies a major shift in building and maintaining AI-powered products.


What is LLMOps

# What is LLMOps
## Conclusion
In conclusion, LLMOps, or Large Language Model Operations, is a critical aspect of managing the lifecycle of applications powered by LLMs. This lesson has provided an overview of the origins and significance of LLMOps, the steps involved in building an LLM-powered application, and the differences between LLMOps and MLOps. We studied the process of selecting a foundation model, adapting it to downstream tasks, evaluating its performance, and deploying and monitoring the model. We've also highlighted the unique challenges posed by LLMs, such as the complexity of evaluating free text outputs and the need for prompt versioning and efficient deployment strategies. The emergence of tools like W&B Prompts and practices like A/B testing are indicative of the rapid evolution of LLMOps. As LLMs continue to revolutionize various applications, the tools and practices associated with LLMOps will undoubtedly become increasingly important in AI.


# Training on Generated Data and Model Collapse
## Introduction
In this lesson, we will examine the phenomenon of model collapse, its stages, its causes, and its implications on the future of Large Language Models. We also draw parallels with related concepts in machine learning, such as catastrophic forgetting and data poisoning. Finally, we contemplate the value of human-generated content in the era of dominant LLMs and the potential risk of widespread model collapse.


# Training on Generated Data and Model Collapse
## Understanding Model Collapse
Model collapse, defined in the paper “[The Curse of Recursion: Training on Generated Data Makes Models Forget](https://arxiv.org/pdf/2305.17493v2.pdf),” is a degenerative process affecting generations of learned generative models. It occurs when the data generated by a model ends up contaminating the training set of subsequent models. As a result, these models start to misinterpret reality, reinforcing their own beliefs instead of learning from real data. Here’s an image exemplifying model collapse. ![Image from the paper “[The Curse of Recursion: Training on Generated Data Makes Models Forget](https://arxiv.org/pdf/2305.17493v2.pdf).” Model Collapse refers to a degenerative learning process where models start forgetting improbable events over time as the model becomes poisoned with its projection of reality.](Training%20on%20Generated%20Data%20and%20Model%20Collapse%20b3c3d80ca3c24b70969d245c626af8ea/Screenshot_2023-08-17_at_12.16.02.png) Image from the paper “[The Curse of Recursion: Training on Generated Data Makes Models Forget](https://arxiv.org/pdf/2305.17493v2.pdf).” Model Collapse refers to a degenerative learning process where models start forgetting improbable events over time as the model becomes poisoned with its projection of reality. There are two distinct stages of model collapse: early and late. - In the **early stage**, the model begins to lose information about the tails of the distribution. - As the process progresses to the **late stage**, the model starts to entangle different modes of the original distributions, eventually converging to a distribution that bears little resemblance to the original one, often with very small variance. Here’s an example of text outputs from sequential generations of 125M parameters LLMs where each generation is trained on data produced by the previous generation. ![Image from the paper “[The Curse of Recursion: Training on Generated Data Makes Models Forget](https://arxiv.org/pdf/2305.17493v2.pdf).”](Training%20on%20Generated%20Data%20and%20Model%20Collapse%20b3c3d80ca3c24b70969d245c626af8ea/Screenshot_2023-08-17_at_12.18.00.png) Image from the paper “[The Curse of Recursion: Training on Generated Data Makes Models Forget](https://arxiv.org/pdf/2305.17493v2.pdf).”


# Training on Generated Data and Model Collapse
## Related Work on Model Collapse
Model collapse shares similarities with two concepts in machine learning literature: catastrophic forgetting and data poisoning. - [Catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference), a challenge in continual learning, refers to the model's tendency to forget previous samples when learning new information. This is particularly relevant in task-free continual learning, where data distributions gradually change without the notion of separate tasks. However, in the context of model collapse, the changed data distributions arise from the model itself as a result of training in the previous iteration. - On the other hand, [data poisoning](https://en.wikipedia.org/wiki/Adversarial_machine_learning#Data_poisoning) involves the insertion of malicious data during training to degrade the model’s performance. This concept becomes increasingly relevant with the rise of contrastive learning and LLMs trained on untrustworthy web sources. Yet, neither catastrophic forgetting nor data poisoning fully explain the phenomenon of model collapse, as they don't account for the self-reinforcing distortions of reality seen in model collapse. However, understanding these related concepts can provide additional insights into model collapse mechanisms and potential mitigation strategies.


# Training on Generated Data and Model Collapse
## Causes of Model Collapse
Model collapse primarily results from two types of errors: statistical approximation error and functional approximation error. - The statistical approximation error is the primary cause. It arises due to the finite number of samples used in training. Despite using a large number of points, significant errors can still occur. This is because there's always a non-zero probability that information can get lost at every step of re-sampling. - The functional approximation error is a secondary cause. It stems from the limitations of our function approximators. Even though neural networks are theoretically capable of approximating any function, in practice, they can introduce non-zero likelihood outside the support of the original distribution, leading to errors.


# Training on Generated Data and Model Collapse
## The Future of the Web with Dominant LLMs
As LLMs become more prevalent in the online text and image ecosystem, they will inevitably train on data produced by their predecessors. This could lead to a cycle where each model generation learns more from previous models' output and less from original human-generated content. The result is a risk of widespread model collapse, with models progressively losing touch with the true underlying data distribution. The model collapse has far-reaching implications. As models start to misinterpret reality, generated content quality could degrade over time. This could profoundly affect various LLMs' applications, from content creation to decision-making systems.


# Training on Generated Data and Model Collapse
## The Value of Human-Generated Content
In the face of model collapse, preserving and accessing data collected from genuine human interactions becomes increasingly valuable. Real human-produced data provides access to the original data distribution, which is crucial in learning where the tails of the underlying distribution matter. As LLMs increasingly generate online content, data from human interactions with these models will become an increasingly valuable resource for training future models.


# Training on Generated Data and Model Collapse
## Conclusion
In this lesson, we've explored the phenomenon of model collapse, a degenerative process that can affect generative models when they are trained on data produced by other models. We've examined the stages of model collapse, from the early loss of information about the tails of distribution to the late-stage entanglement of different modes. We've drawn parallels with related concepts in machine learning: catastrophic forgetting and data poisoning. We also dissected the leading causes of model collapse, namely statistical and functional approximation errors. As Large Language Models become more dominant in the digital landscape, the risk of widespread model collapse increases, potentially leading to a degradation in generated content quality. In this context, we've underscored the importance of preserving and accessing human-generated content, which provides a crucial link to the original data distribution and serves as a valuable resource for training future models. As we continue to harness the power of LLMs, understanding and mitigating model collapse will be essential in ensuring the quality and reliability of their outputs.


Deep Dive into LoRA and SFT

# Deep Dive into LoRA and SFT
## Introduction
In this lesson, we will dive deeper into the mechanics of LoRA, a powerful method for optimizing the fine-tuning process of Large Language Models, its practical uses in various fine-tuning tasks, and the open-source resources that simplify its implementation. We will also introduce QLoRA, a highly efficient version of LoRA. By the end of this lesson, you will have an in-depth understanding of how LoRA and QLoRA can enhance the efficiency and accessibility of fine-tuning LLMs.


Deep Dive into LoRA and SFT

# Deep Dive into LoRA and SFT
## **The Functioning of LoRA in Fine-tuning LLMs**
[LoRA](https://arxiv.org/abs/2106.09685), or Low-Rank Adaptation, is a method developed by Microsoft researchers to optimize the fine-tuning of Large Language Models. This technique tackles the issues related to the fine-tuning process, such as extensive memory demands and computational inefficiency. LoRA introduces a compact set of parameters, referred to as **low-rank matrices**, to store the necessary changes in the model instead of altering all parameters. Here are the key features of how LoRA operates: - **Maintaining Pretrained Weights**: LoRA adopts a unique strategy by preserving the pretrained weights of the model. This approach reduces the risk of catastrophic forgetting, ensuring the model maintains the valuable knowledge it gained during pretraining. - **Efficient Rank-Decomposition**: LoRA incorporates rank-decomposition weight matrices, known as update matrices, to the existing weights. These update matrices have significantly fewer parameters than the original model, making them highly memory-efficient. By training only these newly added weights, LoRA achieves a faster training process with reduced memory demands. These LoRA matrices are typically integrated into the attention layers of the original model. By using the low-rank decomposition approach, the memory demands for training large language models are significantly reduced. This allows running fine-tuning tasks on consumer-grade GPUs, making the benefits of LoRA available to a broader range of researchers and developers.


Deep Dive into LoRA and SFT

# Deep Dive into LoRA and SFT
## **Open-source Resources for LoRA**
The following libraries offer a mix of tools that enhance the efficiency of fine-tuning large language models. They provide optimizations, compatibility with different data types, resource efficiency, and user-friendly interfaces that accommodate various tasks and hardware configurations. - **[PEFT Library](https://github.com/huggingface/peft)**: Parameter-efficient fine-tuning (PEFT) methods facilitate efficient adaptation of pre-trained language models to various downstream applications without fine-tuning all the model's parameters. By fine-tuning only a portion of the model's parameters, PEFT methods like LoRA, Prefix Tuning, and P-Tuning, including QLoRA, significantly reduce computational and storage costs. - **[Lit-GPT](https://github.com/Lightning-AI/lit-gpt):** Lit-GPT from LightningAI is an open-source resource designed to simplify the fine-tuning process, making it easier to apply LoRA's techniques without manually altering the core model architecture. Models available for this purpose include [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/), [Pythia](https://www.eleuther.ai/papers-blog/pythia-a-suite-for-analyzing-large-language-modelsacross-training-and-scaling), and [Falcon](https://falconllm.tii.ae/). Specific configurations can be applied to different weight matrices, and precision settings can be adjusted to manage memory consumption. In this course, we’ll mainly use the PEFT library.


Deep Dive into LoRA and SFT

# Deep Dive into LoRA and SFT
## QLoRA: An Efficient Variant of LoRA
[QLoRA](https://arxiv.org/abs/2305.14314), or Quantized Low-Rank Adaptation, is a popular variant of LoRA that makes fine-tuning large language models even more efficient. QLoRA introduces several innovations to save memory without sacrificing performance. The technique involves backpropagating gradients through a frozen, 4-bit quantized pretrained language model into Low-Rank Adapters. This approach significantly reduces memory usage, enabling the fine-tuning of even larger models on consumer-grade GPUs. For instance, QLoRA can fine-tune a 65 billion parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning task performance. QLoRA uses a new data type known as 4-bit NormalFloat (NF4), which is optimal for normally distributed weights. It also employs double quantization to reduce the average memory footprint by quantizing the quantization constants and paged optimizers to manage memory spikes. The [Guanaco](https://huggingface.co/TheBloke/guanaco-65B-GPTQ) models, which use QLoRA fine-tuning, have demonstrated state-of-the-art performance, even when using smaller models than the previous benchmarks. This shows the power of QLoRA tuning, making it a popular choice for those seeking to democratize the use of large transformer models. The practical implementation of QLoRA for fine-tuning LLMs is very accessible, thanks to open-source libraries and tools. For instance, the [BitsAndBytes library](https://github.com/TimDettmers/bitsandbytes) offers functionalities for 4-bit quantization. We’ll later see a code example showing how to use QLoRA with PEFT.


Deep Dive into LoRA and SFT

# Deep Dive into LoRA and SFT
## Conclusion
In this lesson, we focused on LoRA and QLoRA, two powerful techniques for fine-tuning LLMs. We explored how LoRA works, preserving pretrained weights and introducing low-rank matrices to make the fine-tuning process more memory and computationally efficient. We also introduced open-source libraries like PEFT and Lit-GPT that facilitate the implementation of LoRA. Finally, we discussed QLoRA, an efficient variant of LoRA that uses 4-bit NormalFloat and double quantization to reduce memory usage.


Evaluating LLM Performance

# Evaluating LLM Performance
## Introduction
In this lesson, we will explore two crucial aspects of language model evaluation: objective functions and evaluation metrics. Objective functions, also known as loss functions, play a vital role in guiding the learning process during model training. On the other hand, evaluation metrics provide interpretable measures of the model's capabilities and are used to assess its performance on various tasks. We will dive into the perplexity evaluation metric, commonly used for LLMs, and explore several benchmarking frameworks, such as GLUE, SuperGLUE, BIG-bench, HELM, and FLASK, that help comprehensively evaluate language models across diverse scenarios.


Evaluating LLM Performance

# Evaluating LLM Performance
## Objective Functions and Evaluation Metrics
Objective functions and evaluation metrics are essential components in machine learning models. The **objective function**, also known as the **loss function**, is a mathematical formula used during the training phase. It gives a loss score to the model in function of the model parameters. During training, the learning algorithm computes gradients of the loss function and updates the model parameters to minimize it. As a consequence, to guarantee (smooth) learning, the loss function needs to be differentiable and have an excellent smooth form. The objective function typically used for LLMs is the **cross-entropy loss**. In the case of causal language modeling, the model predicts the next token from a fixed list of tokens, essentially making it a classification problem. On the other hand, **evaluation metrics** are used to assess the model's performance in an interpretable way for people. Unlike the objective function, evaluation metrics are not directly used during training. As a consequence, evaluation metrics don’t need to be differentiable, as we won’t have to compute gradients for them. Standard evaluation metrics include accuracy, precision, recall, F1-score, and mean squared error. Typical evaluation metrics for LLMs can be: - **Intrinsic metrics**, i.e., metrics strictly related to the training objective. A popular example is the **perplexity** metric. - **Extrinsic metrics** are metrics that aim to assess performance on several downstream tasks and are not strictly related to the training objective. The GLUE, SuperGLUE, BIG-bench, HELM, and FLASK benchmarks are popular examples.


Evaluating LLM Performance

# Evaluating LLM Performance
## The Perplexity Evaluation Metric
Perplexity is an evaluation metric used to assess the performance of LLMs. It measures how well a language model predicts a given sample or sequence of words, such as a sentence. The lower the perplexity value, the better the language model is at predicting the sample. LLMs are designed to model the probability distributions of words within sentences. They can generate sentences resembling human writing and assess the sentences' quality. Perplexity is a measure that quantifies the uncertainty or "perplexity" a model experiences when assigning probabilities to sequences of words. The first step in computing perplexity is to calculate the probability of a sentence by multiplying the probabilities of individual words according to the language model. Longer sentences tend to have lower probabilities due to the multiplication of factors smaller than one. To make comparisons between sentences with different lengths possible, perplexity normalizes the probability by dividing it by the number of words in the sentence and taking the geometric mean. ### Perplexity Example Consider an example where a language model is trained to predict the subsequent word in a sentence: "A red fox." For a competent LLM, the predicted word probabilities could be as follows, step by step. > P(“a red fox.”) = > > > = P(“a”) * P(“red” | “a”) * P(“fox” | “a red”) * P(“.” | “a red fox”) = > > = 0.4 * 0.27 * 0.55 * 0.79 = > > = 0.0469 > It would be nice to compare the probabilities assigned to different sentences to see which sentences are better predicted by the language model. However, since the probability of a sentence is obtained from a product of probabilities, the longer the sentence, the lower its probability (since it’s a product of factors with values smaller than one). We should find a way of measuring these sentence probabilities without the influence of the sentence length. This can be done by normalizing the sentence probability by the number of words in the sentence. Since the probability of a sentence is obtained by multiplying many factors, we can average them using the [geometric mean](https://en.wikipedia.org/wiki/Geometric_mean). Let’s call *Pnorm(W)* the normalized probability of the sentence *W*. Let *n* be the number of words in *W*. Then, applying the geometric mean: > Pnorm(W) = P(W) ^ (1 / n) > Using our specific sentence, “*a red fox.*”: > Pnorm(“a red fox.”) = P(“a red ”) ^ (1 / 4) = 0.465 > Great! This number can now be used to compare the probabilities of sentences with different lengths. The higher this number is over a well-written sentence, the better the language model. So, what does this have to do with perplexity? Well, perplexity is just the reciprocal of this number. Let’s call *PP(W)* the perplexity computed over the sentence *W*. Then: > PP(W) = 1 / Pnorm(W) = > > > = 1 / (P(W) ^ (1 / n)) > > = (1 / P(W)) ^ (1 / n) > Let’s compute it with `numpy`: ```python import numpy as np probabilities = np.array([0.4, 0.27, 0.55, 0.79]) sentence_probability = probabilities.prod() sentence_probability_normalized = sentence_probability ** (1 / len(probabilities)) perplexity = 1 / sentence_probability_normalized print(perplexity) # 2.1485556947850033 ``` Suppose we further train the LLM, and the probabilities of the next best word become higher. How would the final perplexity be, higher or lower? ```python probabilities = np.array([0.7, 0.5, 0.6, 0.9]) sentence_probability = probabilities.prod() sentence_probability_normalized = sentence_probability ** (1 / len(probabilities)) perplexity = 1 / sentence_probability_normalized print(perplexity) # 1.516647134682679 -> lower ```


Evaluating LLM Performance

# Evaluating LLM Performance
## The GLUE Benchmark
The [GLUE](https://gluebenchmark.com/) (General Language Understanding Evaluation) benchmark comprises nine diverse English sentence understanding tasks categorized into three groups. - The first group, Single-Sentence Tasks, evaluates the model's ability to determine grammatical correctness (CoLA) and sentiment polarity (SST-2) of individual sentences. - The second group, Similarity, and Paraphrase Tasks, focuses on assessing the model's capacity to identify paraphrases in sentence pairs (MRPC and QQP) and determine the similarity score between sentences (STS-B). - The third group, Inference Tasks, challenges the model to handle sentence entailment and relationships. This includes recognizing textual entailment (RTE), answering questions based on sentence information (QNLI), and resolving pronoun references (WNLI). The final GLUE score is obtained by averaging performance across all nine tasks. By providing a unified evaluation platform, GLUE facilitates a deeper understanding of the strengths and weaknesses of various NLP models.


Evaluating LLM Performance

# Evaluating LLM Performance
## The **SuperGLUE Benchmark**
The [SuperGLUE](https://super.gluebenchmark.com/) benchmark builds upon the GLUE benchmark but introduces more complex tasks to push the boundaries of current NLP approaches. The key features of SuperGLUE are: 1. Tasks: SuperGLUE consists of eight diverse language understanding tasks. These tasks include Boolean question answering, textual entailment, coreference resolution, reading comprehension with commonsense reasoning, and word sense disambiguation. 2. Difficulty: The benchmark retains the two hardest tasks from GLUE and adds new tasks based on the challenges faced by current NLP models, ensuring greater complexity and relevance to real-world language understanding scenarios. 3. Human Baselines: Human performance estimates are included for each task, providing a benchmark for evaluating the performance of NLP models against human-level understanding. 4. Evaluation: NLP models are evaluated on these tasks, and their performance is measured using a single-number overall score obtained by averaging the scores of all individual tasks.


Evaluating LLM Performance

# Evaluating LLM Performance
## The BIG-Bench Benchmark
[BIG-bench](https://github.com/google/BIG-bench) is a large-scale and diverse benchmark designed to evaluate the capabilities of large language models. It consists of 204 or more language tasks that cover a wide range of topics and languages. These are challenging and not entirely solvable by current models. The benchmark supports two types of tasks: JSON-based and programmatic tasks. JSON tasks involve comparing output and target pairs to evaluate performance, while programmatic tasks use Python to measure text generation and conditional log probabilities. The tasks include writing code, common-sense reasoning, playing games, linguistics, and more. The researchers found that aggregate performance improves with model size but still falls short of human performance. Model predictions become better calibrated with increased scale, and sparsity offers benefits. This benchmark is considered a "living benchmark," accepting new task submissions for continuous peer review. The code for BIG-bench is open-source on [GitHub](https://github.com/google/BIG-bench), and the research paper is available on [arXiv](https://arxiv.org/abs/2206.04615).


Evaluating LLM Performance

# Evaluating LLM Performance
## The HELM Benchmark
The [HELM](https://crfm.stanford.edu/2022/11/17/helm.html) (Holistic Evaluation of Language Models) benchmark addresses the lack of a unified standard for comparing language models and aims to assess them in their totality. The benchmark has three main components: 1. Broad Coverage and Recognition of Incompleteness: HELM evaluates language models over a diverse set of scenarios, considering different tasks, domains, languages, and user-facing applications. It acknowledges that not all scenarios can be covered but explicitly identifies major scenarios and missing metrics to highlight improvement areas. 2. Multi-Metric Measurement: HELM evaluates language models based on multiple criteria, unlike previous benchmarks that often focus on a single metric like accuracy. It measures 7 metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. This multi-metric approach ensures that non-accuracy desiderata are not overlooked. 3. Standardization: HELM aims to standardize the evaluation process for different language models. It specifies an adaptation procedure using few-shot prompting, making it easier to compare models effectively. By evaluating 30 models from various providers, HELM improves the overall landscape of language model evaluation and encourages a more transparent and reliable infrastructure for language technologies.


Evaluating LLM Performance

# Evaluating LLM Performance
## The FLASK Benchmark
The [FLASK](https://arxiv.org/abs/2307.10928) (Fine-grained Language Model Evaluation based on Alignment Skill Sets) benchmark is an evaluation protocol for LLMs. It breaks down the evaluation process into 12 specific instance-wise skill sets, each representing a crucial aspect of a model's capabilities. These skill sets comprise logical correctness, logical efficiency, factuality, commonsense understanding, comprehension, insightfulness, completeness, metacognition, readability, conciseness, and harmlessness. By breaking down the evaluation into these specific skill sets, FLASK allows for a precise and comprehensive assessment of a model's performance across various tasks, domains, and difficulty levels. This approach provides a more detailed and nuanced understanding of a language model's strengths and weaknesses, enabling researchers and developers to improve the models in targeted ways and address specific challenges in natural language processing. ![Assessing skills across diverse tasks for a range of LLMs, image credit: [https://arxiv.org/pdf/2307.10928.pdf](https://arxiv.org/pdf/2307.10928.pdf)](Evaluating%20LLM%20Performance%204fb6aa977aaa44a09185b6086c58977a/flask.png) Assessing skills across diverse tasks for a range of LLMs, image credit: [https://arxiv.org/pdf/2307.10928.pdf](https://arxiv.org/pdf/2307.10928.pdf)


Evaluating LLM Performance

# Evaluating LLM Performance
## Conclusion
In this lesson, we explored the essential concepts of evaluating LLM performance through objective functions and evaluation metrics. The objective or loss function plays a critical role during model training. It guides the learning algorithm to minimize the loss score by updating model parameters. For LLMs, the common objective function is the cross-entropy loss. On the other hand, evaluation metrics are used to assess the model's performance more interpretably, though they are not directly used during training. Perplexity is one such intrinsic metric used to measure how well an LLM predicts a given sample or sequence of words. Additionally, the lesson introduced several popular extrinsic evaluation benchmarks, such as GLUE, SuperGLUE, BIG-bench, HELM, and FLASK, which evaluate language models on diverse tasks and scenarios, covering aspects like accuracy, fairness, robustness, and more. By understanding these concepts and using appropriate evaluation metrics and benchmarks, researchers and developers can gain valuable insights into language models' strengths and weaknesses, leading to improving these technologies.


Transformers Architectures

# Transformers Architectures
## Introduction
The transformer architecture has demonstrated its versatility in various applications. The original network was presented as an encoder-decoder architecture for translation tasks. The next evolution of transformer architecture began with the introduction of encoder-only models like BERT, followed by the introduction of decoder-only networks in the first iteration of GPT models. The differences extend beyond just network design and also encompass the learning objectives. These contrasting learning objectives play a crucial role in shaping the model's behavior and outcomes. Understanding these differences is essential for selecting the most suitable architecture for a given task and achieving optimal performance in various applications. In this lesson, we will explore the distinctions between these architectures by loading pre-trained models. The goal is to dive deeper into each architecture.


Transformers Architectures

# Transformers Architectures
## The Encoder-Decoder Architecture
![Image taken from the “*Attention is all you need” paper*](Understanding%20Transformers%2086c4c813b12f485283a3806ec8e25381/Untitled.png) Image taken from the “*Attention is all you need” paper* The encoder-decoder, also known as the full transformer architecture, comprises multiple stacked encoder components connected to several stacked decoder components through a cross-attention mechanism. It is notably well-suited for sequence-to-sequence (i.e., handling text as both input and output) tasks such as translation or summarization, mainly when designing models with multi-modality, like image captioning with the image as input and the corresponding caption as the expected output. Cross-attention will help the decoder focus on the most important part of the content during the generation process. A notable example of this approach is the BART pre-trained model. The architecture incorporates a bi-directional encoder responsible for creating a comprehensive representation of the input, while an autoregressive decoder generates the output one token at a time. The model takes in a randomly masked input along with the input shifted by one token and attempts to reconstruct the original input as a learning objective. The provided code below loads the BART model so we can examine its architecture. ```python from transformers import AutoModel, AutoTokenizer BART = AutoModel.from_pretrained("facebook/bart-large") print(BART) ``` ```python BartModel( (shared): Embedding(50265, 1024, padding_idx=1) (encoder): BartEncoder( (embed_tokens): Embedding(50265, 1024, padding_idx=1) (embed_positions): BartLearnedPositionalEmbedding(1026, 1024) (layers): ModuleList( (0-11): 12 x BartEncoderLayer( (self_attn): BartAttention( (k_proj): Linear(in_features=1024, out_features=1024, bias=True) (v_proj): Linear(in_features=1024, out_features=1024, bias=True) (q_proj): Linear(in_features=1024, out_features=1024, bias=True) (out_proj): Linear(in_features=1024, out_features=1024, bias=True) ) (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (activation_fn): GELUActivation() (fc1): Linear(in_features=1024, out_features=4096, bias=True) (fc2): Linear(in_features=4096, out_features=1024, bias=True) (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) ) (layernorm_embedding): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) (decoder): BartDecoder( (embed_tokens): Embedding(50265, 1024, padding_idx=1) (embed_positions): BartLearnedPositionalEmbedding(1026, 1024) (layers): ModuleList( (0-11): 12 x BartDecoderLayer( (self_attn): BartAttention( (k_proj): Linear(in_features=1024, out_features=1024, bias=True) (v_proj): Linear(in_features=1024, out_features=1024, bias=True) (q_proj): Linear(in_features=1024, out_features=1024, bias=True) (out_proj): Linear(in_features=1024, out_features=1024, bias=True) ) (activation_fn): GELUActivation() (self_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (encoder_attn): BartAttention( (k_proj): Linear(in_features=1024, out_features=1024, bias=True) (v_proj): Linear(in_features=1024, out_features=1024, bias=True) (q_proj): Linear(in_features=1024, out_features=1024, bias=True) (out_proj): Linear(in_features=1024, out_features=1024, bias=True) ) (encoder_attn_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) (fc1): Linear(in_features=1024, out_features=4096, bias=True) (fc2): Linear(in_features=4096, out_features=1024, bias=True) (final_layer_norm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) ) (layernorm_embedding): LayerNorm((1024,), eps=1e-05, elementwise_affine=True) ) ) ``` We are already familiar with most of the layers in the BART model. The model is comprised of both encoder and decoder components, with each component consisting of 12 layers. Additionally, The decoder component, in particular, contains an additional `encoder_attn` layer, referred to as cross-attention. The cross-attention component will condition the decoder’s output based on the encoder representations. We can use the fine-tuned version of this model for summarization using the Transformer’s pipeline functionality. ```python from transformers import pipeline summarizer = pipeline("summarization", model="facebook/bart-large-cnn") sum = summarizer("""Gaga was best known in the 2010s for pop hits like “Poker Face” and avant-garde experimentation on albums like “Artpop,” and Bennett, a singer who mostly stuck to standards, was in his 80s when the pair met. And yet Bennett and Gaga became fast friends and close collaborators, which they remained until Bennett’s death at 96 on Friday. They recorded two albums together, 2014’s “Cheek to Cheek” and 2021’s “Love for Sale,” which both won Grammys for best traditional pop vocal album.""", min_length=20, max_length=50) print(sum[0]['summary_text']) ``` ``` Bennett and Gaga became fast friends and close collaborators. They recorded two albums together, 2014's "Cheek to Cheek" and 2021's "Love for Sale" ```


Transformers Architectures

# Transformers Architectures
## The Encoder-Only Architecture
![Image taken from the “BERT: Pre-training of Deep Bidirectional Transformers for LanguageUnderstanding” paper](Transformers%20Architectures%20f461b29896544831a22e459475ce4023/Untitled.png) Image taken from the “BERT: Pre-training of Deep Bidirectional Transformers for LanguageUnderstanding” paper As implied by the name, the encoder-only models are formed by stacking multiple encoder components. As the encoder output cannot be connected to another decoder, its output can be directly used as a text-to-vector method, for instance, to measure similarity. Alternatively, it can be combined with a classification head (feedforward layer) on top to facilitate label prediction (it is also known as a Pooler layer in libraries such as Huggingface). The primary distinction in the encoder-only architecture lies in the absence of the Masked Self-Attention layer. As a result, the encoder can handle the entire input simultaneously. This differs from decoders, where future tokens need to be masked during training to prevent “cheating” when generating new tokens. Due to this property, they are ideally suited for creating representations from a document while retaining complete information. The BERT paper (or an improved variant like RoBERTa) introduced a widely recognized pre-trained model that significantly improved the state-of-the-art scores on numerous NLP tasks. The model undergoes pre-training with two learning objectives: 1. Masked Language Modeling: masking random tokens from the input and attempting to predict them. 2. Next Sentence Prediction: Present sentences in pairs and assess the likelihood of the second sentence in the subsequent sequence of the first sentence. ```python BERT = AutoModel.from_pretrained("bert-base-uncased") print(BERT) ``` ```python BertModel( (embeddings): BertEmbeddings( (word_embeddings): Embedding(30522, 768, padding_idx=0) (position_embeddings): Embedding(512, 768) (token_type_embeddings): Embedding(2, 768) (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) (encoder): BertEncoder( (layer): ModuleList( (0-11): 12 x BertLayer( (attention): BertAttention( (self): BertSelfAttention( (query): Linear(in_features=768, out_features=768, bias=True) (key): Linear(in_features=768, out_features=768, bias=True) (value): Linear(in_features=768, out_features=768, bias=True) (dropout): Dropout(p=0.1, inplace=False) ) (output): BertSelfOutput( (dense): Linear(in_features=768, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) (intermediate): BertIntermediate( (dense): Linear(in_features=768, out_features=3072, bias=True) (intermediate_act_fn): GELUActivation() ) (output): BertOutput( (dense): Linear(in_features=3072, out_features=768, bias=True) (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) ) ) ) ) (pooler): BertPooler( (dense): Linear(in_features=768, out_features=768, bias=True) (activation): Tanh() ) ) ``` The BERT model adopts the conventional transformer architecture for input embedding and 12 encoder blocks. However, the network’s output will be passed on to a pooler layer, which is a feed-forward linear layer followed by non-linearity that will generate the final representation. This representation will subsequently be utilized for various tasks, such as classification or similarity assessment. The following code uses the fine-tuned version of the BERT model for sentiment analysis. ```python classifier = pipeline("text-classification", model="nlptown/bert-base-multilingual-uncased-sentiment") lbl = classifier("""This restaurant is awesome.""") print(lbl) ``` ```python [{'label': '5 stars', 'score': 0.8550480604171753}] ```


Transformers Architectures

# Transformers Architectures
## The Decoder-Only Architecture
![Image taken from “Improving language understanding with unsupervised learning” paper](Transformers%20Architectures%20f461b29896544831a22e459475ce4023/Untitled%201.png) Image taken from “Improving language understanding with unsupervised learning” paper The decoder-only networks continue to serve as the foundation for most large language models today, with slight variations in some instances. Because of the implementation of masked self-attention, their primary use case revolves around the next-token-prediction task, which sparked the concept of prompting. Research demonstrated that scaling up the decoder-only models can significantly enhance the network's language understanding and generalization capabilities. As a result, they can excel at a diverse range of tasks simply by using different prompts. Large pre-trained models like GPT-4 and LLaMA 2 exhibit the ability to perform tasks such as classification, summarization, translation, etc., by leveraging the appropriate prompt. The large language models, such as those in the GPT family, undergo pre-training using the Causal Language Modeling objective. This means the model aims to predict the next word, while the attention mechanism can only attend to previous tokens on the left. This implies that the model can solely rely on the previous context to predict the next token and is unable to peek at future tokens, preventing any form of cheating. ```python gpt2 = AutoModel.from_pretrained("gpt2") print(gpt2) ``` ```python GPT2Model( (wte): Embedding(50257, 768) (wpe): Embedding(1024, 768) (drop): Dropout(p=0.1, inplace=False) (h): ModuleList( (0-11): 12 x GPT2Block( (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (attn): GPT2Attention( (c_attn): Conv1D() (c_proj): Conv1D() (attn_dropout): Dropout(p=0.1, inplace=False) (resid_dropout): Dropout(p=0.1, inplace=False) ) (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True) (mlp): GPT2MLP( (c_fc): Conv1D() (c_proj): Conv1D() (act): NewGELUActivation() (dropout): Dropout(p=0.1, inplace=False) ) ) ) (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True) ) ``` When examining the architecture, you will notice the standard transformer decoder block with the cross-attention removed. The GPT family also employs different linear layers (Conv1D) that transpose the weights. (Please note that this should not be confused with PyTorch's convolutional layer.) This design choice is specific to OpenAI, while other open-source large language models use the standard linear layer. The provided code illustrates how the pipeline can be used to incorporate the GPT2 model for text generation. It generates four different alternatives to complete the phrase "This movie was a very.” ```python generator = pipeline(model="gpt2") output = generator("This movie was a very", do_sample=True, top_p=0.95, num_return_sequences=4, max_new_tokens=50, return_full_text=False) for item in output: print(">", item['generated_text']) ``` ``` > hard thing to make, but this movie is still one of the most amazing shows I've seen in years. You know, it's sort of fun for a couple of decades to watch, and all that stuff, but one thing's for sure — > special thing and that's what really really made this movie special," said Kiefer Sutherland, who co-wrote and directed the film's cinematography. "A lot of times things in our lives get passed on from one generation to another, whether > good, good effort and I have no doubt that if it has been released, I will be very pleased with it." Read more at the Mirror. > enjoyable one for the many reasons that I would like to talk about here. First off, I'm not just talking about the original cast, I'm talking about the cast members that we've seen before and it would be fair to say that none of ``` <aside> 💡 Please be aware that running the above code will yield different outputs due to the randomness involved in the generation process. </aside>


Transformers Architectures

# Transformers Architectures
## Conclusion
In this lesson, we explored the various types of transformer-based models and their areas of maximum effectiveness. While LLMs may appear to be the ultimate solution for every task, it's essential to note that there are instances where smaller, more focused models can produce equally good results while operating more efficiently. Using a small model like DistilBERT on your local server to measure similarity could be more suitable for specific applications while offering a cost-effective alternative to using proprietary models and APIs. Moreover, the transformer paper introduced an effective architecture. However, various architectures have been experimented with minor code changes, such as different embedding sizes and hidden dimensions. Recent experiments have also shown that relocating the batch normalization layer before the attention mechanism can enhance the model's capabilities. Keep in mind that there could be slight variations in the architecture, especially for proprietary models like GPT-3 that have not released their code. In this [Notebook](https://colab.research.google.com/drive/1k2UF8wO0hYF8Xuj2udVhoRG9hmAx_oR8?usp=sharing), you can find the code for this lesson.


A Timeline of Large Language Models

# A Timeline of Large Language Models
## Introduction
In this lesson, we'll explore the transformative shifts that have reshaped AI, exploring the key features that set LLMs apart from their predecessors. We’ll see how scaling laws, emergent abilities, and innovative architectures have propelled LLMs to tackle complex tasks and define the current landscape of popular LLMs. ---


A Timeline of Large Language Models

# A Timeline of Large Language Models
## From Language Models to Large Language Models
The evolution of language models has undergone a transformative shift from pre-trained language models (LMs) to the emergence of large language models (LLMs). LMs, like ELMo and BERT, initially captured context-aware word representations through pre-training and fine-tuning for specific tasks. However, the introduction of LLMs, exemplified by GPT-3 and PaLM, demonstrated that scaling model size and data can unlock emergent abilities, exceeding the capabilities of their smaller counterparts. These LLMs can tackle more complex tasks through in-context learning. The following image shows the trends of the cumulative numbers of arXiv papers containing the keyphrases “language model” and “large language model,” emphasizing the growing interest in them in the last years. ![From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper](A%20Timeline%20of%20Large%20Language%20Models%20ceae1b1e9692468ca3cfb2d5925164d3/Screenshot_2023-08-09_at_11.15.46.png) From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper


A Timeline of Large Language Models

# A Timeline of Large Language Models
## **Key Characterizing Features of LLMs**
Here are the main characteristics that differentiate LLMs from the previous models: 1. **Scaling Laws for Enhanced Capacity:** Scaling laws play a crucial role in LLM development, indicating a relationship between model performance, model size, dataset size, and training compute. The [KM scaling laws](https://arxiv.org/abs/2001.08361) emphasize the impact of these factors, revealing distinct formulas for their influence on cross-entropy loss. The [Chinchilla scaling laws](https://arxiv.org/abs/2203.15556) provide an alternate approach, optimizing compute allocation between model and data size. 2. **Emergent Abilities:** LLMs possess emergent abilities, defined as capabilities that manifest in large models but are absent in smaller counterparts. One prominent emergent ability is *in-context learning* (ICL), showcased by models like GPT-3. ICL allows LLMs to generate unexpected outputs based on natural language instructions, eliminating the need for further training. 3. **Instruction following**: LLMs can be finetuned to follow text instructions, which further enhances generalization for new tasks. 4. **Step-by-Step Reasoning:** LLMs can perform *step-by-step reasoning* using the [chain-of-thought (CoT)](https://arxiv.org/abs/2201.11903) prompting strategy. This mechanism enables them to solve complex tasks by breaking them down into intermediate reasoning steps, which is particularly beneficial for tasks involving multiple steps like mathematical word problems.


A Timeline of Large Language Models

# A Timeline of Large Language Models
## A Timeline of the Most Popular LLMs
Here’s an overview of the timeline of the most popular LLMs in the last years. ![From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper](A%20Timeline%20of%20Large%20Language%20Models%20ceae1b1e9692468ca3cfb2d5925164d3/timeline.png) From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper **Here is a brief description of some of them.** - **[2018]** [GPT-1](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf) GPT-1 (Generative Pre-Training 1) was introduced by OpenAI in 2018. It laid the foundation for the GPT-series models by employing a generative, decoder-only Transformer architecture. It combined unsupervised pretraining and supervised fine-tuning to predict the next word in natural language text. - **[2019]** [GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) Building upon the architecture of GPT-1, GPT-2 was released in 2019 with an increased parameter scale of 1.5 billion. This model demonstrated potential for solving a variety of tasks using language text as a unified format for input, output, and task information. - **[2020]** [GPT-3](https://arxiv.org/abs/2005.14165) Released in 2020, GPT-3 marked a significant capacity leap by scaling the model to 175 billion parameters. It introduced the concept of in-context learning (ICL), enabling LLMs to understand tasks through few-shot or zero-shot learning. GPT-3 showcased excellent performance in numerous NLP tasks, including reasoning and domain adaptation, highlighting the potential of scaling up model size. - **[2021]** [Codex](https://en.wikipedia.org/wiki/OpenAI_Codex) Codex was introduced by OpenAI in July 2021 as a fine-tuned version of GPT-3 specifically trained on a large corpus of GitHub code. It demonstrated enhanced ability in solving programming and mathematical problems, showcasing the potential of training LLMs on specialized data. - **[2021]** [LaMDA](https://blog.google/technology/ai/lamda/) LaMDA (Language Models for Dialog Applications) was introduced by researchers from DeepMind. LaMDA focuses on enhancing dialog applications and dialog generation tasks. It has a significant number of parameters, with the largest model consisting of 137 billion parameters, making it slightly smaller than GPT-3. - **[2021]** [Gopher](https://www.deepmind.com/blog/language-modelling-at-scale-gopher-ethical-considerations-and-retrieval) In 2021, DeepMind introduced Gopher, a language model with an impressive parameter scale of 280 billion. Notably, Gopher demonstrated a remarkable capability to approach human expert performance on the Massive Multitask Language Understanding (MMLU) benchmark. However, like its predecessors, Gopher exhibited certain limitations, including tendencies for repetition, biases, and propagation of incorrect information. - **[2022]** [InstructGPT](https://arxiv.org/abs/2203.02155) In 2022, InstructGPT was proposed as an enhancement to GPT-3 for human alignment. It utilized reinforcement learning from human feedback (RLHF) to improve the model's instruction-following capacity and mitigate issues like generating harmful content. This approach proved valuable for training LLMs to align with human preferences. - **[2022]** [Chinchilla](https://arxiv.org/abs/2203.15556) Chinchilla, introduced in 2022 by DeepMind, is a family of large language models that build upon the discovered scaling laws of LLMs. With a focus on efficient utilization of compute resources, Chinchilla boasts 70 billion parameters and achieves a remarkable 67.5% accuracy on the MMLU benchmark—a 7% improvement over Gopher. - **[2022]** [PaLM](https://arxiv.org/abs/2204.02311) Pathways Language Model (PaLM) was introduced by Google Research in 2022, showcasing a leap in model scale with a whopping 540 billion parameters. Leveraging the proprietary Pathways system for distributed computation, PaLM exhibited great few-shot performance across an array of language understanding, reasoning, and code-related tasks. - **[2022]** [ChatGPT](https://openai.com/blog/chatgpt) In November 2022, OpenAI released ChatGPT, a conversation model based on GPT-3.5 and GPT-4. Specially optimized for dialogue, ChatGPT exhibited great abilities in communicating with humans, reasoning, and aligning with human values. - **[2023]** [LLaMA](https://arxiv.org/abs/2302.13971) LLaMA (Large Language Model Meta AI) emerged in February 2023 from Meta AI. It introduced a family of large language models available in varying sizes from 7 billion to 65 billion parameters. LLaMA's release marked a departure from the limited access trend, as its model weights were made available to the research community under a noncommercial license. Subsequent developments, including [Llama 2](https://arxiv.org/abs/2307.09288) and other chat models, further emphasized accessibility, this time with a license for


A Timeline of Large Language Models

# A Timeline of Large Language Models
## A Timeline of the Most Popular LLMs
commercial use. - **[2023]** [GPT-4](https://arxiv.org/abs/2303.08774) In March 2023, GPT-4 was released, extending text input to multimodal signals. With stronger capacities than GPT-3.5, GPT-4 demonstrated significant performance improvements on various tasks. If you want to dive deeper into these models, I suggest reading the paper “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf).” Here’s a table summarizing the architectural and training details of all the mentioned models (and others). ![From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper](A%20Timeline%20of%20Large%20Language%20Models%20ceae1b1e9692468ca3cfb2d5925164d3/Screenshot_2023-08-09_at_12.25.37.png) From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper Moreover, here’s an image showing the evolution of the LLaMA models into other fine-tuned models made by online communities, highlighting the great interest it sparked. ![From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper](A%20Timeline%20of%20Large%20Language%20Models%20ceae1b1e9692468ca3cfb2d5925164d3/Screenshot_2023-08-09_at_12.25.47.png) From “[A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)” paper ---


A Timeline of Large Language Models

# A Timeline of Large Language Models
## Conclusion
In this lesson, we learned more about the transition from pre-trained language models (LMs) to the emergence of large language models (LLMs). We explored the key differentiating features of LLMs, including the influence of scaling laws and the manifestation of emergent abilities like in-context learning, step-by-step reasoning strategies, and instruction following. We also saw a brief timeline of the most popular LLMs: from the foundational GPT-1 to the revolutionary GPT-3, the specialized Codex, LaMDA, Gopher, and Chinchilla, to PaLM, ChatGPT, and LLaMA.


Techniques for Fine-Tuning LLMs

# Techniques for Fine-Tuning LLMs
## Introduction
In this lesson, we will examine the main techniques for fine-tuning Large Language Models for superior performance on specific tasks. We explore why and how to fine-tune LLMs, the strategic importance of instruction fine-tuning, and several fine-tuning methods, such as Full Finetuning, Low-Rank Adaptation (LoRA), Supervised Finetuning (SFT), and Reinforcement Learning from Human Feedback (RLHF). We also touch upon the benefits of the Parameter-Efficient Fine-tuning (PEFT) approach using Hugging Face's PEFT library, promising both efficiency and performance gains in fine-tuning.


Techniques for Fine-Tuning LLMs

# Techniques for Fine-Tuning LLMs
## Why We Finetune LLMs
While pretraining provides Language Models (LLMs) with a broad understanding of language, it doesn't equip them with the specialized knowledge needed for complex tasks. For instance, a pre-trained LLM may excel at generating text but encounter difficulties when tasked with sentiment analysis of financial news. This is where fine-tuning comes into play. Fine-tuning is the process of adapting a pretrained model to a specific task by further training it using task-specific data. For example, if we aim to make an LLM proficient in answering questions about medical texts, we would fine-tune it using a dataset comprising medical question-answer pairs. This process enables the model to recalibrate its internal parameters and representations to align with the intended task, enhancing its capacity to address domain-specific challenges effectively. However, fine-tuning LLMs conventionally can be resource-intensive and costly. It involves adjusting all the parameters in the pretrained LLM models, which can number in the billions, necessitating significant computational power and time. Consequently, it's crucial to explore more efficient and cost-effective methods for fine-tuning, such as Low-Rank Adaptation (LoRA).


Techniques for Fine-Tuning LLMs

# Techniques for Fine-Tuning LLMs
## A Reminder On Instruction Finetuning
Instruction fine-tuning is a specific type of fine-tuning that grants precise control over a model's behavior. The objective is to train a Language Model (LLM) to interpret prompts as instructions rather than simply treating them as text to continue generating. For example, when given the instruction, "Analyze the sentiment of this text and tell us if it's positive," a model with instruction fine-tuning would perform sentiment analysis rather than continuing the text in some manner. This technique offers several advantages. It involves training models on tasks described using instructions, enabling LLMs to generalize to new tasks based on additional instructions. This approach circumvents the need for extensive amounts of task-specific data and relies on textual instructions to guide the learning process.


Techniques for Fine-Tuning LLMs

# Techniques for Fine-Tuning LLMs
## A Reminder of the Techniques For Finetuning LLMs
There are several techniques to make the finetuning process more efficient and effective: - **Full Finetuning:** This method involves adjusting all the parameters in the pretrained LLM models to adapt to a specific task. While effective, it is resource-intensive and requires extensive computational power, therefore it’s rarely used. - **Low-Rank Adaptation (LoRA):** LoRA is a technique that aims to adapt LLMs to specific tasks and datasets while simultaneously reducing computational resources and costs. By applying low-rank approximations to the downstream layers of LLMs, LoRA significantly reduces the number of parameters to be trained, thereby lowering the GPU memory requirements and training costs. We’ll also see QLoRA, a variant of LoRA that is more optimized and leverages quantization. With a focus on the number of parameters involved in finetuning, there are multiple methods, such as: - **Supervised Finetuning (SFT):** SFT involves doing standard supervised finetuning with a pretrained LLM on a small amount of demonstration data. This method is less resource-intensive than full finetuning but still requires significant computational power. - **Reinforcement Learning from Human Feedback (RLHF):** RLHF is a training methodology where models are trained to follow human feedback over multiple iterations. This method can be more effective than SFT, as it allows for continuous improvement based on human feedback. We’ll also see some alternatives to RLHF, such as Direct Preference Optimization (DPO), and Reinforcement Learning from AI Feedback (RLAIF).


Techniques for Fine-Tuning LLMs

# Techniques for Fine-Tuning LLMs
## Efficient Finetuning with Hugging Face PEFT Library
Parameter-Efficient Fine-tuning (PEFT) approaches address the need for computational and storage efficiency in fine-tuning LLMs. Hugging Face developed the [PEFT library](https://github.com/huggingface/peft) specifically for this purpose. PEFT leverages architectures that only fine-tune a small number of additional model parameters while freezing most parameters of the pretrained LLMs, significantly reducing computational and storage costs. PEFT methods offer benefits beyond just efficiency. These methods have been proven to outperform standard fine-tuning methods, particularly in low-data situations, and provide improved generalization for out-of-domain scenarios. Furthermore, they contribute to the portability of models by generating tiny model checkpoints that require substantially less storage space compared to extensive full fine-tuning checkpoints. By integrating PEFT strategies, we make way for comparable performance gains to full fine-tuning with only a fraction of the trainable parameters. This, in effect, broadens our capacity to harness the prowess of LLMs, regardless of the hardware limitations we might encounter. Providing easy integration with the Hugging Face's [Transformers](https://github.com/huggingface/transformers) and [Accelerate](https://github.com/huggingface/accelerate) libraries, the PEFT library supports popular methods such as Low-Rank Adaptation (LoRA) and Prompt Tuning.


Techniques for Fine-Tuning LLMs

# Techniques for Fine-Tuning LLMs
## Conclusion
In this lesson, we've learned that while pretraining equips LLMs with a broad understanding of language, fine-tuning is necessary to specialize these models for complex tasks. We've introduced various fine-tuning techniques, including Full Finetuning, Low-Rank Adaptation (LoRA), Supervised Finetuning (SFT), and Reinforcement Learning from Human Feedback (RLHF). We've also highlighted the importance of instruction fine-tuning for precise control over model behavior. Finally, we've examined the benefits of Parameter-Efficient Fine-tuning (PEFT) approaches, mainly using Hugging Face's PEFT library, which promises both efficiency and performance gains in fine-tuning. This equips us to harness the power of LLMs more effectively and efficiently, regardless of hardware limitations, and to adapt these models to a wide range of tasks and domains.


# Advanced Topics and Future Directions Module
## Advanced Topics and Future Directions
**Goals:** Equip students with an understanding of the latest trends in LLM. This section examines the nuances of multimodal LLMs, scaling laws in training, sophisticated prompting frameworks like ReAct, and the complexities of training on generated data. The participants will unravel the ways they redefine interactions with language models. FlashAttention, Sparse Attention, and ALiBi's roles in expanding the context window offer insights into LLM optimization strategies. The module also confronts the challenges of training on generated data, spotlighting the critical phenomenon of Model Collapse. - **Multimodal LLMs:** Discover the significance of multimodality in LLMs, integration of varied data like text and images. The lesson highlights popular models adept at multimodality and provides an overview of their functionalities. - **Scaling Laws in LLM Training**: To achieve compute-optimal training results, it's essential to maintain a harmonious balance between the size of the LLM and the number of training tokens. This session offers guidance on how to effectively scale both elements in tandem. - **ReAct framework and ChatGPT plugins**: The ReAct framework offers a sophisticated approach to enhancing interactions with language models, which also abilitates ChatGPT plugins. - **Expanding the context window**: Detailed workings of FlashAttention and Sparse Attention in the transformer structures, aiming for efficient computation. Emphasis is also given to ALiBi's significance in this context. - **Training on generated data: Model Collapse:** The problem of training on generated data, particularly emphasizing the phenomenon of Model Collapse. Understanding this topic is crucial, as training on bad data influences the effectiveness and reliability of LLMs. - **New Challenges in LLM Research**: The Emerging challenges in LLM research, are cover agents, retriever architectures, larger context windows, efficient attention, and cost-effective pre-training and fine-tuning. Insights from recent studies guide the exploration, preparing students for the evolving LLM landscape. As we wrap up this comprehensive course on Training and Fine-Tuning Large Language Models, students have covered a vast spectrum of topics, from the basic architecture of LLMs to deployment, from specialized Fine-Tuning methods to anticipated challenges in LLM research. With this knowledge, students are well-prepared to apply LLMs in practical scenarios, evaluate their performance effectively, innovate with fine-tuning strategies, and stay abreast of emerging trends and challenges in the domain of artificial intelligence and machine learning.


Improving LLMs with RLHF Module

# Improving LLMs with RLHF Module
## Improving LLMs with RLHF
Goals: Equip students with knowledge and practical skills for implementing RLHF. This short module is focussed on Reinforcement Learning from Human Feedback (RLHF). We learn how to incorporate human feedback into the training process through a reward model that learns the desired patterns to amplify the model’s output. - **Deep Dive into RLHF**: This lesson explores the mechanics and applications of RLHF. We will provide a robust understanding of how RLHF functions and its significance in LLM training and optimization. - **Improving trained models with RLHF**: This lesson provides a practical guide on RLHF as a fine-tuning technique for LLMs. We build upon our previous fine tuning example by implementing RLHF.


Prompting and Few-Shot Prompting

# Prompting and Few-Shot Prompting
## **Introduction**
In this lesson, we will explore prompting and prompt engineering, which allow us to interact with LLMs effectively for various applications. We can leverage LLMs to perform tasks such as answering questions, text generation, and more by crafting specific prompts. We will delve into zero-shot prompting, where the model produces results without explicit examples, and then transition to in-context learning and few-shot prompting, where the model learns from demonstrations to handle complex tasks with minimal training data.


Prompting and Few-Shot Prompting

# Prompting and Few-Shot Prompting
## Prompting and Prompt Engineering
Prompting is a very important technique that involves designing and optimizing prompts to interact effectively with LLMs for various applications. The process of **prompt engineering** enables developers and researchers to harness the capabilities of LLMs and utilize them for tasks such as answering questions, arithmetic reasoning, text generation, and more. At its core, prompting involves presenting a specific task or instruction to the language model, which then generates a response based on the information provided in the prompt. A prompt can be as simple as a question or instruction or include additional context, examples, or inputs to guide the model towards producing desired outputs. The quality of the results largely depends on the precision and relevance of the information provided in the prompt. Let's incorporate these ideas in the code examples. Before running them, remember to load your environment variables from your `.env` file as follows. ```python from dotenv import load_dotenv load_dotenv() ``` ### Example: Story Generation In this example, the prompt sets up the start of a story, providing initial context ("a world where animals could speak") and a character ("a courageous mouse named Benjamin"). The model's task is to generate the rest of the story based on this prompt. Note that in this example we are defining separately a `prompt_system` and a `prompt`. This is because the OpenAI API works this way, requiring a “system prompt” to steer the model behaviour. This is different from other LLMs that require only a standard prompt. ```python import openai prompt_system = "You are a helpful assistant whose goal is to help write stories." prompt = """Continue the following story. Write no more than 50 words. Once upon a time, in a world where animals could speak, a courageous mouse named Benjamin decided to""" response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": prompt_system}, {"role": "user", "content": prompt} ] ) print(response.choices[0]['message']['content']) ``` ``` embark on a journey to find the legendary Golden Cheese. With determination in his heart, he ventured through thick forests and perilous mountains, facing countless obstacles. Little did he know that his bravery would lead him to the greatest adventure of his life. ``` ### Example: **Product Description** Here, the prompt is a request for a product description with key details ("luxurious, hand-crafted, limited-edition fountain pen made from rosewood and gold"). The model is tasked with writing an appealing product description based on these details. ```python import openai prompt_system = "You are a helpful assistant whose goal is to help write product descriptions." prompt = """Write a captivating product description for a luxurious, hand-crafted, limited-edition fountain pen made from rosewood and gold. Write no more than 50 words.""" response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": prompt_system}, {"role": "user", "content": prompt} ] ) print(response.choices[0]['message']['content']) ``` ``` Experience the epitome of elegance with our luxurious limited-edition fountain pen. Meticulously handcrafted from exquisite rosewood and shimmering gold, this writing instrument exudes sophistication in every stroke. Elevate your writing experience to new heights with this opulent masterpiece. ```


Prompting and Few-Shot Prompting

# Prompting and Few-Shot Prompting
## Zero-Shot Prompting
In the context of prompting, “**zero-shot prompting**” is where we directly ask for the result without providing reference examples for the task. For many tasks, LLMs are smart enough to produce great results. This is exactly what we did in the examples above. Here’s a new example where we ask an LLM to write a short poem about summer. ```python import openai prompt_system = "You are a helpful assistant whose goal is to write short poems." prompt = """Write a short poem about {topic}.""" response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": prompt_system}, {"role": "user", "content": prompt.format(topic="summer")} ] ) print(response.choices[0]['message']['content']) ``` ``` In the realm of golden rays, Summer dances in perfect sway, Nature's canvas aglow with hues, Kissing warmth upon the dews. Breezes whisper through the trees, Serenading the humming bees, Joyful laughter fills the air, As sunshine gleams without a care. Sand between our toes, so fine, Waves crashing in rhythmic rhyme, Picnics filled with sweet delight, Summer's pleasures, pure and bright. Days stretch long, nights invite, Stargazing dreams take flight, Fireflies dance in twilight's haze, Summer's magic shall never fade. ``` The generated poem is nice, but what if we have a specific style of the poem we’d like it to generate? We could try the descriptive approach or simply provide relevant examples of what we need in the prompt.


Prompting and Few-Shot Prompting

# Prompting and Few-Shot Prompting
## In-Context Learning And Few-Shot Prompting
In the context of LLMs, **in-context learning** is a powerful approach where the model learns from demonstrations or exemplars provided within the prompt. **Few-shot prompting** is a technique under in-context learning that involves giving the language model a few examples or demonstrations of the task at hand to help it generalize and perform better on complex tasks. Few-shot prompting allows language models to learn from a limited amount of data, making them more adaptable and capable of handling tasks with minimal training samples. Instead of relying solely on zero-shot capabilities (where the model predicts outputs for tasks it has never seen before), few-shot prompting leverages the in-context demonstrations to improve performance. In few-shot prompting, the prompt typically includes multiple questions or inputs along with their corresponding answers. The language model learns from these examples and generalizes to respond to similar queries. ```python import openai prompt_system = "You are a helpful assistant whose goal is to write short poems." prompt = """Write a short poem about {topic}.""" examples = { "nature": "Birdsong fills the air,\nMountains high and valleys deep,\nNature's music sweet.", "winter": "Snow blankets the ground,\nSilence is the only sound,\nWinter's beauty found." } response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": prompt_system}, {"role": "user", "content": prompt.format(topic="nature")}, {"role": "assistant", "content": examples["nature"]}, {"role": "user", "content": prompt.format(topic="winter")}, {"role": "assistant", "content": examples["winter"]}, {"role": "user", "content": prompt.format(topic="summer")} ] ) print(response.choices[0]['message']['content']) ``` ``` Golden sunbeams shine, Warm sands between toes divine, Summer memories, mine. ``` ### **Limitations of Few-shot Prompting** Despite its effectiveness, few-shot prompting does have limitations, especially for more complex reasoning tasks. In such cases, advanced techniques like chain-of-thought prompting have gained popularity. Chain-of-thought prompting breaks down complex problems into multiple steps and provides demonstrations for each step, enabling the model to reason more effectively.


Prompting and Few-Shot Prompting

# Prompting and Few-Shot Prompting
## Conclusion
In this lesson, we explored prompting in the context of language models. Prompting involves presenting specific tasks or instructions to an LLM to generate desired responses. We learned that the quality of results largely depends on the precision and relevance of the information provided in the prompt. Through code examples, we saw how to use prompts for story generation and product descriptions. We also explored zero-shot prompting, where the model can perform tasks without explicit reference examples. However, we introduced few-shot prompting, a powerful in-context learning approach to improve the model's performance on more complex tasks. Few-shot prompting allows the model to learn from a limited number of examples, making it more adaptable and capable of handling tasks with minimal training data. However, we also recognized that few-shot prompting has its limitations, particularly for complex reasoning tasks. In such cases, advanced techniques like chain-of-thought prompting are gaining popularity by breaking down complex problems into multiple steps with demonstrations for each step.


Domain-Specific LLMs

# Domain-Specific LLMs
## Introduction
Domain-specific Language Models are tailored for specific industries or use cases. Unlike generalized language models attempting to comprehend a wide array of topics, domain-specific LLMs are finely tuned to understand a particular domain's unique terminology, context, and intricacies. In this lesson, we’ll see what it takes to create a domain-specific LLM, how to do it, and what popular domain-specific LLMs are.


Domain-Specific LLMs

# Domain-Specific LLMs
## When Do Domain-Specific LLMs Make Sense?
Domain-specific LLMs offer distinct advantages over their generalized counterparts in scenarios where precision, accuracy, and context are very important. They excel in industries where specialized knowledge is essential for generating relevant and accurate outputs. Moreover, they are also good in certain scenarios for safety (constraining what the model knows about), and smaller model have lower latency and are cheaper to host/infer with respect to an LLM, especially if for a single task. So, what are some specific industries where domain-specific LLMs could work well? Two notable examples are: 1. **Finance**: Domain-specific LLMs can provide personalized investment recommendations based on an individual's financial goals, optimizing investment strategies. 2. **Healthcare**: A domain-specific LLM trained in medical data can comprehend complex medical queries and offer accurate advice, enhancing patient care and medical consultation. Why don’t we use general-purpose LLMs in these fields, too? General-purpose LLMs gather their knowledge from their pre-training phase and are “steered” into valuable assistants in the finetuning phase. GPT-4 may know the correct answer to a medical question since it may be trained on medical papers too, but it may not be appropriately steered into a good “medical assistant” that would also ask meaningful questions. However, we’re still in the infancy of LLM research, so it’s still hard to correctly reason how they work. As long as the required knowledge was in the pre-training data, in principle, the LLM is likely able to behave in the “correct” way if appropriately finetuned. If we think the required knowledge wasn’t in the training data, we’d have to pre-train a new LLM from scratch. If we think otherwise, then we could focus on finetuning. In finance, Bloomberg recently trained a proprietary LLM from scratch using a mix of general-purpose and financial data called BloombergGPT.


Domain-Specific LLMs

# Domain-Specific LLMs
## **BloombergGPT**
[BloombergGPT](https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/) is a proprietary domain-specific 50B LLM trained for the financial domain. Its **training dataset** is called “FinPile” and is made of many English financial documents derived from diverse sources, encompassing financial news, corporate filings, press releases, and even social media taken from Bloomberg archives (thus, it’s a proprietary dataset). Data ranges from company filings to market-relevant news from March 2007 to July 2022. The dataset is further augmented by the integration of publicly available general-purpose text, creating a balance between domain specificity and the broader linguistic landscape. In the end, the final domain is approximately half domain-specific (51.27%) and half general-purpose (48.73%). The model is **based on the BLOOM model.** It’s a decoder-only transformer with 70 layers of decoder blocks, multi-head self-attention, layer normalization, and feed-forward networks equipped with the GELU non-linear function. The model is based on the Chinchilla scaling laws. BloombergGPT outperforms other models like GPT-NeoX, OPT-66B, and BLOOM-176B on financial tasks. However, when confronted with GPT-3 on general-purpose tasks, GPT-3 achieves better results.


Domain-Specific LLMs

# Domain-Specific LLMs
## **The FinGPT Project**
The [FinGPT](https://github.com/AI4Finance-Foundation/FinGPT) project aims to bring the power of LLMs into the world of finance. It aims to do so in two ways: 1. Providing open finance datasets. 2. Finetuning open-source LLMs on finance datasets for several use cases. Many datasets collected by FinGPT are specifically for financial sentiment analysis. What do we mean by “financial sentiment”? For example, the sentence “Operating profit rose to EUR 13.1 mn from EUR 8.7 mn in the corresponding period in 2007 representing 7.7 % of net sales” merely states facts; therefore, its “normal” sentiment would be neutral. However, it states that the company's operating profit rose, which is good news for someone who wants to invest in that company, and therefore the financial sentiment is “positive.” Similarly, the sentence “The international electronic industry company Elcoteq has laid off tens of employees from its Tallinn facility” has “negative” financial sentiment. Some datasets for financial sentiment classification are: - [Financial Phrasebank](https://huggingface.co/datasets/financial_phrasebank): It contains 4840 sentences from English financial news, categorized by financial sentiment (written by agreement between 5-8 annotators). - [Financial Opinion Mining and Question Answering (FIQA)](https://huggingface.co/datasets/pauri32/fiqa-2018)**:** Consists of 17k sentences from microblog headlines and financial news, classified with financial sentiment. - [Twitter Financial Dataset (sentiment)](https://huggingface.co/datasets/zeroshot/twitter-financial-news-sentiment): about 10k tweets with financial sentiment. Here, you can find a [notebook](https://github.com/AI4Finance-Foundation/FinGPT/tree/master/fingpt/FinGPT-v3) showing how to finetune a model with these datasets and how to use the final model for predictions.


Domain-Specific LLMs

# Domain-Specific LLMs
## Med-PaLM for the Medical Domain
[Med-PaLM](https://sites.research.google/med-palm/) is a finetuned version of PaLM (by Google) specifically for the medical domain. The first iteration of Med-PaLM, introduced in late 2022 and subsequently published in Nature in July 2023, marked a milestone by surpassing the pass mark on US Medical License Exam (USMLE) style questions. Building upon this success, Google Health unveiled the latest iteration, [Med-PaLM 2](https://blog.google/technology/health/ai-llm-medpalm-research-thecheckup/), during its annual health event, The Check Up, in March 2023. Med-PaLM 2 represents a substantial leap forward, achieving an accuracy rate of 86.5% on USMLE-style questions.


Domain-Specific LLMs

# Domain-Specific LLMs
## Conclusion
Domain-specific LLMs are specialized tools finely tuned for domain expertise. They are indicated for specific fields like finance and healthcare, where nuanced understanding is very important. Examples include BloombergGPT for finance, FinGPT for financial sentiment analysis, and Med-PaLM for medical inquiries.


Datasets for Training LLMs

# Training LLMs Module
## Training LLMs
Goals: Provide a hands-on coding experience for training an LLM from scratch in the cloud. Introduce domain-specific LLMs and guide students on benchmarking their custom LLMs. As we transition into the practical aspects, the focus shifts to training LLMs in the cloud and the significance of efficient scaling techniques. With a focus on benchmarking LLMs and the strategic application of domain-specific models in various sectors, it provides a meticulous understanding of tools like Deep Lake, their role in refining LLM training, and the importance of dataset curation. - **When to Train an LLM from Scratch**: This lesson will help with the decision of when to train an LLM from scratch versus utilizing a pre-trained model and the tradeoffs of using both proprietary and open-source models. Specific references to models like BloombergGPT, FinGPT, and LawGPT will be highlighted, offering real-world context. We also address the ongoing debate about the advantages and challenges of training domain-specific LLMs. - **LLMOps**: This lesson touches upon LLMOps, a specialized practice catering to the operational needs of Large Language Models. It is imperative to have dedicated operations for LLMs to streamline deployment, maintenance, and scaling. We will underscore the significance of tools like Weights & Biases in managing and optimizing LLMs, emphasizing their role in modern LLMOps practices. - **Overview of the Training Process**: This lesson provides sequential steps essential to an LLM training process. We will gather and refine data, then move to model initialization and set training parameters, often using the Trainer class. The lesson continues with monitoring via an evaluation dataset. - **Deep Lake and Data Loaders**: This lesson covers Deep Lake and its affiliated data loaders and their role in the LLM training and finetuning process. We will discuss the utility of these tools and gain insights into how they streamline data handling and model optimization. - **Datasets for Training LLMs**: This lesson dives into diverse datasets for LLM training in text and coding. Students learn the intricacies of curating specialized datasets, with an example of storing data in Deep Lake. We emphasize on data quality, referencing the "Textbooks Are All You Need" research. - **Train an LLM in the Cloud**: The lesson lays out the process of training LLMs in the cloud using the Hugging Face Accelerate library and Lambda. We offer practical insights into integrating these platforms with hands-on guidance on leveraging data from a Deep Lake dataset. - **Tips for Training LLMs at Scale:** This lesson offers students valuable strategies for efficiently scaling LLM training. It emphasizes advanced techniques and optimizations. - **Benchmarking your own LLM**: The lesson centers around the importance of benchmarking LLMs. We will discuss tools such as InstructEval and Eleuther’s Language Model Evaluation Harness. We also provide hands-on experience in assessing their LLM's performance against recognized benchmarks. - **Domain-specific LLMs**: The lesson covers the strategic use of domain-specific LLMs. We explore scenarios where these specialized models are most effective, with a spotlight on popular instances like FinGPT and BloombergGPT. By analyzing these cases, we will understand the nuances of utilizing domain-specific LLMs to cater to unique industry demands. Upon completing this comprehensive module, participants have gained insights into the multifaceted landscape of training LLMs. From understanding the tradeoffs between training from scratch versus leveraging pre-existing models to LLMOps, students are well-equipped to benchmark their LLMs effectively and understand the nuanced value of domain-specific models in meeting specific industry requirements. The following section will introduce learners to the complexities that come with finetuning techniques and practical hands-on projects.


Expanding the Context Window

# Expanding the Context Window
## Introduction
In this lesson, we will discuss context windows in language models, their importance, and the limitations of the original Transformer architecture in handling large context lengths. We explore various optimization techniques that have been developed to expand the context window, including ALiBi Positional Encoding, Sparse Attention, FlashAttention, Multi-Query Attention, and the use of large RAM GPUs. We also introduce the latest advancements in this field, such as FlashAttention-2 and LongNet, which aim to push the context window to an unprecedented scale.


Expanding the Context Window

# Expanding the Context Window
## The Importance of The Context Length
The context window refers to the number of input tokens the model can process simultaneously. In current models like [GPT-4](https://en.wikipedia.org/wiki/GPT-4), this context window is around 32K tokens. To put this into perspective, this roughly translates to the size of 50 pages. However, recent advancements have pushed this limit to an impressive 100K tokens (check [Claude by Anthropic](https://www.anthropic.com/index/100k-context-windows)), equivalent to 156 pages. The context length of an LLM is a critical factor for several reasons. Firstly, it allows the model to process larger amounts of data at once, providing a more comprehensive understanding of the context. This is particularly useful when you want to feed a large amount of custom data into an LLM and ask questions about this specific data. For instance, you might want to input a large document related to a specific company or problem and ask the model questions about this document. With a larger context window, the LLM can scan and retain more of this custom information, leading to more accurate and personalized responses.


Expanding the Context Window

# Expanding the Context Window
## Limitations of the Original Transformer Architecture
The original Transformer architecture, however, has some limitations when it comes to handling large context lengths. The main issue lies in the computational complexity of the Transformer architecture. Specifically, the attention layer computations in the Transformer architecture have a quadratic time and space complexity with respect to the number of input tokens $n$. This means that as the context length increases, the computational resources required for training and inference increase exponentially. To understand this better, let's understand the computational complexity of the Transformer architecture. The complexity of the attention layer in the Transformer model is $O(n²d + nd²)$, where $n$ is the context length (number of input tokens) and $d$ is the embedding size. This complexity arises from two main operations in the attention layer: linear projections to get Query, Key, and Value matrices (complexity ~ $O(nd²)$) and multiplications of these matrices (complexity ~ $O(n²d)$). As the context length or embedding size increases, the computational complexity grows quadratically, making it increasingly challenging to process larger context lengths.