Expanding the Context Window

# Expanding the Context Window
## Optimization Techniques to Expand the Context Window
Despite these challenges, researchers have developed several optimization techniques to speed up the Transformer and increase the context length to 100K tokens. Let's explore some of these techniques: 1. **[ALiBi Positional Encoding](https://arxiv.org/abs/2108.12409)**: The original Transformer uses Positional Sinusoidal Encoding, which lacks the ability to extrapolate to larger context lengths. ALiBi, or Attention with Linear Biases, is a positional encoding technique that can be used to train the model on a small context and then fine-tune it on a larger one. 2. **[Sparse Attention](https://ai.googleblog.com/2021/03/constructing-transformers-for-longer.html)**: This technique reduces the number of computations by considering only some tokens when calculating the attention scores. This makes the computation linear with respect to n, significantly reducing the computational complexity. 3. **[FlashAttention](https://arxiv.org/abs/2205.14135)**: This is an efficient implementation of the attention layer for GPU. It optimizes the memory utilization of the GPU by splitting the input matrices into blocks and computing the attention output with respect to these blocks. 4. **[Multi-Query Attention (MQA)](https://arxiv.org/pdf/1911.02150.pdf)**: MQA optimizes the memory consumption of the key/value decoder cache by sharing weights across all attention heads when linearly projecting Key and Value matrices. 5. **Large RAM GPUs**: You need a lot of RAM in the GPU to fit a large context. Therefore, models with larger context windows are often trained on GPUs with large RAM, such as 80GB A100 GPUs.


Expanding the Context Window

# Expanding the Context Window
## FlashAttention-2
Building on the success of FlashAttention, researchers have recently developed [FlashAttention-2,](https://crfm.stanford.edu/2023/07/17/flash2.html?utm_content=bufferca8a7&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer) a more efficient version of the algorithm that further optimizes the attention layer's speed and memory usage. This new version has been completely rewritten from scratch, leveraging the new primitives from Nvidia. The result is a version that is about 2x faster than its predecessor, reaching up to 230 TFLOPs/s on A100 GPUs. FlashAttention-2 introduces several improvements over the original FlashAttention. - Firstly, it reduces the number of non-matmul FLOPs, which are 16x more expensive than matmul FLOPs, by tweaking the algorithm to spend more time on matmul FLOPs. - Secondly, it optimizes parallelism by parallelizing over batch size, number of heads, and the sequence length dimension. This results in significant speedup, especially for long sequences. - Lastly, it improves work partitioning within each thread block to reduce the amount of synchronization and communication between different warps, resulting in fewer shared memory reads/writes. - In addition to these improvements, FlashAttention-2 also introduces new features, such as support for head dimensions up to 256 and multi-query attention (MQA), further expanding the context window. With these advancements, FlashAttention-2 is a step forward in expanding the context window (without overcoming the fundamental limitations of the original Transformer architecture).


Expanding the Context Window

# Expanding the Context Window
## LongNet: A Leap Towards Billion-Token Context Window
Building on the advancements in Transformer optimization, a recent innovation comes from the paper "[LONGNET: Scaling Transformers to 1,000,000,000 Tokens](https://arxiv.org/pdf/2307.02486.pdf)". This paper introduces a novel approach to handling the computational complexity of the Transformer architecture, pushing the context window potentially to an unprecedented 1 billion tokens. The core innovation in LongNet is the introduction of "dilated attention.” This novel attention mechanism expands the attentive field exponentially as the distance between tokens grows, thereby decreasing attention allocation exponentially as the distance increases. This design principle helps to balance the limited attention resources with the necessity to access every token in the sequence. ![Image from the paper "[LONGNET: Scaling Transformers to 1,000,000,000 Tokens](https://arxiv.org/pdf/2307.02486.pdf)". Building blocks of dilated attention used in LONGNET. It consists of a series of attention patterns for modeling short- and long-range dependency. The number of attention patterns can be extended according to the sequence length.](Expanding%20the%20Context%20Window%208e1fbda8eb48406e8a358960164bf9df/Screenshot_2023-08-17_at_14.48.55.png) Image from the paper "[LONGNET: Scaling Transformers to 1,000,000,000 Tokens](https://arxiv.org/pdf/2307.02486.pdf)". Building blocks of dilated attention used in LONGNET. It consists of a series of attention patterns for modeling short- and long-range dependency. The number of attention patterns can be extended according to the sequence length. The dilated attention mechanism in LongNet achieves a linear computational complexity, a significant improvement over the quadratic complexity of the standard Transformer. ![Image from the paper "[LONGNET: Scaling Transformers to 1,000,000,000 Tokens](https://arxiv.org/pdf/2307.02486.pdf)". Comparison of computation complexity among different methods. N is the sequence length, and d is the hidden dimension.](Expanding%20the%20Context%20Window%208e1fbda8eb48406e8a358960164bf9df/Screenshot_2023-08-17_at_14.47.38.png) Image from the paper "[LONGNET: Scaling Transformers to 1,000,000,000 Tokens](https://arxiv.org/pdf/2307.02486.pdf)". Comparison of computation complexity among different methods. N is the sequence length, and d is the hidden dimension.


Expanding the Context Window

# Expanding the Context Window
## Conclusion
In this lesson, we examined the limitations of the original Transformer architecture in handling large context lengths, primarily due to its quadratic computational complexity. We then explored various optimization techniques developed to overcome these limitations, including ALiBi Positional Encoding, Sparse Attention, FlashAttention, Multi-Query Attention, and the use of large RAM GPUs. We also discussed the latest advancements in this field, such as FlashAttention-2, which further optimizes the speed and memory usage of the attention layer, and LongNet, a novel approach that introduces "dilated attention" to potentially expand the context window to an unprecedented 1 billion tokens. These advancements are critical in pushing the boundaries of language models, enabling them to process larger amounts of data at once and providing a more comprehensive understanding of the context, leading to more accurate and personalized responses.


Benchmarking Your Own LLM

# Benchmarking Your Own LLM
## Introduction
In a previous lesson, we touched upon several methodologies for assessing the effectiveness of different language models. Even with various available methodologies for evaluating a large language model, the process continues to pose significant challenges. The primary challenge stems from the inherent subjectivity in determining what constitutes a good answer. As an example, let's consider a generative task such as summarization. Quantifying a specific summary as the definitive correct answer in terms of context is difficult and hard to define. This challenge is prevalent across every generative task, making it a pervasive source of difficulty. In this lesson, we’ll see how benchmarks are a candidate solution to this problem.


Benchmarking Your Own LLM

# Benchmarking Your Own LLM
## Benchmarks Over Several Tasks
The solution suggests curating a set of benchmarks to evaluate the model's performance across various tasks. The benchmarks encompass assessments for world knowledge, following complex instructions, arithmetic, programming, and more. Several leaderboards exist to monitor the progress of LLMs, including the “*[Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)*” and the “*[InstructEval Leaderboard](https://declare-lab.net/instruct-eval/)*.” In some cases, these benchmarks share similar metrics. Here are some examples of tasks tested in these benchmarks: - **[AI2 Reasoning Challenge](https://allenai.org/data/arc)** (ARC): The dataset is exclusively comprised of natural, grade-school science questions designed for human tests. - **[HumanEval](https://paperswithcode.com/dataset/humaneval)**: It is used to measure program synthesis from docstrings. It includes 164 original programming problems assessing language comprehension, algorithms, and math, some resembling software interview questions. - **[HellaSwag](https://paperswithcode.com/dataset/hellaswag)**: A challenge to measure commonsense inference, and shows it remains difficult for state-of-the-art models. While humans achieve >95% accuracy on trivial questions, the models struggle with <48% accuracy. - ****[Measuring Massive Multitask Language Understanding](https://paperswithcode.com/dataset/mmlu) (**MMLU): An evaluation metric for text models' multitask accuracy, covering 57 tasks, including math, US history, computer science, law, etc. High accuracy requires extensive knowledge of the world and problem-solving ability. - **[TruthfulQA](https://paperswithcode.com/dataset/truthfulqa)**: A truthfulness benchmark designed to assess the accuracy of language models in generating answers to questions. It consists of 817 questions across 38 categories, encompassing topics such as health, law, finance, and politics.


Benchmarking Your Own LLM

# Benchmarking Your Own LLM
## **Language Model Evaluation Harness**
'wmt20-cs-en', 'wmt20-de-en', 'wmt20-de-fr', 'wmt20-en-cs', 'wmt20-en-de', 'wmt20-en-iu', 'wmt20-en-ja', 'wmt20-en-km', 'wmt20-en-pl', 'wmt20-en-ps', 'wmt20-en-ru', 'wmt20-en-ta', 'wmt20-en-zh', 'wmt20-fr-de', 'wmt20-iu-en', 'wmt20-ja-en', 'wmt20-km-en', 'wmt20-pl-en', 'wmt20-ps-en', 'wmt20-ru-en', 'wmt20-ta-en', 'wmt20-zh-en', 'wnli', 'wsc', 'wsc273', 'xcopa_et', 'xcopa_ht', 'xcopa_id', 'xcopa_it', 'xcopa_qu', 'xcopa_sw', 'xcopa_ta', 'xcopa_th', 'xcopa_tr', 'xcopa_vi', 'xcopa_zh', 'xnli_ar', 'xnli_bg', 'xnli_de', 'xnli_el', 'xnli_en', 'xnli_es', 'xnli_fr', 'xnli_hi', 'xnli_ru', 'xnli_sw', 'xnli_th', 'xnli_tr', 'xnli_ur', 'xnli_vi', 'xnli_zh', 'xstory_cloze_ar', 'xstory_cloze_en', 'xstory_cloze_es', 'xstory_cloze_eu', 'xstory_cloze_hi', 'xstory_cloze_id', 'xstory_cloze_my', 'xstory_cloze_ru', 'xstory_cloze_sw', 'xstory_cloze_te', 'xstory_cloze_zh', 'xwinograd_en', 'xwinograd_fr', 'xwinograd_jp', 'xwinograd_pt', 'xwinograd_ru', 'xwinograd_zh'] ``` To execute the evaluation, you must utilize the `main.py` file previously cloned from Github. Ensure that you are in the same directory as the specified file to execute the command successfully. By running the provided command, you will utilize the `facebook/opt-1.3b` model and evaluate its performance on the `hellaswag` dataset using GPU acceleration. ( Note that the displayed output is truncated. For the complete output, feel free to explore the attached notebook.) ```bash python main.py \ --model hf-causal \ --model_args pretrained=facebook/opt-1.3b \ --tasks hellaswag \ --device cuda:0 ``` ```python Running loglikelihood requests 100% 40145/40145 [29:44<00:00, 22.50it/s] { "results": { "hellaswag": { "acc": 0.4146584345747859, "acc_stderr": 0.00491656121359129, "acc_norm": 0.5368452499502091, "acc_norm_stderr": 0.004976214989483508 } }, "versions": { "hellaswag": 0 }, "config": { "model": "hf-causal", "model_args": "pretrained=facebook/opt-1.3b", "num_fewshot": 0, "batch_size": null, "batch_sizes": [], "device": "cuda:0", "no_cache": false, "limit": null, "bootstrap_iters": 100000, "description_dict": {} } } hf-causal (pretrained=facebook/opt-1.3b), limit: None, provide_description: False, num_fewshot: 0, batch_size: None | Task |Version| Metric |Value | |Stderr| |---------|------:|--------|-----:|---|-----:| |hellaswag| 0|acc |0.4147|± |0.0049| | | |acc_norm|0.5368|± |0.0050| ``` For a more in-depth exploration, the `--model` argument offers three options to choose from: `hf-causal` for specifying the language model, `hf-causal-experimental` for utilizing multiple GPUs, and `hf-seq2seq` for evaluating encoder-decoder models. Consequently, the `--model_args` parameter can be used to pass any additional arguments to the model. For instance, to employ a specific revision of the model with the float data type, utilize the following input: `--model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float`.” The arguments vary based on the chosen model and the inputs accepted by Huggingface, as this library leverages Huggingface to load open-source models. Otherwise, you can use the following to specify the engine type while evaluating OpenAI models: `--model_args engine=davinci`. Lastly, performing a combined evaluation using a combination of tasks is possible. To achieve this, simply pass a comma-separated string of available metrics like `--tasks hellaswag, and arc_challenge`, enabling the usage of both Hellaswag and ARC metrics simultaneously. To evaluate proprietary models from OpenAI, you need to set the `OPENAI_API_SECRET_KEY` environmental variable with the secret key. You can obtain this key from the OpenAI dashboard and use it accordingly. ```bash export OPENAI_API_SECRET_KEY=YOUR_KEY_HERE python main.py \ --model gpt3 \ --model_args engine=davinci \ --tasks hellaswag ```


Benchmarking Your Own LLM

# Benchmarking Your Own LLM
## **InstructEval**
There are other efforts to evaluate the language model, like the [InstructEval leaderboard](https://declare-lab.net/instruct-eval/), which is an effort that combines automated evaluation and using the GPT-4 model for scoring different models. Additionally, it is worth mentioning that they mainly focus on instruction-tuned models. ![Image from the [InstructEval paper](https://arxiv.org/abs/2306.04757).](Benchmarking%20Your%20Own%20LLM%2012b96e09de8d45ae83ae15c0cfea1404/Untitled.png) Image from the [InstructEval paper](https://arxiv.org/abs/2306.04757). The evaluation is broken down into three distinct tasks. **1. Problem-Solving Evaluation** It consists of the following test to evaluate the model’s ability on **World** **Knowledge** using [Massive Multitask Language Understanding](https://paperswithcode.com/dataset/mmlu) (MMLU), **Complex Instructions** using [BIG-Bench Hard](https://paperswithcode.com/dataset/big-bench) (BBH), **Comprehension and Arithmetic** using [Discrete Reasoning Over Paragraphs](https://paperswithcode.com/dataset/drop) (DROP), **Programming** using [HumanEval](https://paperswithcode.com/dataset/humaneval), and lastly **Causality** using [Counterfactual Reasoning Assessment](https://arxiv.org/abs/2112.11941) (CRASS). These automated evaluations assess the model's performance across various tasks. **2. Writing Evaluation** This category will evaluate the model based on the following subjective metrics: Informative, Professional, Argumentative, and Creative. They used the GPT-4 model to evaluate the output of different models by presenting a rubric and asking the model to score the outputs on the [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) between 1 and 5. **3. Alignment to Human Values** Finally, a crucial aspect of instruction-tuned models is their alignment with human values. We anticipate these models will uphold values such as helpfulness, honesty, and harmlessness. The leaderboard will evaluate the model by presenting pairs of dialogues and asking it to choose the appropriate one. We won't dive into an extensive explanation of the evaluation process as it closely resembles the previous benchmark, involving the execution of a Python script. Please follow the link to their [GitHub repository](https://github.com/declare-lab/instruct-eval), where they provide sample usage.


Benchmarking Your Own LLM

# Benchmarking Your Own LLM
## Conclusion
Having standardized metrics for evaluating different models is essential; otherwise, comparing the capabilities of various models would become impractical. In this lesson, we introduced several widely used metrics along with a script that facilitates the evaluation of LLMs. It is essential to emphasize the importance of keeping track of the latest leaderboard and evaluation metrics based on specific use cases. In most instances, having a model that excels in all tasks may not be necessary, so staying updated with relevant metrics helps identify the most suitable model for tailored requirements. --- >> [Notebook](https://colab.research.google.com/drive/1d4gJso06wgSq6Rj7JmPKnNjY8i1Bd78g?usp=sharing). ---


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Introduction
In this lesson, we will examine an emerging field of interest: The next progression in the evolution of LLMs, **Large multimodal models (LMMs)**. This topic adds a new layer to the material we've covered throughout the course. Simply put, multimodal models are designed to handle and interpret different data types - or **"modalities"** - such as text, images, audio, and video, all within a single coordinated system. This integration allows for a more comprehensive analysis and understanding than models processing only one data type, such as text in standard LLMs. For instance, supplementing a text prompt with voice or image inputs can enable these models to capture a more complex representation of the conveyed information. This is achieved by analyzing additional layers of data, such as the tone and cadence of your voice or the visual context provided by images, thus enhancing the depth and richness of the analysis. With the recent increase in the popularity of large language models, it is unsurprising that researchers are now exploring the potential of extending these models to handle multiple data types, aiming to create more versatile and valuable **general-purpose assistants**. Models that can solve arbitrary tasks specified by the user. In the following sections, we will explore the current implementations of LMMs and introduce key concepts on how they manage multimodality. We will also learn about their **emergent abilities** and explore the idea of **Instruction-tuned** LMMs. Finally, in this lesson, we will learn how **Deep Lake** by ActiveLoop can be helpful in training or fine-tuning large multimodal models.


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Common Architectures and Training Objectives
By definition, multimodal models are designed to process various input modalities, such as text, images, and videos, and generate outputs in multiple modalities. However, a notable subset of currently popular LMMs primarily focuses on accepting image inputs and is limited to generating text outputs. These specialized LMMs often leverage pre-trained large-scale vision or language models as their foundation. We can categorize them as 'Image-to-Text Generative Models,' also known as visual language models (VLMs). They generally perform tasks related to image understanding, such as question answering and image captioning. Examples include [GIT](https://arxiv.org/abs/2205.14100) by Microsoft, [BLIP2](https://arxiv.org/pdf/2301.12597.pdf) by SalesForce, and [Flamingo](https://arxiv.org/pdf/2204.14198.pdf) by DeepMind. ### **Model Architecture** These models use an **image encoder** to extract visual features and a standard LLM to output a text sequence. The image encoder can be based on convolutional neural networks (CNNs), such as [ResNet](https://arxiv.org/abs/1512.03385), or a transformer-based architecture like the [Vision Transformer (ViT)](https://arxiv.org/abs/2010.11929). The image encoder and the language model can be trained from scratch or using pre-trained models. Most state-of-the-art models opt for the latter approach; an example is the pre-trained image encoder from the model [CLIP](https://arxiv.org/pdf/2103.00020.pdf) by OpenAI. The options for language models are also extensive: one could choose from various open-source pre-trained models, such as Meta's [OPT](https://arxiv.org/pdf/2205.01068.pdf), [Llama 2](https://arxiv.org/abs/2307.09288), or Google's instruction-trained [FlanT5](https://arxiv.org/abs/1910.10683) models. Optionally, models like [BLIP2](https://arxiv.org/pdf/2301.12597.pdf) introduce a trainable lightweight connection module connecting the vision and language modalities. Since BLIP2 only trains this light module, it is cheaper and faster than other methods while still managing a strong zero-shot performance on image understanding tasks. ![From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled.png) From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper ### **Training Objective** Similar to what we've seen in the course, LMMs are trained using an **auto-regressive loss** function applied to the output text tokens. When using a [Vision Transformer](https://arxiv.org/abs/2010.11929) architecture, the concept of '**image tokens**,' which is analogous to text tokenization, is introduced. Just like text can be divided into smaller units like sentences, words, or sub-words for easier processing, images can be segmented into smaller, non-overlapping patches, known as 'tokens.' The exact attention mechanisms come into play in the Transformer architecture employed by these LMMs. Image tokens can 'attend' to each other, meaning they can influence each other's representation in the model. Meanwhile, the generation of each text token depends on all the image tokens and the previously generated text tokens. Check out our lesson about **[Understanding Transformers](https://www.notion.so/Understanding-Transformers-c4a6f85b4f0f4828ab6e2dc1c2a7b775?pvs=21)** if you are still getting familiar with these concepts. ![From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%201.png) From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Differences in Training Schemes
While having the same training objective, some variations emerge in the training schemes of different Language-Multimodal Models (LMMs). Most models, such as GIT and BLIP2, employ only image-text pairs for training. This approach allows them to establish connections between the text and image representations effectively but requires a large, curated image-text pairs dataset. ![From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%202.png) From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper On the other hand, the Flamingo model has some architectural innovations that allow for unlabeled web training data. They extract the text and images from the HTML of 43M webpages. They also determine the positions of images relative to the text based on the relative positions of the text and image elements in the Document Object Model (DOM). This model can ingest a **multimodal prompt** containing images and/or videos interleaved with text as input and generate text in an open-ended manner. It can produce text for tasks such as image captioning or visual question-answering. The system connects the different modalities and enables multimodal prompting through steps. Initially, a **Perceiver Resampler** module receives spatiotemporal features from visual data, such as an image or video, processed by the pre-trained Vision Encoder. The Perceiver then generates a fixed number of 'visual tokens.' These visual tokens serve as inputs to condition a frozen language model, a pre-trained language model that is not updated during this process. The conditioning is facilitated by adding newly initialized **cross-attention layers** interleaved with the language model's pre-existing layers. These new layers are not frozen and will be updated during training. While this architecture is less efficient by having more parameters to train than the one from BLIP2, it provides a powerful way for the language model to incorporate visual cues. ![From “[Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/pdf/2204.14198.pdf)” paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%203.png) From “[Flamingo: a Visual Language Model for Few-Shot Learning](https://arxiv.org/pdf/2204.14198.pdf)” paper ![From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%204.png) From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper ### Discovering Emergent Abilities - Few-shot In-Context-Learning Its flexible architecture allows Flamingo to be trained with multimodal prompts that interleave text with visual tokens. This enables the model to demonstrate emergent abilities, such as few-shot in-context learning, **analogous to GPT-3**. You can see some examples in the figure below. ![From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%205.png) From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Open-sourcing Flamingo
The state-of-the-art results reported in the Flamingo paper are exciting and clearly show significant progress in the field of LMMs. However, DeepMind has yet to make the Flamingo model publicly available. To fill this gap, HuggingFace's team took the initiative to create an open-source reproduction of Flamingo, known as **[IDEFICS](https://huggingface.co/blog/idefics)**. This replica is constructed entirely using publicly accessible resources, including the LLaMA v1 and OpenCLIP models. IDEFICS is offered in the 'base' and the 'instructed' variants. Both of these are available in two sizes—9 billion parameters and 80 billion parameters. IDEFICS offers comparable results to Flamingo. The team used a mixture of openly available datasets such as Wikipedia, Public Multimodal Dataset, and LAION to train these models. They also created a new 115B token dataset called **[OBELICS](https://huggingface.co/datasets/HuggingFaceM4/OBELICS)**. It has 141 million interleaved image-text documents scraped from the web and contains 353 million images, replicating the dataset described by DeepMind in the Flamingo paper. IDEFICS is available through the Transformers library, and a demo of it is available [here](https://huggingface.co/spaces/HuggingFaceM4/idefics_playground). Another open-source replication of Flamingo is called [Open Flamingo](https://github.com/mlfoundations/open_flamingo) at the 9B parameter size, offering similar performance to the original model. ![From “**[Introducing IDEFICS: An Open Reproduction of State-of-the-Art Visual Language Model](https://huggingface.co/blog/idefics)**” blog post.](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%206.png) From “**[Introducing IDEFICS: An Open Reproduction of State-of-the-Art Visual Language Model](https://huggingface.co/blog/idefics)**” blog post.


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Instruction-tuned LMMs
As demonstrated by GPT-3's emergent abilities with few-shot prompting, where the model could tackle tasks it hadn't seen during training, there's been a rising interest in instruction-fine-tuned LMMs. By allowing the models to be instruction-tuned, we can expect these models to perform a broader set of tasks and better alignment with human intents. This is line with the work done by OpenAI with [InstructGPT](https://openai.com/research/instruction-following) and, more recently, GPT-4. OpenAI has showcased the capability of their newer “GPT-4 with vision” model to follow instructions using visual inputs in their [GPT4 technical report](https://arxiv.org/pdf/2303.08774.pdf) and [GPT-4V(ision) System Card](https://cdn.openai.com/papers/GPTV_System_Card.pdf). ![From the “[GPT-4 Technical Report](https://arxiv.org/pdf/2303.08774.pdf)”](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%207.png) From the “[GPT-4 Technical Report](https://arxiv.org/pdf/2303.08774.pdf)” Following the announcement of OpenAI's **[multimodal GPT-4](https://openai.com/research/gpt-4)**, there has been a surge in related research. As a result, multiple research labs have introduced instruction-tuned LMMs, including **[LLaVA](https://arxiv.org/abs/2304.08485)**, **[MiniGPT-4](https://arxiv.org/abs/2304.10592)**, and **[InstructBlip](https://arxiv.org/abs/2305.06500)**. They feature similar network architectures as previous LMMs but train on instruction-following datasets. ![From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%208.png) From [“Multimodal Foundation Models: From Specialists to General-Purpose Assistants”](https://arxiv.org/pdf/2309.10020.pdf) paper ### Exploring LLaVA - an instruction-tuned LMM The network architecture of LLaVA resembles the one we reviewed before. This model connects a pre-trained CLIP visual encoder and the [Vicuna](https://lmsys.org/blog/2023-03-30-vicuna/) language model via a projection matrix. In other words, they consider a simple linear layer to connect image features into the word embedding space. Specifically, they apply a trainable projection matrix called W to convert the image features into language embedding tokens with the same dimensionality as the word embedding space in the language model. The authors of LLaVA chose these new linear projection layers that are more lightweight than the Q-Former connection module we saw for BLIP2 and the Perceiver Resampler and cross-attention layers from Flamingo. ![From “[Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf)” paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%209.png) From “[Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf)” paper The authors then adopt a two-stage instruction-tuning procedure to train the model. First, they pre-train the projection matrix using a subset of the [CC3M](https://aclanthology.org/P18-1238.pdf) dataset, consisting of images and captions. Then, the model is finetuned end-to-end. Both the projection matrix and the LLM are trained on the proposed multimodal instruction-following data for daily user-oriented applications. They also leverage GPT-4 to create a **synthetic dataset** consisting of multimodal instructions, drawing from widely available image-pair data. In the dataset construction process, GPT-4 is shown symbolic representations of images using **captions** and the **coordinates of bounding boxes**, as depicted in the figure below. These representations are derived from the COCO dataset. This information is fed into GPT-4 as a prompt to generate training samples. The generated samples fall into **three categories**: question-answer conversations, detailed descriptions, and complex reasoning questions and answers. They create 158K training samples in total. ![From “[Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf)” paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%2010.png) From “[Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf)” paper The LLaVA model demonstrates the effectiveness of visual instruction tuning using language-only GPT-4. They show its capabilities by prompting the model with the same question and image as in the GPT-4 report. You can see the result below. The authors also report a new SOTA by fine-tuning [ScienceQA](https://scienceqa.github.io/), a benchmark that contains 21k multimodal multiple-choice questions with rich domain diversity across three subjects, 26 topics, 127 categories, and 379 skills. ![From “[Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf)” paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%2011.png) From “[Visual Instruction Tuning](https://arxiv.org/pdf/2304.08485.pdf)” paper


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Beyond vision and language
In recent months, Image-to-text generative models have dominated the Large Multimodal Model (LMM) landscape. However, newer models have emerged that embrace a wider range of modalities beyond just vision and language. For instance, **[PandaGPT](https://panda-gpt.github.io/)** is designed to handle any input data type, thanks to its integration with the **[ImageBind](https://imagebind.metademolab.com/)** encoder. There's also **[SpeechGPT](https://github.com/0nutation/SpeechGPT#speechgpt-empowering-large-language-models-with-intrinsic-cross-modal-conversational-abilities)**, a model that integrates text and speech data and generates speech alongside text. **[NExT-GPT](https://arxiv.org/pdf/2309.05519.pdf)** stands out as a versatile model capable of receiving and producing outputs in any modality." ![From “[NExT-GPT: Any-to-Any Multimodal LLM](https://arxiv.org/pdf/2309.05519.pdf)” paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%2012.png) From “[NExT-GPT: Any-to-Any Multimodal LLM](https://arxiv.org/pdf/2309.05519.pdf)” paper [HuggingGPT](https://arxiv.org/pdf/2303.17580.pdf) is a novel system that integrates with the HuggingFace platform. It employs a Large Language Model (LLM) as its central controller. This LLM determines which specific model on HuggingFace is best suited for a task, selects that model, and then returns the model's output. ![From “[HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://arxiv.org/pdf/2303.17580.pdf)” paper](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Untitled%2013.png) From “[HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://arxiv.org/pdf/2303.17580.pdf)” paper


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Deep Lake and Multimodal LLMs
Deep Lake, differently from many other data lake and vector store products, is multi-modal and can store any data, from texts to images, videos, or audio. If you’re interested in building multi-modal LLMs, this is something to consider, as you can store the different types of data you need in the same place. See this [code example](https://docs.activeloop.ai/getting-started/deep-learning/creating-datasets-manually) to see how to store images and texts in the same dataset. Moreover, here’s the [full list of data types](https://docs.deeplake.ai/en/latest/Htypes.html) that can be managed with Deep Lake. ![List of data types managed by Deep Lake. It can be found at [https://docs.deeplake.ai/en/latest/Htypes.html](https://docs.deeplake.ai/en/latest/Htypes.html).](Introduction%20to%20Large%20Multimodal%20Models%2064ee4abd8d474902a26bedb90897cf53/Screenshot_2023-09-26_at_16.10.55.png) List of data types managed by Deep Lake. It can be found at [https://docs.deeplake.ai/en/latest/Htypes.html](https://docs.deeplake.ai/en/latest/Htypes.html).


Introduction to Large Multimodal Models

# Introduction to Large Multimodal Models
## Conclusion
In this module, we delved into the emergent field of LMMs. We examined the leading models that combine both vision and language modalities. We learned that instruction-tuning allows these models to achieve greater generalization on tasks they haven't encountered before. Furthermore, we were introduced to advanced LMMs capable of integrating an even wider range of modalities. Lastly, we discussed the utility of Deep Lake in fine-tuning LMMs.


Deep Lake and Data Loaders

# Deep Lake and Data Loaders
## Introduction
In this lesson, we focus on Deep Lake, a powerful AI data system that merges the capabilities of Data Lakes and Vector Databases. We'll explore how Deep Lake can be leveraged for training and fine-tuning Large Language Models, with a focus on its efficient data streaming capabilities. We'll also learn how to create a Deep Lake dataset, add data, and load data using both Deep Lake's and PyTorch's data loaders.


Deep Lake and Data Loaders

# Deep Lake and Data Loaders
## Deep Lake
In the following lessons about training and finetuning LLMs, we’ll need to store the training datasets somewhere, especially for pretraining, since their size is usually too big to be memorized in a single computing node. Ideally, we’d store the datasets elsewhere and efficiently download data in batches when needed. This is where Deep Lake is most useful. Deep Lake is a multi-modal AI data system that merges the capabilities of [Data Lakes](https://docs.activeloop.ai/) and [Vector Databases](https://docs.activeloop.ai/quickstart). Deep Lake is particularly beneficial for businesses looking to train or fine-tune LLMs on their own data. It efficiently streams data from remote storage to GPUs during model training, making it a powerful tool for deep learning applications. Data loaders in Deep Lake are essential components that facilitate efficient data streaming and are very useful for training and fine-tuning LLMs. They are responsible for fetching, decompressing, and transforming data, and they can be optimized to improve performance in GPU-bottlenecked scenarios. Once we store our datasets in Deep Lake, [it’s possible to easily create a PyTorch Dataloader or a TensorFlow Dataset](https://docs.deeplake.ai/en/latest/Pytorch-and-Tensorflow-Support.html). Deep Lake offers two types of data loaders: the [Open Source data loader](https://docs.activeloop.ai/getting-started/deep-learning/connecting-to-ml-frameworks) and the [Performant data loader](https://docs.activeloop.ai/enterprise-features/compute-engine/performant-dataloader). The Performant version, built on a C++ implementation, is faster and optimizes asynchronous data fetching and decompression. It's approximately 1.5 to 3 times faster than the OSS version, depending on the complexity of the transformation and the number of workers available for parallelization, and it supports distributed training.


Deep Lake and Data Loaders

# Deep Lake and Data Loaders
## Creating a Deep Lake Dataset and Adding Data
Now, let's walk through an example of creating a Deep Lake dataset and fetching some data from it. Deep Lake supports a variety of data formats, and you can ingest them directly with a single line of code. Deep Lake can be installed with pip as follows: `pip install deeplake`. Please note that the performant version can be used for free up to 200GB of data stored in the cloud, which is more than we’ll need for the course. Then, create an account at the [Activeloop website](https://www.activeloop.ai/). Next, you’ll need an Activeloop API token, which will allow you to identify yourself from your Python code. To get it, click on the “Create API token” button that you can see at the top of your webpage once you’re logged in, and then proceed to create one by clicking on the other “Create API token” button inside the page. Remember to check the token's expiration date: once it’s expired, you’ll need to create a new one from this page to continue using Deep Lake with Python linked to your account. Once you have your Activeloop token, save it into the `ACTIVELOOP_TOKEN` environmental variable. You can do so by adding it to your `.env` file, which will then be loaded, executing the following Python code with the `dotenv` library. ```python from dotenv import load_dotenv load_dotenv() ``` You are now ready to use Deep Lake! The following Python code shows how we can create a dataset using Deep Lake. Make sure to replace `<YOUR_ACTIVELOOP_USERNAME>` with your username on Activeloop. You can easily find it in the URL of your webpage, which should have the form `https://app.activeloop.ai/<YOUR_ACTIVELOOP_USERNAME>/home`. ```python import deeplake # env variable ACTIVELOOP_TOKEN must be set with your API token # create dataset on deeplake username = "<YOUR_ACTIVELOOP_USERNAME>" dataset_name = "test_dataset" ds = deeplake.dataset(f"hub://{username}/{dataset_name}") # create column text ds.create_tensor('text', htype="text") # add some texts to the dataset texts = [f"text {i}" for i in range(1, 11)] for text in texts: ds.append({"text": text}) ``` In the previous code, we created a Deep Lake dataset named `test_dataset`. We specify that it contains texts, and then we add 10 data samples to it, one by one. Visit the [API docs](https://docs.deeplake.ai/en/latest/Datasets.html) of Deep Lake to learn about the other available methods. Once done, you should see printed text like the following. ``` This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/genai360/test_dataset ``` By clicking on the URL contained in it, you’ll see your dataset directly from the Activeloop website. ![Screenshot 2023-09-06 at 17.27.52.png](Deep%20Lake%20and%20Data%20Loaders%20d98a337b42e24eaea3083dd276bfb604/Screenshot_2023-09-06_at_17.27.52.png) [Deep Lake dataset version control](https://docs.activeloop.ai/getting-started/deep-learning/dataset-version-control) allows you to manage changes to datasets with commands very similar to Git. It provides critical insights into how your data is evolving, and it works with datasets of any size. Execute the following code to commit your changes to the dataset. ```python ds.commit("added texts") ```


Deep Lake and Data Loaders

# Deep Lake and Data Loaders
## Retrieving Data From Deep Lake
Now, let’s get some data from our Deep Lake dataset. There are two main syntaxes for getting data from Deep Lake datasets: 1. The first one uses the Deep Lake **dataloader**. ****It’s highly optimized and has the fastest data streaming. However, it doesn’t support custom sampling or full-random shuffling. It is possible to use PyTorch datasets and data loaders. If you’re interested in knowing more about how to use the Deep Lake data loader in cases where data shuffling is important, read [this guide](https://docs.activeloop.ai/technical-details/shuffling-in-dataloaders). 2. The second one uses plain PyTorch datasets and data loaders, enabling all the customizability that PyTorch supports. However, they have highly sub-optimal streaming using Deep Lake datasets and may result in 5X+ slower performance compared to using Deep Lake data loaders. ### The Deep Lake Data Loader for PyTorch Here’s a code example of creating a Deep Lake data loader for PyTorch. The following code leverages the performant Deep Lake data loader. It’s the fastest and most optimized way of loading data in batches for model training. ```python # create PyTorch data loader batch_size = 3 train_loader = ds.dataloader()\ .batch(batch_size)\ .shuffle()\ .pytorch() # loop over the elements for i, batch in enumerate(train_loader): print(f"Batch {i}") samples = batch.get("text") for j, sample in enumerate(samples): print(f"Sample {j}: {sample}") print() pass ``` You should see the following printed output, showing the retrieved batches. ``` Please wait, filling up the shuffle buffer with samples. Shuffle buffer filling is complete. Batch 0 Sample 0: text 1 Sample 1: text 7 Sample 2: text 8 Batch 1 Sample 0: text 2 Sample 1: text 9 Sample 2: text 6 Batch 2 Sample 0: text 10 Sample 1: text 3 Sample 2: text 4 Batch 3 Sample 0: text 5 ``` ### PyTorch Datasets and PyTorch Data Loaders using Deep Lake This code enables all the customizability supported by PyTorch at the cost of having highly slower streaming compared to using Deep Lake data loaders. The reason for the slower performance is that this approach does not take advantage of the inherent dataset format that was designed for fast streaming by Activeloop. First, we create a subclass of the PyTorch `Dataset`, which stores a reference to the Deep Lake dataset and implements the `__len__` and `__getitem__` methods. ```python from torch.utils.data import DataLoader, Dataset class DeepLakePyTorchDataset(Dataset): def __init__(self, ds): self.ds = ds def __len__(self): return len(self.ds) def __getitem__(self, idx): texts = self.ds.text[idx].text().astype(str) return { "text": texts } ``` Inside the `__getitem__` method, we retrieve the strings stored in the `text` tensor of the dataset at the position `idx`. Then, we instantiate it using a reference to our Deep Lake dataset `ds`, transform it into a PyTorch `DataLoader`, and eventually loop over the elements just like we did with the Deep Lake dataloader example. ```python # create PyTorch dataset ds_pt = DeepLakePyTorchDataset(ds) # create PyTorch data loader from PyTorch dataset dataloader_pytorch = DataLoader(ds_pt, batch_size=3, shuffle=True) # loop over the elements for i, batch in enumerate(dataloader_pytorch): print(f"Batch {i}") samples = batch.get("text") for j, sample in enumerate(samples): print(f"Sample {j}: {sample}") print() pass ``` You should see the following output, showing the retrieved batches. ``` Batch 0 Sample 0: text 8 Sample 1: text 3 Sample 2: text 1 Batch 1 Sample 0: text 4 Sample 1: text 5 Sample 2: text 9 Batch 2 Sample 0: text 7 Sample 1: text 2 Sample 2: text 6 Batch 3 Sample 0: text 10 ```


Deep Lake and Data Loaders

# Deep Lake and Data Loaders
## Getting the Best High-Quality Data for your Models
Recent research, such as from the “[LIMA: Less Is More for Alignment](https://arxiv.org/abs/2305.11206)” and “[Textbooks Are All You Need](https://arxiv.org/abs/2306.11644)” papers, suggests that data quality is very important both for training and finetuning LLMs. As a consequence, Deep Lake has several additional features that can help users investigate the quality of the datasets they are using and eventually filter samples. Deep Lake provides the Tensor [Query Language (TQL)](https://docs.deeplake.ai/en/latest/Tensor-Query-Language.html), an SQL-like language used for [Querying in Activeloop Platform](https://docs.activeloop.ai/enterprise-features/querying-datasets) as well as in **`[ds.query](https://docs.deeplake.ai/en/latest/deeplake.core.dataset.html#deeplake.core.dataset.Dataset.query)`** in the Python API. This allows data scientists to filter datasets and focus their work on the most relevant data. The following code shows how we can filter our dataset using a TQL query and print all the samples in the resulting view. ```python ds_view = ds.query("select * where contains(text, '1')") # code that creates a data loader and prints the batches ... ``` ``` Batch 0 Sample 0: text 1 Sample 1: text 10 ``` Now, we can save our dataset view as follows. ```python ds_view.save_view(id="strings_with_1") ``` And we can read from it as follows. ```python ds = deeplake.dataset(f"hub://{username}/{dataset_name}/.queries/strings_with_1") ``` Another feature is the [samplers](https://docs.deeplake.ai/en/latest/Sampler.html). Samplers can be used to assign a discrete distribution of weights to the dataset's samples, which are then sampled according to the weight distribution. This can be useful for focusing training on higher-quality data.


Deep Lake and Data Loaders

# Deep Lake and Data Loaders
## Conclusion
In this lesson, we explored some of the capabilities of Deep Lake, a multi-modal AI data system that merges the functionalities of Data Lakes and Vector Databases. We've learned how Deep Lake can efficiently stream data from remote storage to GPUs during model training, making it an ideal tool for training and fine-tuning Large Language Models. We've also covered the creation of a Deep Lake dataset, adding data to it, and retrieving data using both Deep Lake's data loaders and PyTorch's data loaders. This will be useful as we continue exploring training and fine-tuning Large Language Models.


What are Large Language Models

# What are Large Language Models
## **Introduction**
Welcome to our introductory module on Large Language Models or **LLMs**. LLMs, or Large Language Models, are a specific category of neural network models characterized by having an exceptionally high number of parameters, often in the billions. These parameters are essentially the variables within the model that allow it to process and generate text. They are trained on vast quantities of textual data, which provides them with a broad understanding of language patterns and structures. The main goal of LLMs is to comprehend and produce text that closely resembles human-written language, enabling them to capture the subtle complexities of both syntax (the arrangement of words in a sentence) and semantics (the meaning conveyed by those words). These models undergo training with a simple objective: predicting the subsequent word in a sentence. However, they develop a range of **emergent abilities** during this training process. For example, they can perform tasks such as arithmetic calculations and word unscrambling and even achieve remarkable feats like [successfully passing professional-level exams such as the US Medical Licensing Exam](https://healthitanalytics.com/news/chatgpt-passes-us-medical-licensing-exam-without-clinician-input). They generate text in an autoregressive manner, generating the next tokens one by one based on the tokens they have previously generated. The **attention mechanism** plays a key role in enabling these models to establish connections between words and produce coherent and contextually relevant text. LLMs have significantly advanced the natural language processing (NLP) field, revolutionizing our approach to tasks like machine translation, natural language generation, part-of-speech tagging, parsing, information retrieval, and more. As we dive further into this module, we will explore the capabilities of these models, their practical applications, and their exciting future possibilities. ---


What are Large Language Models

# What are Large Language Models
## **Language Modeling**
Language modeling is a fundamental task in Natural Language Processing (NLP). It involves explicitly learning the probability distribution of the words in a language. This is generally learned by predicting the next token in a sequence. This task is typically approached using statistical methods or deep learning techniques. LLMs are trained to predict the next token (word, punctuation, etc.) based on the previous tokens in the text. The models achieve this by learning the distribution of tokens in the training data.


What are Large Language Models

# What are Large Language Models
## **Tokenization**
The first step in this process is tokenization, where the input text is broken down into smaller units called tokens. Tokens can be as small as individual characters or as large as whole words. The choice of token size can significantly affect the model's performance. Some models even use subword tokenization, where words are broken down into smaller units that capture meaningful linguistic information. For example, let’s consider the sentence "The child’s book.” We could split the text whenever we find white space characters. The output would be: ```python ["The", "child's", "book."] ``` As you can see, the punctuation is still attached to the words *"child’s"* and *"book."* Otherwise, we could split the text according to white spaces and punctuation. The output would be: ```python ["The", "child", "'", "s", "book", "."] ``` Importantly, tokenization is model-specific, meaning different models require different tokenization processes, which can complicate pre-processing and multi-modal modeling.


What are Large Language Models

# What are Large Language Models
## **Model Architecture and Attention**
The core of a language model is its architecture. Recurrent Neural Networks (RNNs) were traditionally used for this task, as they are capable of processing sequential data by maintaining an internal state that captures the information from previous tokens. However, they struggle with long sequences due to the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem). To overcome these limitations, transformer-based models have become the standard for language modeling tasks. These models use a mechanism called **attention**, which allows them to weigh the importance of different tokens when making predictions. This allows them to capture long-range dependencies between tokens and generate high-quality text.


What are Large Language Models

# What are Large Language Models
## **Training**
The model is trained on a large corpus of text to predict the next token of a sentence correctly. The goal is to adjust the model's parameters to maximize the probability of the observed data. Typically a model is trained on a very large general dataset of texts from the Internet, such as [The Pile](https://pile.eleuther.ai/) or [CommonCrawl](https://commoncrawl.org/). Sometimes also more specific datasets are used, such as the [Stackoverflow Posts](https://huggingface.co/datasets/mikex86/stackoverflow-posts) dataset. > The model learns to predict the next token in a sequence by adjusting its parameters to maximize the probability of outputting the correct next token from the training data. >


What are Large Language Models

# What are Large Language Models
## **Prediction**
Once the model is trained, it can be used to generate text by predicting the next token in a sequence. This is done by feeding the sequence into the model, which outputs a probability distribution over the possible subsequent tokens. The next token is then chosen based on this distribution. This process can be repeated to generate sequences of arbitrary length.


What are Large Language Models

# What are Large Language Models
## **Fine-Tuning**
The model is often fine-tuned on a specific task after pre-training. This involves continuing the training process on a smaller, task-specific dataset. This allows the model to adapt its learned knowledge to the specific task (e.g. text translation) or specialized domain (e.g. biomedical, finance, etc), improving its performance. This is a brief explanation, but the actual process can be much more complex, especially for state-of-the-art models like GPT-4. These models use advanced techniques and large amounts of data to achieve impressive results.


What are Large Language Models

# What are Large Language Models
## Context Size
The context size, or context window, in LLMs is the maximum number of tokens that the model can handle in one go. The context size is significant because it determines the length of the text that can be processed at once, which can impact the model's performance and the results it generates. Different LLMs have different context sizes. For instance, the OpenAI “gpt-3.5-turbo-16k” model has a context window of 16,000 tokens. There is a natural limit to the number of tokens a model can produce. Smaller models can go up to 1k tokens, while larger models can go up to 32k tokens, like GPT-4.


What are Large Language Models

# What are Large Language Models
## Let’s Generate Some Text
Let’s try generating some text with LLMs. You must first generate an API key to use OpenAI’s models in your Python environment. You can follow the below steps to generate the API key: 1. After creating an OpenAI account, log in. 2. After logging in, choose Personal from the top-right menu, then choose “View API keys.” 3. The “Create new secret key” button is on the page containing API keys once step 2 has been finished. Clicking on that generates a secret key. Save this because it will be required in further lessons. After that, you can save your key in a `.env` file like this: ```python OPENAI_API_KEY="<YOUR-OPENAI-API-KEY>" ``` Every time you start a Python script with the following lines, your key will be loaded into an environment variable called `OPENAI_API_KEY`. This environment variable will then be used by the `openai` library whenever you want to generate text. ```python from dotenv import load_dotenv load_dotenv() ``` We are now ready to generate some text! Here’s an example of it. ```python from dotenv import load_dotenv load_dotenv() import os import openai # English text to translate english_text = "Hello, how are you?" response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": f'Translate the following English text to French: "{english_text}"'} ], ) print(response['choices'][0]['message']['content']) ``` ``` Bonjour, comment ça va? ``` By using `dotenv`, you can safely store sensitive information, such as API keys, in a separate file and avoid accidentally exposing it in your code. This is particularly important when working with open-source projects or sharing your code with others, as it ensures that the sensitive information remains secure.


What are Large Language Models

# What are Large Language Models
## **Few-Shot Learning**
Few-shot learning in the context of LLMs refers to providing the model with a few examples before making predictions. These examples "teach" the model how to reason and act as "filters" to help the model search for relevant patterns in the dataset. The idea of few-shot learning is fascinating as it suggests that the model can be quickly reprogrammed for new tasks. While LLMs like GPT3 excel at language modeling tasks like machine translation, they may struggle with more complex reasoning tasks. The few-shot examples are helping the model search for relevant patterns in the dataset. The dataset, which is effectively compressed into the model's weights, can be searched for patterns that strongly respond to these provided examples. These patterns are then used to generate the model's output. The more examples provided, the more precise the output becomes. Here’s an example of few-shot learning: ```python from dotenv import load_dotenv load_dotenv() import os import openai # Prompt for summarization prompt = """ Describe the following movie using emojis. {movie}: """ examples = [ { "input": "Titanic", "output": "🛳🌊❤🧊🎶🔥🚢💔👫💑" }, { "input": "The Matrix", "output": "🕶💊💥👾🔮🌃👨🏻💻🔁🔓💪" } ] movie = "Toy Story" response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": prompt.format(movie=examples[0]["input"])}, {"role": "assistant", "content": examples[0]["output"]}, {"role": "user", "content": prompt.format(movie=examples[1]["input"])}, {"role": "assistant", "content": examples[1]["output"]}, {"role": "user", "content": prompt.format(movie=movie)}, ] ) print(response['choices'][0]['message']['content']) ``` ```python 🧸🤠👦🧒🎢🌈🌟👫🚁👽🐶🚀 ```


What are Large Language Models

# What are Large Language Models
## **Scaling Laws**
Scaling laws refer to the relationship between the model's performance and factors such as the number of parameters, the size of the training dataset, the compute budget, and the network architecture. They were discovered after a lot of experiments and are described in the [Chinchilla paper](https://arxiv.org/abs/2203.15556). These laws provide insights into how to allocate resources when training these models optimally. The main elements characterizing a language model are: 1. The number of parameters (N) reflects the model's capacity to learn from data. More parameters allow the model to capture complex patterns in the data. 2. The size of the training dataset (D) is measured in the number of tokens (small pieces of text ranging from a few words to a single character). 3. FLOPs (floating point operations per second) measure the compute budget used for training. The researchers trained the Chinchilla model, which has 70B parameters, on 1.4 trillion tokens. This aligns with ****the rule of thumb proposed in the paper**: for a model with X parameters, it is optimal to train it on approximately X * 20 tokens.** For example, in the context of this rule, a model with 100 billion parameters would be optimally trained on approximately 2 trillion tokens. Applying this rule, the Chinchilla model, though smaller, performed better than other LLMs. It showed gains in language modeling and task performance and needed less memory and computing power. You can read more about Chinchilla in its paper “[Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556)”.


What are Large Language Models

# What are Large Language Models
## **Emergent Abilities in LLMs**
Emergent abilities in LLMs refer to the sudden appearance of new capabilities as the size of the model increases. These abilities, which include performing arithmetic, answering questions, summarizing passages, and more, are not explicitly trained in the model. Instead, they seem to arise spontaneously as the model scales, hence the term "emergent." > LLMs are probabilistic models that learn patterns in natural language. When these models are scaled up, they not only improve quantitatively in their ability to learn patterns, but they also exhibit qualitative changes in their behavior. > Traditionally, the models require task-specific fine-tuning and architectural modifications to perform specific tasks. However, when scaled, these models can perform these tasks without any architectural modifications or task-specific training. They can do this simply by phrasing the tasks in terms of natural language. This capability of LLMs to perform tasks without fine-tuning is remarkable in itself. What's even more intriguing is how these abilities appear. As LLMs grow, they rapidly and unpredictably transition from near-zero to sometimes state-of-the-art performance. This phenomenon suggests that these abilities are emergent properties of the model's scale rather than being explicitly programmed into the model. This concept of emergent abilities in LLMs has significant implications for the field of AI, as it suggests that scaling up models can lead to the spontaneous development of new capabilities.


What are Large Language Models

# What are Large Language Models
## ****Prompts****
The text containing the instructions that we pass to LLMs is commonly known as prompts. > Prompts are instructions given to AI systems like OpenAI's GPT-3 and GPT-4, providing context to generate human-like text. The more detailed the prompt, the better the model's output. > Shorter, concise, and descriptive prompts tend to yield better results as they leave room for the LLM's creativity. Specific words or phrases can help narrow down potential outcomes and ensure relevant content generation. Writing effective prompts requires a clear goal, simplicity, strategic use of keywords, and actionability. Testing the prompts before publishing ensures the output is relevant and error-free. Here are some prompting tips: 1. Use **precise language** when crafting a prompt – this will help ensure accuracy in the generated output: - Less Precise Prompt: "Write about dogs." - More Precise Prompt: "Write a 500-word informative article about the dietary needs of adult Golden Retrievers." 2. Provide **enough context** around each prompt – this will give a better understanding of what kind of output should be produced: - Less Contextual Prompt: "Write a story." - More Contextual Prompt: "Write a short story set in Victorian England featuring a young detective solving his first major case." 3. Test different **variations** of each prompt – this allows you to experiment with different approaches until you find one that works best: - Initial Prompt: "Write a blog post about the benefits of yoga." - Variation 1: "Compose a 1000-word blog post detailing the physical and mental benefits of regular yoga practice." - Variation 2: "Create an engaging blog post that highlights the top 10 benefits of incorporating yoga into daily routine." 4. **Review** generated outputs before publishing them – while most automated systems produce accurate results, occasionally mistakes occur so it’s always wise to double-check everything before releasing any content into production environments: - Before Review: "Yoga is a great way to improve your flexibility and strength. It can also help reduce stress and improve mental clarity. However, it's important to remember that all yoga poses are suitable for everyone." - After Review (correcting inaccurate information): "Yoga is a great way to improve your flexibility and strength. It can also help reduce stress and improve mental clarity. However, it's important to remember that not all yoga poses are suitable for everyone. Always consult with a healthcare professional before starting any new exercise regimen."


What are Large Language Models

# What are Large Language Models
## Hallucinations and Biases in LLMs
The term **hallucinations** refers to instances where AI systems generate outputs, such as text or images, that don't align with real-world facts or inputs. For example, ChatGPT might generate a plausible-sounding answer to an entirely incorrect factual question. > Hallucinations in LLMs refer to instances where the model generates outputs that do not align with real-world facts or context. This can lead to the propagation of misinformation, especially in critical sectors like healthcare and education where the accuracy of information is of utmost importance. Similarly, bias in LLMs can result in outputs that favor certain perspectives over others, potentially leading to the reinforcement of harmful stereotypes and discrimination. > Consider an interaction where a user asks, "Who won the World Series in 2025?" If the LLM responds with, "The New York Yankees won the World Series in 2025," it's a clear case of hallucination. As of now (July 2023), the 2025 World Series hasn't taken place, so any claim about its outcome is a fabrication. **Bias** in AI and LLMs is another significant issue. It refers to these models' inclination to favor specific outputs or decisions based on their training data. If the training data is predominantly from a specific region, the model might show a bias toward that region's language, culture, or perspectives. If the training data contains inherent biases, such as gender or racial bias, the AI system might produce skewed or discriminatory outputs. For example, if a user asks an LLM, "Who is a nurse?" and it responds with, "She is a healthcare professional who cares for patients in a hospital,” it shows a gender bias. The model automatically associates nursing with women, which doesn't accurately reflect the reality where both men and women can be nurses. Mitigating hallucinations and bias in AI systems involves refining model training, using verification techniques, and ensuring the training data is diverse and representative. Finding a balance between maximizing the model's potential and avoiding these issues remains challenging. Interestingly, in creative domains like media and fiction writing, these "hallucinations" can be beneficial, enabling the generation of unique and innovative content. The ultimate goal is to develop LLMs that are not only powerful and efficient but also reliable, fair, and trustworthy. By doing so, we can maximize the potential of LLMs while minimizing their risks, ensuring that the benefits of this technology are accessible to all.


What are Large Language Models

# What are Large Language Models
## Conclusion
In this introductory module, we explored the fascinating world of LLMs. These powerful models, trained on vast amounts of text data, can understand and generate human-like text. They're built on transformer architectures, allowing them to capture long-range dependencies in language and generate text in an autoregressive manner. We covered the capabilities of LLMs, discussing their impact on the field of NLP. We've learned about few-shot learning, scaling laws, and the emergent abilities of these models. We also acknowledged the challenges that come with these models, including hallucinations and biases, emphasizing the importance of mitigating these issues. In the next lesson, we’ll see a timeline of machine learning models used for language modeling up to the beginning of Large Language Models.


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
> A*n ability is considered as emergent when larger models exhibit it, but it's absent in smaller models - a key factor contributing to the success of Large Language Models.* >


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
## Introduction
In this lesson, we’ll dive more into the concept of **emergent abilities**, the empirical phenomenon of the new abilities that language models get when their size increases over specific thresholds. Emergent abilities become apparent as we scale up the models and are influenced by factors such as training compute and model parameters. We'll also explore various instances of these emergent abilities, focusing on scenarios like few-shots and augmented prompting, and examine the reasons behind the emergence of these abilities and whether further scaling could reveal more of them.


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
## What Are Emergent Abilities
Emergent abilities in LLMs are defined as significant improvements in task performance that become apparent as the model size or scale increases. These abilities, which are not present or noticeable in smaller or less complex models, become evident in larger or more complex models. This suggests that the model is learning and generalizing from its pre-training in ways that were not explicitly programmed or expected. When visualized on a scaling curve, emergent abilities show a pattern where performance is almost random until a certain scale threshold, after which performance increases significantly. This is known as a phase transition, a dramatic change in behavior that could not have been predicted by examining smaller-scale systems. In the following image, taken from the paper “[Emergent Abilities of Large Language Models](https://arxiv.org/pdf/2206.07682.pdf),” we see several charts showing the emergence of abilities of LLMs (whose performance is shown on the y-axis) with respect to the model scale (shown on the x-axis). ![From the paper “[Emergent Abilities of Large Language Models](https://arxiv.org/pdf/2206.07682.pdf)”](Emergent%20Abilities%20in%20LLMs%203c98e33e74e7444bb666d000c6f348ff/Screenshot_2023-07-27_at_16.07.19.png) From the paper “[Emergent Abilities of Large Language Models](https://arxiv.org/pdf/2206.07682.pdf)” Language models have been scaled primarily along computation amount, model parameters, and training dataset size. The emergence of abilities may occur with less training computation or fewer model parameters for models trained on higher-quality data. It also depends on factors such as the amount of data, its quality, and the number of parameters in the model. Emergent abilities in LLMs appear as the models scale up and cannot be predicted by simply extrapolating from smaller models.


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
## Evaluation Benchmarks for Emergent Abilities
Several benchmarks are used to evaluate the emergent abilities of language models. These include the BIG-Bench suite, TruthfulQA, the Massive Multi-task Language Understanding (MMLU) benchmark, and the Word in Context (WiC) benchmark. 1. The first of these is the **BIG-Bench suite**, a comprehensive set of over 200 benchmarks that test a model's capabilities across a variety of tasks. These tasks include **arithmetic operations** where the model is expected to perform the four basic operations (example: “Q: What is 132 plus 762? A: 894), transliteration from the International Phonetic Alphabet (IPA) to measure if the model is able to manipulate and use rare words (example: “English: The 1931 Malay census was an alarm bell. IPA: ðə 1931 ˈmeɪleɪ ˈsɛnsəs wɑz ən əˈlɑrm bɛl.”), word unscrambling that analyzes the model’s ability to work with alphabets. A large number of benchmarks can be found within [the Github repository](https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/README.md) where you can delve into their specific details. The performance of models like GPT-3 and LaMDA on these tasks starts near zero but jumps to significantly above random at a certain scale, demonstrating emergent abilities. 2. Another benchmark is **[TruthfulQA](https://github.com/sylinrl/TruthfulQA)**, which measures a model's capacity to provide truthful responses when addressing questions. The evaluation consists of two tasks: 1) Generation: The model will be asked to answer a question with 1 or 2 sentences. 2) Multiple-choices: The second task involves multiple-choice questions, where the model must choose the correct answer from either 4 options or True/False statements. When the Gopher model is scaled up to its largest size, its performance jumps to more than 20% above random, indicating the emergence of this ability. 3. **The Massive Multi-task Language Understanding ([MMLU](https://arxiv.org/abs/2009.03300))** is another key benchmark. The primary objective of this benchmark is to evaluate models for their ability to demonstrate a broad range of world knowledge and problem-solving skills. The test encompasses 57 tasks, spanning areas such as elementary mathematics, US history, computer science, law, and more. GPTs, Gopher, and Chinchilla models of a specific scale do not perform better than guessing on average of all the topics, but scaling up to a larger size enables performance to surpass random, indicating the emergence of this ability. 4. Finally, the **Word in Context (WiC)** is a semantic understanding benchmark. WiC is a binary classification task for context-sensitive word embeddings. It involves target words (verbs or nouns) with two provided contexts, aiming to determine if they share the same meaning. Chinchilla fails to achieve the one-shot performance of better than random, even when scaled to its largest model size. Above-random performance eventually emerged when PaLM was scaled to a much larger size, suggesting the emergence of this ability at a larger scale.


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
## Other Factors That Could Give Rise To Emergent Abilities
- Multi-step reasoning is a strategy where a model is guided to produce a sequence of intermediate steps before giving the final answer. This strategy, known as **chain-of-thought prompting**, only surpasses standard prompting when applied to a sufficiently large model. - **Instruction following** is another strategy that involves fine-tuning a model on a mixture of tasks phrased as instructions. This strategy only improves performance when applied to a model of a specific size.


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
## **Risks With Emergent Abilities**
As we scale up language models, we also need to be aware of the emergent risks that come with it. These risks could be societal issues related to truthfulness, bias, and toxicity. These risks can be avoided by applying strategies, such as giving model prompts that encourage them to be "helpful, harmless, and honest.” The WinoGender benchmark, which measures gender bias in occupations, has shown that scaling can improve performance but also increase bias in ambiguous contexts. Larger models were found to be more likely to memorize training data, although deduplication methods can reduce this risk. Emergent risks also include phenomena that might only exist in future language models or that have not yet been characterized in current models. These could include backdoor vulnerabilities or harmful content synthesis.


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
## A Shift Towards General-Purpose Models
The emergence of abilities has led to sociological changes in how the community views and uses these models. Historically, NLP focused on task-specific models. Scaling models has led to an explosion in research on "general purpose" models that aim to perform a range of tasks not explicitly encoded in the training data. This shift towards general-purpose models is evident when scaling enables a few-shot prompted general-purpose model to outperform prior state-of-the-art held by fine-tuned task-specific models. For example, GPT-3 achieved a new state-of-the-art on the TriviaQA and PiQA question-answering benchmarks; PaLM achieved a new state-of-the-art on three arithmetic reasoning benchmarks; and the multimodal Flamingo model achieved a new state of the art on six visual question answering benchmarks. The ability of general-purpose models to perform unseen tasks, given only a few examples, has also led to many new applications of language models outside the NLP research community. For instance, language models have been used by prompting to translate natural language instructions into actions that are executable by robots, interact with users, and facilitate multi-modal reasoning.


Emergent Abilities in LLMs

# Emergent Abilities in LLMs
## **Conclusion**
Emergent abilities in LLMs are capabilities that appear as the models scale up and are a key factor in their success. These abilities, unpredictable from smaller models, become evident after reaching a certain scale threshold. They have been observed in various contexts, such as in a few-shot prompting and augmented prompting strategies. Scaling up LLMs also introduces emergent risks like increased bias and toxicity, which can be avoided with appropriate strategies. The emergence of these abilities has led to a shift towards general-purpose models and opened up new applications outside the traditional NLP research community. In the next lesson, we’ll dive into today's most popular proprietary LLMs and describe the tradeoffs between proprietary and open-source LLMs.


Model Quantization

# Model Quantization
## Introduction
As AI models, including large language models, grow more advanced, their increasing number of parameters leads to significant memory usage. This, in turn, increases the costs of hosting and deploying these tools. In this lesson, we will learn about **quantization,** a process that can be employed to **diminish the memory requirements** of these models. We will explore the various types of quantization, such as scalar and product quantization. We will also learn how fine-tuning techniques like [QLoRA](https://arxiv.org/abs/2305.14314) use quantization. Finally, we will examine applying these techniques to AI models using a CPU with methods implemented in the Intel® [neural compressor library](https://github.com/intel/neural-compressor/blob/master/docs/source/quantization.md).


Model Quantization

# Model Quantization
## **Overview of Quantization**
In deep learning, quantization is a technique that **reduces the numerical precision** of model parameters, such as the weights and biases. This reduction helps decrease the model’s memory footprint and computational requirements, enabling easier deployment on resource-constrained devices such as mobile phones, smartwatches, and other embedded systems. ### **Everyday Example** To understand the concept of quantization, consider an everyday scenario. Imagine two friends, Jay and John. Jay asks John, "What’s the time?" John can reply with the exact time, 10:58 p.m., or he can say it's around 11 p.m. In the latter response, John simplifies the time, making it less precise but easier to communicate and understand. This is a basic example of quantization, which is analogous to the process in deep learning, where the precision of model parameters is reduced to make the model more efficient, albeit at the cost of some accuracy. ### **Quantization in Machine Learning** In Machine Learning, different floating point data types can be used for model parameters, a characteristic also called precision. The precision of the data types affects the amount of memory required by the model. Defining the parameters in higher precision types, like Float32 or Float64, provides greater accuracy but requires more memory, while lower precision types, like Float16 or BFloat16, use less memory but may result in a loss of accuracy. In the figure below, you can see the main floating point data types. ![From "[A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes](https://huggingface.co/blog/hf-bitsandbytes-integration)” blog post.](Model%20Quantization%20a26c6e34fa1341bd90928d5989efc540/Untitled.png) From "[A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes](https://huggingface.co/blog/hf-bitsandbytes-integration)” blog post. We can estimate the memory required for an AI model with its number of parameters. For example, consider the **[Llama2 70B](https://arxiv.org/abs/2307.09288)** model that uses Float16 precision for its parameters. Each parameter requires **two bytes**. To calculate the memory required in gigabytes (GB), where 1GB = 1024^3 bytes, the calculation is as follows: $(70,000,000,000 * 2)/ 1024^3 = 130.385 GB$ Now, let's explore the different basic quantization techniques. ### **Scalar Quantization** In scalar quantization, each dimension of the dataset is treated independently. The maximum and minimum values are calculated for each dimension across the dataset. The range between the maximum and minimum values in each dimension is then divided into equal-sized bins. Each value in the dataset is mapped to one of these bins, effectively quantizing the data. For example, consider a dataset of 2000 vectors with 256 dimensions sampled from a Gaussian distribution. The goal is to perform scalar quantization on this dataset. ```python import numpy as np dataset = np.random.normal(size=(2000, 256)) # Calculate and store minimum and maximum across each dimension ranges = np.vstack((np.min(dataset, axis=0), np.max(dataset, axis=0))) ``` Now, calculate each dimension's start value and step size. The start value is the minimum value, and the step size is determined by the number of discrete bins in the integer type being used. This example uses 8-bit unsigned integers (**`uint8`**), providing 256 bins. ```python starts = ranges[0,:] steps = (ranges[1,:] - ranges[0,:]) / 255 ``` The quantized dataset is then calculated as follows: ```python scalar_quantized_dataset = np.uint8((dataset - starts) / steps) ``` The overall scalar quantization process can be encapsulated in a function: ```python def scalar_quantisation(dataset): # Calculate and store minimum and maximum across each dimension ranges = np.vstack((np.min(dataset, axis=0), np.max(dataset, axis=0))) starts = ranges[0,:] steps = (ranges[1,:] - starts) / 255 return np.uint8((dataset - starts) / steps) ``` ### **Product Quantization** In scalar quantization, the data distribution in each dimension should ideally be considered to avoid loss of information. Product quantization can preserve more


Model Quantization

# Model Quantization
## **Overview of Quantization**
information by dividing each vector into sub-vectors and quantizing each sub-vector independently. For example, consider the following array: ```python array = [ [ 8.2, 10.3, 290.1, 278.1, 310.3, 299.9, 308.7, 289.7, 300.1], [ 0.1, 7.3, 8.9, 9.7, 6.9, 9.55, 8.1, 8.5, 8.99] ] ``` Quantizing this array to a 4-bit integer using scalar quantization results in significant information loss: ```python quantized_array = [[ 0 0 14 13 15 14 14 14 14] [ 0 0 0 0 0 0 0 0 0]] ``` In contrast, **product quantization** involves the following steps: 1. Divide each vector in the dataset into *m* disjoint sub-vectors. 2. For each sub-vector, cluster the data into *k* centroids (using *k-*means, for example). 3. Replace each sub-vector with the index of the nearest centroid in the corresponding codebook. Let's proceed with the Product Quantization of the given array with m=3 (number of sub-vectors) and k=2 (number of centroids) ```python from sklearn.cluster import KMeans import numpy as np # Given array array = np.array([ [8.2, 10.3, 290.1, 278.1, 310.3, 299.9, 308.7, 289.7, 300.1], [0.1, 7.3, 8.9, 9.7, 6.9, 9.55, 8.1, 8.5, 8.99] ]) # Number of subvectors and centroids m, k = 3, 2 # Divide each vector into m disjoint sub-vectors subvectors = array.reshape(-1, m) # Perform k-means on each sub-vector independently kmeans = KMeans(n_clusters=k, random_state=0).fit(subvectors) # Replace each sub-vector with the index of the nearest centroid labels = kmeans.labels_ # Reshape labels to match the shape of the original array quantized_array = labels.reshape(array.shape[0], -1) # Output the quantized array quantized_array ``` ```python # Result array([[0, 1, 1], [0, 0, 0]], dtype=int32) ``` By quantizing the vectors and storing only the indices of the centroids, the memory footprint is significantly reduced. This method can help preserve more information than scalar quantization, especially when the distributions of different dimensions are diverse. Product quantization can significantly reduce memory footprint and speed up the nearest neighbor search but at the cost of accuracy. The tradeoff in product quantization is based on the **number of centroids** and the number of sub-vectors we use. The more centroids we use, the better the accuracy, but the memory footprint would not decrease and vice versa.


Model Quantization

# Model Quantization
## Quantizing Large Models
We learned about two relatively basic quantization techniques that can be used with deep learning models. While these simple techniques can work well enough with models with few parameters, they usually lead to a [drop in accuracy](https://arxiv.org/pdf/2208.07339.pdf) for larger models with billions of parameters. ![From “[LLM.int8(): 8bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/pdf/2208.07339.pdf)” paper](Model%20Quantization%20a26c6e34fa1341bd90928d5989efc540/Untitled%201.png) From “[LLM.int8(): 8bit Matrix Multiplication for Transformers at Scale](https://arxiv.org/pdf/2208.07339.pdf)” paper Large models contain a **greater amount of information** in their parameters. With more neurons and layers, large models can represent more complex functions. They can capture deeper and more intricate relationships in the data, which smaller models might not be able to handle. Thus, the quantization process, which reduces the precision of these parameters, can significantly lose this information, resulting in a substantial drop in model accuracy and performance. Optimizing the quantization process for large models is also more difficult due to the larger parameter space. Finding the optimal quantization strategy that minimizes the loss of accuracy while reducing the model size is a more complex task for larger models. ### Popular ****(Post-Training Quantization) Methods for LLMs**** Fortunately, more sophisticated quantization techniques have been released to address these problems, aiming to maintain the accuracy of large models while effectively reducing their size. [**LLM.int8()](https://arxiv.org/abs/2208.07339)** This research paper observes that activation outliers (activation values significantly different from the others) break the quantization of larger models and proposes keeping them in higher precision. By keeping doing that, the performance of the model is not negatively affected. [**GPTQ**](https://arxiv.org/abs/2210.17323) This technique allows for faster text generation. The quantization is done layer by layer, minimizing the mean squared error (MSE) between the quantized and full-precision weights when given an input. The algorithm uses a mixed int4-fp16 quantization scheme where weights are quantized as int4 while activations remain in float16. During inference, weights are de-quantized on the fly, and the actual compute is performed in float16. This method makes use of a calibration dataset. The GPTQ algorithm requires calibrating the quantized weights of the model by making inferences on the quantized model. **[AWQ](https://arxiv.org/abs/2306.00978)** This method is grounded in the observation that not all weights contribute equally to Large Language Models performance. It identifies a small fraction (0.1%-1%) of 'important' or 'salient' weights, the quantization of which, if skipped, can substantially mitigate quantization loss. Unlike traditional approaches that focus on weight distribution, the AWQ method selects these salient weights based on the magnitude of their activations. This approach leads to a notable enhancement in performance. By maintaining only 0.1%-1% of the weight channels, corresponding to larger activations, in the FP16 format, the method significantly boosts the performance of quantized models. ![From “[AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/pdf/2306.00978.pdf)” paper](Model%20Quantization%20a26c6e34fa1341bd90928d5989efc540/Untitled%202.png) From “[AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://arxiv.org/pdf/2306.00978.pdf)” paper The authors note that retaining certain weights in FP16 format can cause hardware inefficiency due to using mixed-precision data types. To address this, they propose a method where all weights, including the salient ones, are quantized to avoid mixed-precision data types. However, before the quantization process, the weights are scaled. This scaling step is crucial as it helps protect the outlier weight channels during quantization, ensuring that the important information they hold is not lost or significantly altered during the quantization process. This method aims to strike a balance, allowing the model to benefit from the quantization efficiency while preserving the essential information in the salient weights.


Model Quantization

# Model Quantization
## Using Quantized models
Many open-source LLMs are available for download in a quantized format. As we learned in this lesson, these models will have reduced memory requirements. You can look at the [model's section](https://huggingface.co/models) on HuggingFace to find and use a quantized model. This platform hosts a variety of models. For instance, you can try the latest [Mistral-7B-Instruct](https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GPTQ) model, which has been quantized using the GPTQ method. ### Quantizing Your Own LLM You can use the [Intel® Neural Compressor Library](https://github.com/intel/neural-compressor/tree/master) to quantize your own Large Language Model. This library offers various techniques for model quantization, some of which have been discussed in this module. To get started, follow the step-by-step guide provided in the **[repository](https://github.com/intel/neural-compressor/tree/8568fd57f8c54032fecbcd3217d1c2ddc35b2402/examples/pytorch/nlp/huggingface_models/language-modeling/quantization/ptq_weight_only)**. This guide will walk you through quantizing a model, ensuring you have all the necessary components and knowledge to proceed. Before beginning the quantization process, ensure you have installed the **`neural-compressor`** library and **`lm-evaluation-harness`**. Inside the cloned [neural compressor directory](https://github.com/intel/neural-compressor/tree/master), navigate to the appropriate directory and install the required packages by running the following commands: ```bash cd examples/pytorch/nlp/huggingface_models/language-modeling/quantization/ptq_weight_only pip install -r requirements.txt ``` As an example, to quantize the **`opt-125m`** model with the GPTQ algorithm, use the following command: ```bash python examples/pytorch/nlp/huggingface_models/language-modeling/quantization/ptq_weight_only/run-gptq-llm.py \ --model_name_or_path facebook/opt-125m \ --weight_only_algo GPTQ \ --dataset NeelNanda/pile-10k \ --wbits 4 \ --group_size 128 \ --pad_max_length 2048 \ --use_max_length \ --seed 0 \ --gpu ``` This command will quantize the **`opt-125m`** model using the specified parameters.


Model Quantization

# Model Quantization
## **How Quantization is used in [QLoRA](https://arxiv.org/pdf/2305.14314.pdf)**
We saw in a [previous lesson](https://www.notion.so/Deep-Dive-into-LoRA-and-SFT-c64980e122b54597843f620db2b557da?pvs=21) how fine-tuning can be achieved using fewer resources using [QLoRa](https://arxiv.org/pdf/2305.14314.pdf), a popular variant of LoRA that makes fine-tuning large language models even more accessible. In the course, we saw that QLoRA involves backpropagating gradients through a frozen, 4-bit quantized pre-trained language model into Low-Rank Adapters. To accomplish this, QLoRA employs a novel data type, the **4-bit NormalFloat (NF4)**, which is theoretically optimal for normally distributed weights. This optimality stems from quantile quantization, a technique particularly suited for normally distributed values. It ensures that each quantization bin holds **an equal number of values** from the input tensor, minimizing quantization error and providing a more uniform data representation. Since pre-trained neural network weights typically exhibit a **zero-centered normal distribution** with a standard deviation (σ), QLoRA transforms all weights into a unified fixed distribution. This transformation is achieved by scaling σ to ensure the distribution aligns perfectly within the range of the NF4 data type, further enhancing the efficiency and accuracy of the quantization process. This new fine-tuning technique shows no accuracy degradation in their experiments and matches BFloat16 performance. ![From “[QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/pdf/2305.14314.pdf)” paper](Model%20Quantization%20a26c6e34fa1341bd90928d5989efc540/Untitled%203.png) From “[QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/pdf/2305.14314.pdf)” paper


Model Quantization

# Model Quantization
## **Conclusion**
In this lesson, we explored the concept of quantization, a technique that can reduce the memory requirements of large models and, in some cases, enhance the text generation speed for language models. We delved into some state-of-the-art quantization techniques suitable for models with billions of parameters, examining the unique contributions of each method. We also learned how to quantize our own models using the [Intel® Neural Compressor Library](https://github.com/intel/neural-compressor/tree/master), which supports many popular quantization methods. Lastly, we revisited QLoRA, understanding how it leverages quantization to make the fine-tuning of models more accessible to a broader audience. --- *Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries.* Special thanks to [Sahibpreet Singh](https://www.linkedin.com/in/sahibpreet-singh-16572b171/) for contributing to this lesson!


Deploying an LLM on a Cloud CPU

# Deploying an LLM on a Cloud CPU
## Introduction
Training a language model can be costly, and expenses associated with deploying it can quickly accumulate over time. Utilizing optimization techniques that enhance the efficiency of the inference process is crucial for minimizing hosting expenses. In this lesson, we will discuss the utilization of the Intel® [Neural Compressor](https://huggingface.co/docs/optimum/main/en/intel/optimization_inc) library to implement quantization techniques. This approach aims to enhance the cost-effectiveness and speed of models when running on CPU instances (it supports also AMD CPU, ARM CPU, and NVidia GPU through ONNX Runtime, but with limited testing). Various techniques can be employed for optimizing a network. Pruning involves trimming the parameter count by targeting less important weights, while knowledge distillation transfers insights from a larger model to a smaller one. Lastly, quantization decreases weight precision from 32 bits to 8 bits. It will significantly decrease the memory needed for loading models and generating responses with minimal accuracy loss. ![Credit: [Deci.ai](https://deci.ai/quantization-and-quantization-aware-training/)](Deploying%20an%20LLM%20on%20a%20Cloud%20CPU%205838e847653d4d999231d249cc1e61a5/deci-quantization-blog-1b.png.webp) Credit: [Deci.ai](https://deci.ai/quantization-and-quantization-aware-training/) The primary focus of this lesson is the quantization technique. We will apply it to an LLM and demonstrate how to perform inference using the quantized model. Ultimately, we will execute several experiments to assess the resulting acceleration. We'll begin by setting up the necessary libraries. Install the `optimum-intel` package directly from its GitHub repository. ```bash pip install git+https://github.com/huggingface/optimum-intel.git@v1.10.1 pip install neural_compressor===2.2.1 pip install onnx===1.14.1 ```


Deploying an LLM on a Cloud CPU

# Deploying an LLM on a Cloud CPU
## Quantization
You can utilize the `optimum-cli` command within the terminal to execute dynamic quantization. Dynamic quantization stands as the recommended approach for transformer-based neural networks. You have the choice to either specify the path to your custom model or select a model from the Huggingface Hub, which will be designated using the `--model` parameter. The `--output` parameter determines the name of the resulting model. We are conducting tests on Facebook's OPT model with 1.3 billion parameters. ```bash optimum-cli inc quantize --model facebook/opt-1.3b --output opt1.3b-quantized ``` If the script fails to recognize your model, you can employ the `--task` parameter. You might use `--task text-generation` for language models. Check the source code for a complete [list of supported tasks](https://github.com/huggingface/optimum/blob/3ffebf994fbd579ab9acb589edc401fa66413928/optimum/exporters/tasks.py#L152). The library also provides a constrained quantization method, enabling you to define a specific target. For example, you can employ an evaluation function to request quantization of the model while experiencing no more than a 5% reduction in accuracy. For further details regarding constrained quantization, please refer to the [library documentation](https://huggingface.co/docs/optimum/main/en/intel/optimization_inc).