# Fine-Tuning using Cohere for Medical Data
## The Dataset
[{'db_name': 'MESH', 'db_id': 'D007022'}]}, {'id': '6', 'offsets': [[306, 322]], 'text': ['alpha-methyldopa'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D008750'}]}, {'id': '7', 'offsets': [[354, 362]], 'text': ['naloxone'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D009270'}]}, {'id': '8', 'offsets': [[364, 372]], 'text': ['Naloxone'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D009270'}]}, {'id': '9', 'offsets': [[469, 481]], 'text': ['hypertensive'], 'type': 'Disease', 'normalized': [{'db_name': 'MESH', 'db_id': 'D006973'}]}, {'id': '10', 'offsets': [[487, 496]], 'text': ['clonidine'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D003000'}]}, {'id': '11', 'offsets': [[563, 576]], 'text': ['[3H]-naloxone'], 'type': 'Chemical', 'normalized': []}, {'id': '12', 'offsets': [[589, 597]], 'text': ['naloxone'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D009270'}]}, {'id': '13', 'offsets': [[637, 646]], 'text': ['clonidine'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D003000'}]}, {'id': '14', 'offsets': [[671, 695]], 'text': ['[3H]-dihydroergocryptine'], 'type': 'Chemical', 'normalized': []}, {'id': '15', 'offsets': [[750, 762]], 'text': ['hypertensive'], 'type': 'Disease', 'normalized': [{'db_name': 'MESH', 'db_id': 'D006973'}]}, {'id': '16', 'offsets': [[865, 873]], 'text': ['naloxone'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D009270'}]}, {'id': '17', 'offsets': [[878, 887]], 'text': ['clonidine'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D003000'}]}, {'id': '18', 'offsets': [[1026, 1035]], 'text': ['clonidine'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D003000'}]}, {'id': '19', 'offsets': [[1039, 1055]], 'text': ['alpha-methyldopa'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D008750'}]}], 'relations': [{'id': 'R0', 'type': 'CID', 'arg1_id': 'D008750', 'arg2_id': 'D007022'}]}], 'dataset_type': 'train'} ``` Now, we can iterate through the dataset, extract the abstracts and related entities, and include the necessary instructions for training. There are two sets of instructions: the first set aids the model in understanding the task, while the second set prompts it how to generate the response. ```python instruction = "The following article contains technical terms including diseases, drugs and chemicals. Create a list only of the diseases mentioned.\n\n" output_instruction = "\n\nList of extracted diseases:\n" ``` The `instruction` variable establishes the guidelines, while the `output_instruction` defines the desired format for the output. Now, we loop through the dataset and format each instance. ```python the_list = [] for item in data: dis = [] if item['dataset_type'] != "test": continue; # Don't use test set # Extract the disease names for ent in item['passages'][1]['entities']: # The annotations if ent['type'] == "Disease": # Only select disease names if ent['text'][0] not in dis: # Remove duplicate diseases in a text dis.append(ent['text'][0]) the_list.append( {'prompt': instruction + item['passages'][1]['text'] + output_instruction, 'completion': "- "+ "\n- ".join(dis)} ) ``` The mention code may appear complex, but for each sample, it essentially iterates through all the annotations and selectively chooses only the disease-related ones. We employ this approach because the dataset includes extra labels for chemicals, which are not relevant to our model. Finally, it will generate a dictionary containing the `prompt` and `completion` keys. The prompt will incorporate the paper abstract and append the instructions to it, whereas the completion will contain a list of disease names, with each name on a separate line. Now, use the following code to save the dataset in JSONL format. ```python # Writing to sample.json with open("disease_instruct_all.jsonl", "w") as outfile: for item in the_list: outfile.write(json.dumps(item) + "\n") ``` The formatted dataset will be saved in a file called `disease_instruct_all.jsonl`. Also, it worth noting that we are concatenating the training and validation set to make a total of 1K samples. The final dataset that is used for fine-tuning has 3K samples which is consists of 1K for diseases + 1K for Chemicals + 1K for their relationships. <aside> 💡 Below is the link to the final preprocessed dataset. [both_rel_instruct_all.jsonl](Fine-Tuning%20using%20Cohere%20for%20Medical%20Data%207aaba9f14aa7475b8dc4f5331866abb5/both_rel_instruct_all.jsonl) </aside>


# Fine-Tuning using Cohere for Medical Data
## The Fine-Tuning
Now, it's time to employ the prepared dataset for the fine-tuning process. The good news is that we have completed the majority of the challenging tasks. The Cohere platform will only request a nickname to save your custom model. It's worth noting that they provide advanced options if you wish to train your model for a longer duration or adjust the learning rate. Here is a detailed guide on training a custom model, and you can also refer to the Cohere documentation for [Training Custom Models](https://docs.cohere.com/docs/finetuning), complete with helpful screenshots. You should navigate to the models page using the sidebar and click on the “Create a custom model” button. On the next page, you will be prompted to select the model type, which, in our case, will be the `Generate` option. It is time to proceed to upload a dataset, either from the previous step or from your custom dataset. Afterward, click the "Review data" button to display a few samples from the dataset. This step is designed to verify that the platform can read your data as expected. If everything appears to be in order, click the "Continue" button. The last step is to chose a nickname for your model. Also, you can change the training hyperparameters by clicking on the “HYPERPARAMETERS (OPTIONAL)” link. You have options such as `train_steps` to determine the duration, `learning_rate` to adjust the model's speed of adaptation, and `batch_size`, which specifies the number of samples the model processes in each iteration, among others. In our experience, the default parameters worked well, but feel free to experiment with these settings. Once you are ready, click the "Initiate training" button. That’s it! Cohere will send you an email once the fine-tuning process is completed, providing you with the model ID for use in your APIs. ### Extract Disease Names In the code snippet below, we employ the same prompt as seen in the first section; however, we use the model ID of the network we just fine-tuned. Let’s see if there are any improvements. ```python response = co.generate( model='2075d3bc-eacf-472e-bd26-23d0284ec536-ft', prompt=prompt, max_tokens=200, temperature=0.750) disease_model = response.generations[0].text print(disease_model) ``` ```python - neurodegeneration - glaucoma - blindness - POAG - glaucomas - retinal degenerative diseases - neurodegeneration - neurodegeneration ``` As evident from the output, the model can now identify a broad spectrum of new diseases, highlighting the effectiveness of the fine-tuning approach. The Cohere platform offers both a user-friendly interface and a potent base model to build upon. ### Extract Chemical Names In the upcoming test, we will assess the performance of our custom models in extracting chemical names compared to the baseline model. To eliminate the need for redundant code mentions, we will only present the prompt, followed by the output of each model for easy comparison. We utilized the following prompt to extract information from a text within the test set. ```python prompt = """The following article contains technical terms including diseases, drugs and chemicals. Create a list only of the chemicals mentioned. To test the validity of the hypothesis that hypomethylation of DNA plays an important role in the initiation of carcinogenic process, 5-azacytidine (5-AzC) (10 mg/kg), an inhibitor of DNA methylation, was given to rats during the phase of repair synthesis induced by the three carcinogens, benzo[a]-pyrene (200 mg/kg), N-methyl-N-nitrosourea (60 mg/kg) and 1,2-dimethylhydrazine (1,2-DMH) (100 mg/kg). The initiated hepatocytes in the liver were assayed as the gamma-glutamyltransferase (gamma-GT) positive foci formed following a 2-week selection regimen consisting of dietary 0.02% 2-acetylaminofluorene coupled with a necrogenic dose of CCl4. The results obtained indicate that with all three carcinogens, administration of 5-AzC during repair synthesis


# Fine-Tuning using Cohere for Medical Data
## The Fine-Tuning
increased the incidence of initiated hepatocytes, for example 10-20 foci/cm2 in 5-AzC and carcinogen-treated rats compared with 3-5 foci/cm2 in rats treated with carcinogen only. Administration of [3H]-5-azadeoxycytidine during the repair synthesis induced by 1,2-DMH further showed that 0.019 mol % of cytosine residues in DNA were substituted by the analogue, indicating that incorporation of 5-AzC occurs during repair synthesis. In the absence of the carcinogen, 5-AzC given after a two thirds partial hepatectomy, when its incorporation should be maximum, failed to induce any gamma-GT positive foci. The results suggest that hypomethylation of DNA per se may not be sufficient for initiation. Perhaps two events might be necessary for initiation, the first caused by the carcinogen and a second involving hypomethylation of DNA. List of extracted chemicals:""" ``` First, we will examine the output of the base model. ``` - 5-azacytidine (5-AzC) - benzo[a]-pyrene - N-methyl-N-nitrosourea - 1,2-dimethylhydrazine - CCl4 - 2-acetylaminofluorene ``` Followed by the predictions generated by the custom fine-tuned model. ``` - 5-azacytidine - 5-AzC - benzo[a]-pyrene - N-methyl-N-nitrosourea - 1,2-dimethylhydrazine - 1,2-DMH - 2-acetylaminofluorene - CCl4 - [3H]-5-azadeoxycytidine - cytosine ``` It is clear that the custom model is better suited for our specific task and adapts readily based on the samples it has encountered. ### Extract Relations The final test involves employing the model to extract complex relationships between chemicals and the diseases they impact. It is an advanced task that could pose some challenges for the base model. As previously done, we begin by introducing the prompt we employed from the test set. ```python prompt = """The following article contains technical terms including diseases, drugs and chemicals. Create a list only of the influences between the chemicals and diseases mentioned. The yield of severe cirrhosis of the liver (defined as a shrunken finely nodular liver with micronodular histology, ascites greater than 30 ml, plasma albumin less than 2.2 g/dl, splenomegaly 2-3 times normal, and testicular atrophy approximately half normal weight) after 12 doses of carbon tetrachloride given intragastrically in the phenobarbitone-primed rat was increased from 25% to 56% by giving the initial "calibrating" dose of carbon tetrachloride at the peak of the phenobarbitone-induced enlargement of the liver. At this point it was assumed that the cytochrome P450/CCl4 toxic state was both maximal and stable. The optimal rat size to begin phenobarbitone was determined as 100 g, and this size as a group had a mean maximum relative liver weight increase 47% greater than normal rats of the same body weight. The optimal time for the initial dose of carbon tetrachloride was after 14 days on phenobarbitone. List of extracted influences:""" ``` Here is the output generated by the base model. ``` severe cirrhosis of the liver influences shrinking, finely nodular, ascites, plasma albumin, splenomegaly, testicular atrophy, carbon tetrachloride, phenobarbitone ``` And here are the generations produced by the custom model. ``` - Chemical phenobarbitone influences disease cirrhosis of the liver - Chemical carbon tetrachloride influences disease cirrhosis of the liver ``` The base model evidently attempts to establish some connections within the text. Nevertheless, it's evident that the custom fine-tuned model excels in producing well-formatted output and distinctly connecting each chemical to the respective disease. This task poses a significant challenge for a general-purpose model; however, it showcases the effectiveness of fine-tuning by simply providing a couple of thousands samples of the task we aim to accomplish.


# Fine-Tuning using Cohere for Medical Data
## Conclusion
As we have examined in various lessons within this chapter, the fine-tuning process has demonstrated itself as a potent tool for extending the capabilities of large language models, even when working with a relatively small amount of data, all while maintaining cost efficiency. For individuals new to the field of AI, especially those who are not well-versed in coding, the no-code approach offered by the Cohere service is an exceptionally powerful option. Our custom model demonstrated superior performance over the base model across three distinct tasks by performing one fine-tuning process, showcasing its capability to effectively follow the patterns presented in the dataset. --- >> [Notebook](https://colab.research.google.com/drive/14NG4M5MA-8BMXUP0qY-nHw7-Urt8RCqN?usp=sharing).


Scaling Laws in LLM Training

# Scaling Laws in LLM Training
## Introduction
In this lesson, we will study the relations between language model performance and parameters like model scale, model shape, and compute budget. The lesson is a small summary of extracts from the papers “[Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361)” and “[Training Compute-Optimal Large Language Models](https://arxiv.org/pdf/2203.15556.pdf).”


Scaling Laws in LLM Training

# Scaling Laws in LLM Training
## **A study on language modeling performance**
The paper [Scaling Laws for Neural Language Models](https://arxiv.org/pdf/2001.08361.pdf) (2020) contains a study of empirical scaling laws for [language model](https://en.wikipedia.org/wiki/Language_model) performance on the cross-entropy loss, focusing on the [Transformer](https://arxiv.org/abs/1706.03762) architecture. The experiments show the test loss scales as a [power-law](https://en.wikipedia.org/wiki/Power_law) with model size, dataset size, and the amount of compute used for training; with some trends spanning more than seven orders of magnitude. This means simple equations govern the relationships between these variables, and these equations can be used to create an optimally efficient training configuration for training a very large language model. Moreover, it looks like other architectural details, such as network width or depth have minimal effects within a wide range. As deduced from the experiments and the derived equations, larger models are significantly more sample efficient, i.e., optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.


Scaling Laws in LLM Training

# Scaling Laws in LLM Training
## **Experiments**
To study language model scaling, a variety of models have been trained with different factors, including: - Model size (*N*): ranging in size from 768 to 1.5 billion non-embedding parameters. - Dataset size (*D*): ranging from 22 million to 23 billion tokens. - Model shape: including depth, width, attention heads, and feed-forward dimension. - Context length: 1024 for most runs, with some experiments with shorter contexts. - Batch size: 2^19 for most runs, with some variations to measure the critical batch size. Training at the critical batch size provides a roughly optimal compromise between time and compute efficiency. Let’s define the following training variables as well: - Let *L* be the test cross-entropy loss. - Let *C* be the amount of compute used to train a model.


Scaling Laws in LLM Training

# **Key findings**
Taking inspiration from section 1.1 of the [paper](https://arxiv.org/pdf/2001.08361.pdf), we summarize the results of the experiments. - **Performance depends strongly on model scale, weakly on model shape:** Model performance depends most strongly on scale, which consists of three factors: the number of model parameters *N* (excluding embeddings), the size of the dataset *D*, and the amount of compute *C* used for training. Within reasonable limits, performance depends very weakly on other architectural hyperparameters, such as depth vs. width. - **Smooth power laws**: Performance has a power-law relationship with each of the three scale factors *N*, *D*, and *C* when not bottlenecked by the other two, with trends spanning more than six orders of magnitude. ![Language modeling performance improves smoothly as we increase the amount of compute, dataset size, and model size used for training. For optimal performance, all three factors must be scaled up in tandem. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf).](Scaling%20Laws%20in%20LLM%20Training%20516524f6257542b4bec5025994e978a3/Screenshot_2023-08-23_at_16.40.25.png) Language modeling performance improves smoothly as we increase the amount of compute, dataset size, and model size used for training. For optimal performance, all three factors must be scaled up in tandem. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf). The paper differentiates between embedding and non-embedding parameters because their size correlates differently with model performance. When including embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters. When excluding embedding parameters, the performance of models with different depths converges to a single trend. ![Left: When including embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters. Right: When excluding embedding parameters, the performance of models with different depths converges to a single trend. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf).](Scaling%20Laws%20in%20LLM%20Training%20516524f6257542b4bec5025994e978a3/Screenshot_2023-08-23_at_16.50.41.png) Left: When including embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters. Right: When excluding embedding parameters, the performance of models with different depths converges to a single trend. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf). - The **universality of overfitting:** Performance improves predictably as long as we scale up *N* and *D* in tandem but leads to diminishing returns if either *N* or *D* is held fixed while the other increases. ![The early-stopped test loss depends predictably on the dataset size D and model size N. Left: For large D, performance is a straight power law in N. Performance stops improving for a more minor fixed D as N increases and the model begins to overfit. Right: The extent of overfitting depends predominantly on a relationship between N and D. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf).](Scaling%20Laws%20in%20LLM%20Training%20516524f6257542b4bec5025994e978a3/Screenshot_2023-08-23_at_16.52.01.png) The early-stopped test loss depends predictably on the dataset size D and model size N. Left: For large D, performance is a straight power law in N. Performance stops improving for a more minor fixed D as N increases and the model begins to overfit. Right: The extent of overfitting depends predominantly on a relationship between N and D. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf). - The **universality of training**: Training curves follow predictable power-laws whose parameters are roughly independent of the model size. Extrapolating the early part of a training curve can roughly predict the loss that would be achieved if trained for much longer. - **Sample efficiency**: Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps and data points. ![A series of language model training runs, with models ranging in size from 10^3 to 10^9 parameters (excluding embeddings). Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf).](Scaling%20Laws%20in%20LLM%20Training%20516524f6257542b4bec5025994e978a3/Screenshot_2023-08-23_at_17.43.12.png) A series of language model training runs, with models ranging in size from 10^3 to 10^9 parameters (excluding embeddings). Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf). ![Left: The early-stopped test loss L(N, D) varies predictably with the


Scaling Laws in LLM Training

# **Key findings**
dataset size D and model size N. Right: After an initial transient period, learning curves for all model sizes N can be fit with an equation parameterized in terms of the number of steps (Smin) when training at large batch size. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf).](Scaling%20Laws%20in%20LLM%20Training%20516524f6257542b4bec5025994e978a3/Screenshot_2023-08-23_at_17.44.11.png) Left: The early-stopped test loss L(N, D) varies predictably with the dataset size D and model size N. Right: After an initial transient period, learning curves for all model sizes N can be fit with an equation parameterized in terms of the number of steps (Smin) when training at large batch size. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf). - **Convergence is inefficient**: When working within a fixed compute budget *C* but without any other restrictions on the model size *N* or available data *D*, we attain optimal performance by training very large models and stopping significantly short of convergence. ![As more computing becomes available, choosing how much to allocate towards training larger models, using larger batches, and training for more steps is possible. This image illustrates this billion-fold increase in computing. Most of the increase should go towards increased model size for optimally compute-efficient training. A relatively small increase in data is needed to avoid reuse. Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf).](Scaling%20Laws%20in%20LLM%20Training%20516524f6257542b4bec5025994e978a3/Screenshot_2023-08-23_at_17.45.23.png) As more computing becomes available, choosing how much to allocate towards training larger models, using larger batches, and training for more steps is possible. This image illustrates this billion-fold increase in computing. Most of the increase should go towards increased model size for optimally compute-efficient training. A relatively small increase in data is needed to avoid reuse. Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required. Image from [https://arxiv.org/pdf/2001.08361.pdf](https://arxiv.org/pdf/2001.08361.pdf). These results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute. Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Larger language models are expected to perform better and be more sample-efficient than current models.


Scaling Laws in LLM Training

# **Key findings**
## **Considerations**
When training large language models, it’s possible to use the relations between *N*, *D*, and *L* to derive the compute scaling, magnitude of overfitting, early stopping step, and data requirements. > The derived scaling relations can be used as a predictive framework. One might interpret these relations as analogs of the [ideal gas law](https://en.wikipedia.org/wiki/Ideal_gas_law), which relates the macroscopic properties of a gas in a universal way, independent of most of the details of its microscopic constituents. > It would be interesting to investigate whether these scaling relations hold in other generative modeling tasks with a maximum likelihood loss and perhaps in other settings and domains (such as images, audio, and video models) as well.


Scaling Laws in LLM Training

# **Key findings**
## **Chinchilla Scaling Laws for Compute-Optimal Training of LLMs**
In 2022, Google DeepMind published the paper “[Training Compute-Optimal Large Language Models](https://arxiv.org/pdf/2203.15556.pdf)” that further explored the scaling laws of LLMs. The researchers conducted extensive experiments to understand the relationship between model size, the number of training tokens, and the compute budget. > The key finding of this study was that current LLMs, such as GPT-3 (175B), Gopher (280B), and Megatron (530B), are significantly undertrained. While these models have increased the number of parameters, the training data remained constant. > The authors proposed that the number of training tokens and model size must be scaled equally for compute-optimal training. They trained approximately 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens. This extensive experimentation led to the creation of a new LLM, Chinchilla, which outperformed its larger counterparts. ![Current LLMs. We show five of the current largest dense transformer models, their size, and the number of training tokens. Other than LaMDA, most models are trained for approximately 300 billion tokens. We introduce Chinchilla, a substantially smaller model, trained for much longer than 300B tokens. Image from [https://arxiv.org/pdf/2203.15556.pdf](https://arxiv.org/pdf/2203.15556.pdf).](Scaling%20Laws%20in%20LLM%20Training%20516524f6257542b4bec5025994e978a3/Screenshot_2023-08-23_at_17.54.53.png) Current LLMs. We show five of the current largest dense transformer models, their size, and the number of training tokens. Other than LaMDA, most models are trained for approximately 300 billion tokens. We introduce Chinchilla, a substantially smaller model, trained for much longer than 300B tokens. Image from [https://arxiv.org/pdf/2203.15556.pdf](https://arxiv.org/pdf/2203.15556.pdf). With 70B parameters and four times more training data, Chinchilla was trained using the same compute budget as the 280B Gopher. The results showed that smaller models could deliver better performance if trained on more data. These smaller models are easier to fine-tune and have less latency at inference. Moreover, they do not need to be trained to their lowest possible loss to be compute optimal. The researchers explored three different approaches to answer the question: "Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?" They assumed a power-law relationship between compute and model size. 1. The first approach involved fixing model sizes and varying the number of training tokens. 2. The second approach, called IsoFLOP profiles, varied the model size for a fixed set of different training FLOP counts. 3. The third approach combined the final loss of the above two approaches as a parametric function of model parameters and the number of tokens. All three approaches suggested that as the compute budget increases, the model size and the training data amount should be of approximately equal proportions. The first and second approaches yielded similar predictions for optimal model sizes, while the third suggested that smaller models would be optimal for larger compute budgets.


Scaling Laws in LLM Training

# **Key findings**
## Conclusion
This lesson has explored the relationship between language model performance and parameters such as model size, dataset size, and compute budget. We've learned that performance scales are a power law with these variables and larger models tend to be more sample-efficient. We also explored the Chinchilla Scaling Laws, which suggest that the number of training tokens and model size should be scaled equally for compute-optimal training. This has led to the creation of smaller models, like Chinchilla, that outperform larger counterparts when trained on more data. These findings provide a predictive framework for training large language models and may have implications for other generative modeling tasks and domains.


Model Quantization

# Introduction
Deep learning has revolutionized various fields, from computer vision to natural language processing. However, one drawback of deep neural networks is their large size and computational demands. These resource-intensive models can significantly hinder deployment, especially in resource-constrained environments like mobile devices and embedded systems. This is where model pruning comes into play as a powerful technique for reducing the size of neural networks without compromising their performance. In this blog post, we will explore what model pruning is, why it's useful, and various methods to achieve it.


Model Quantization

# Introduction
## What is Model Pruning?
Model pruning reduces the size of a deep neural network by removing certain neurons, connections, or even entire layers. The goal is to create a smaller and more efficient model while preserving its accuracy to the greatest extent possible. This reduction in model size leads to benefits such as faster inference times, lower memory footprint, and improved energy efficiency, making it ideal for deployment in resource-limited scenarios. Pruned models are smaller and require fewer computational resources during inference. This is crucial for applications like mobile apps, IoT devices, and edge computing, where computational resources are limited. Moreover, pruned models typically execute faster and are more energy-efficient, enabling real-time applications and improving user experience.


Model Quantization

# Introduction
## Different Types of Model Pruning
There are several techniques and methodologies for model pruning, each with its own advantages and trade-offs. Some of the commonly used methods include: ### **Magnitude-based Pruning (or Unstructured Pruning)** In this approach, model weights or activations with small magnitudes are pruned. The intuition is that small weights contribute less to the model's performance and can be safely removed. The paper titled "[Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures](https://arxiv.org/pdf/1607.03250.pdf)" presented this approach to optimize deep neural networks by pruning unimportant neurons. This technique, known as network trimming, is based on the observation that a significant number of neurons in a large network produce zero outputs, regardless of the inputs received. These zero activation neurons are considered redundant and are removed without impacting the overall accuracy of the network. The process involves iterative pruning and retraining of the network, with the weights before pruning used as initialization. The authors demonstrate through experiments on computer vision neural netowrks that this approach can achieve a high compression ratio of parameters without compromising, and sometimes even improving, the accuracy of the original network. ![Image from [https://arxiv.org/pdf/1607.03250.pdf](https://arxiv.org/pdf/1607.03250.pdf).](Model%20Pruning%200add2ac45f114f53a9bc26b2af127ead/new_image.png) Image from [https://arxiv.org/pdf/1607.03250.pdf](https://arxiv.org/pdf/1607.03250.pdf). The paper "[Learning Efficient Convolutional Networks through Network Slimming](https://arxiv.org/abs/1708.06519)" presented variations of the pruning scheme for deep convolutional neural networks aimed at reducing the model size, decreasing the run-time memory footprint, and lowering the number of computing operations without compromising accuracy. The paper “[A Simple and Effective Pruning Approach for Large Language Models](https://arxiv.org/pdf/2306.11695v1.pdf)” introduces a pruning method called Wanda (Pruning by Weights and activations) for pruning Large Language Models. Pruning is a technique that eliminates a subset of network weights to maintain performance while reducing the model's size. Wanda prunes weights based on the smallest magnitudes multiplied by the corresponding input activations, on a per-output basis. This method is inspired by the recent observation of emergent large magnitude features in LLMs. The key advantage of Wanda is that it does not require retraining or weight updates, and the pruned LLM can be used directly. ![Illustration of our proposed method Wanda (Pruning by Weights and activations), compared with the magnitude pruning approach. Given a weight matrix W and input feature activations X, Wanda computes the weight importance as the elementwise product between the weight magnitude and the norm of input activations (|W| · ∥X∥2). Weight importance scores are compared on a per-output basis (within each row in W), rather than globally across the entire matrix. Image from [https://arxiv.org/pdf/2306.11695v1.pdf](https://arxiv.org/pdf/2306.11695v1.pdf)](Model%20Pruning%200add2ac45f114f53a9bc26b2af127ead/new_image%201.png) Illustration of our proposed method Wanda (Pruning by Weights and activations), compared with the magnitude pruning approach. Given a weight matrix W and input feature activations X, Wanda computes the weight importance as the elementwise product between the weight magnitude and the norm of input activations (|W| · ∥X∥2). Weight importance scores are compared on a per-output basis (within each row in W), rather than globally across the entire matrix. Image from [https://arxiv.org/pdf/2306.11695v1.pdf](https://arxiv.org/pdf/2306.11695v1.pdf) ### **Structured Pruning** Structured pruning targets specific structures within the network, such as channels in convolutional layers or neurons in fully connected layers. The paper "[Structured Pruning of Deep Convolutional Neural Networks](https://arxiv.org/abs/1512.08571)" introduces a new method of network pruning that incorporates structured sparsity at different scales, including channel-wise, kernel-wise, and intra-kernel strided sparsity. This approach is beneficial for computational resource savings. The method uses a particle filtering approach to determine the significance of network connections and paths, assigning importance based on the misclassification rate associated with each connectivity pattern. After pruning, the network is re-trained to compensate for any losses. ### The Lottery Ticket Hypothesis The paper “[The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://arxiv.org/pdf/1803.03635.pdf)” presents an innovative perspective on neural


Model Quantization

# Introduction
## Different Types of Model Pruning
network pruning, introducing the "Lottery Ticket Hypothesis". This hypothesis suggests that within dense, randomly-initialized, feed-forward networks, there exist smaller subnetworks ("winning tickets") that, when trained separately, can achieve test accuracy similar to the original network in a comparable number of iterations. These "winning tickets" are characterized by their initial weight configurations, which make them particularly effective for training. The authors propose an algorithm to identify these "winning tickets" and present a series of experiments to support their hypothesis. They consistently discover "winning tickets" that are 10-20% the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10 datasets. Interestingly, these subnetworks not only match the performance of the original network, but often surpass it, learning faster and achieving higher test accuracy.


Model Quantization

# Introduction
## Intel® Neural Compressor Library
The Intel® Neural Compressor Library is valuable for leveraging already implemented model pruning techniques. Read [this page](https://github.com/intel/neural-compressor/tree/master/neural_compressor/compression/pruner#pruning-types) to learn more about the pruning methods implemented. Here are a couple of pruning methods specifically for LLMs. The paper “[A Fast Post-Training Pruning Framework for Transformers](https://arxiv.org/pdf/2204.09656.pdf)” presents a fast post-training pruning framework for Transformer models, designed to reduce the high inference cost associated with these models. Unlike previous pruning methods that necessitate model retraining, this framework eliminates the need for retraining, thus reducing both the training cost and complexity of model deployment. The framework uses structured sparsity methods to automatically prune the Transformer model given a resource constraint and a sample dataset. To maintain high accuracy, the authors introduce three new techniques: a lightweight mask search algorithm, mask rearrangement, and mask tuning. ![(a) Prior pruning frameworks require additional training on the entire training set and involve user intervention for hyperparameter tuning. This complicates the pruning process and requires a large amount of time (e.g., ∼30 hours). (b) Our pruning framework does not require retraining. It outputs pruned Transformer models satisfying the FLOPs/latency constraints within considerably less time (e.g., ∼3 minutes), without user intervention. Image from [https://arxiv.org/pdf/2204.09656.pdf](https://arxiv.org/pdf/2204.09656.pdf).](Model%20Pruning%200add2ac45f114f53a9bc26b2af127ead/new_image%202.png) (a) Prior pruning frameworks require additional training on the entire training set and involve user intervention for hyperparameter tuning. This complicates the pruning process and requires a large amount of time (e.g., ∼30 hours). (b) Our pruning framework does not require retraining. It outputs pruned Transformer models satisfying the FLOPs/latency constraints within considerably less time (e.g., ∼3 minutes), without user intervention. Image from [https://arxiv.org/pdf/2204.09656.pdf](https://arxiv.org/pdf/2204.09656.pdf). The paper titled "[SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/pdf/2301.00774.pdf)" presents a pruning method, SparseGPT, that can reduce the size of large-scale generative pretrained transformer (GPT) models by at least 50% in a single step, without retraining and with minimal loss of accuracy. The authors demonstrate that SparseGPT can be applied to the very large models OPT-175B and BLOOM-176B, in less than 4.5 hours. The method can achieve 60% unstructured sparsity, meaning that over 100 billion weights can be disregarded during inference without a significant increase in perplexity. ![Sparsity-vs-perplexity comparison of SparseGPT against magnitude pruning on OPT-175B, when pruning to different uniform per-layer sparsities. Image from [https://arxiv.org/pdf/2301.00774.pdf](https://arxiv.org/pdf/2301.00774.pdf).](Model%20Pruning%200add2ac45f114f53a9bc26b2af127ead/new_image%203.png) Sparsity-vs-perplexity comparison of SparseGPT against magnitude pruning on OPT-175B, when pruning to different uniform per-layer sparsities. Image from [https://arxiv.org/pdf/2301.00774.pdf](https://arxiv.org/pdf/2301.00774.pdf).


Model Quantization

# Introduction
## Conclusion
In conclusion, model pruning is a powerful technique for reducing the size of deep neural networks without significantly compromising their performance. It is a valuable tool for deploying models in resource-constrained environments, such as mobile devices and embedded systems. Various pruning methods exist, including magnitude-based pruning and structured pruning, each with its unique advantages and trade-offs. The Intel® Neural Compressor Library provides a practical implementation of these techniques, with specific methods designed for Large Language Models. By understanding and applying these pruning techniques, we can create smaller, faster, and more efficient models that maintain high accuracy, thereby improving the feasibility and user experience of deploying deep learning models in real-world applications. --- *For more information on Intel® Accelerator Engines, visit [this resource page](https://download.intel.com/newsroom/2023/data-center-hpc/4th-Gen-Xeon-Accelerator-Fact-Sheet.pdf). Learn more about Intel® Extension for Transformers, an Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere [here](https://github.com/intel/intel-extension-for-transformers).* *Intel, the Intel logo, and Xeon are trademarks of Intel Corporation or its subsidiaries.*


Understanding Hallucinations and Bias

# Understanding Transformers
## Introduction
In this lesson, we will dive deeper into Transformers and provide a comprehensive understanding of their various components. We will also cover the network's inner mechanisms. We will look into the seminal paper “Attention is all you need” and examine a diagram of the components of a Transformer. Last, we see how Hugging Face uses these components in the popular `transformers` library.


Understanding Hallucinations and Bias

# Understanding Transformers
## Attention Is All You Need
The Transformer architecture was proposed as a collaborative effort between Google Brain and the University of Toronto in a paper called “[Attention is All You Need](https://arxiv.org/abs/1706.03762).” It presented an encoder-decoder network powered by attention mechanisms for automatic translation tasks, demonstrating superior performance compared to previous benchmarks ([WMT 2014 translation tasks](https://paperswithcode.com/dataset/wmt-2014)) at a fraction of the cost. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. While Transformers have proven to be highly effective in various tasks such as classification, summarization, and, more recently, language generation, their proposal of training highly parallelized networks is equally significant. The expansion of the architecture into three distinct categories allowed for greater flexibility and specialization in handling different tasks: - The **encoder-only** category focused on extracting meaningful representations from input data. An example model of this category is [BERT](https://arxiv.org/abs/1810.04805). - The **encoder-decoder** category enabled sequence-to-sequence tasks such as translation and summarization or training multimodal models like caption generators. An example model of this category is [BART](https://arxiv.org/abs/1910.13461). - The **decoder-only** category specializes in generating outputs based on given instructions, as we have in Large Language Models. An example model of this category is [GPT](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf).


Understanding Hallucinations and Bias

# Understanding Transformers
## The Architecture
Now, let's examine the crucial elements of the Transformer model in more detail. ![The overview of Transformer architecture. The left component is called the encoder, which is connected to the decoder using a cross-attention mechanism. (Image taken from the “*Attention is all you need” paper*)](Understanding%20Transformers%2086c4c813b12f485283a3806ec8e25381/Untitled.png) The overview of Transformer architecture. The left component is called the encoder, which is connected to the decoder using a cross-attention mechanism. (Image taken from the “*Attention is all you need” paper*) ### Input Embedding The initial procedure involves translating the input tokens into embeddings. These embeddings are acquired vectors symbolizing the input tokens, facilitating the model's ability to grasp the semantic meanings of the words. The size of the embedding vector varied based on the model's scale and design preferences. For instance, OpenAI's GPT-3 uses a 12,000-dimensional embedding vector, while smaller models like BERT could have a size as small as 768. ### Positional Encoding Given that the Transformer lacks the recurrence feature found in RNNs to feed the input one at a time, it necessitates a method for considering the position of words within a sentence. This is accomplished by adding positional encodings to the input embeddings. These encodings are vectors that keep the location of a word in the sentence. ### Self-Attention Mechanism At the core of the Transformer model lies the self-attention mechanism, which calculates a weighted sum of the embeddings of all words in a sentence for each word. These weights are determined based on some learned “attention” scores between words. The terms with higher relevance to one another will receive higher “attention” weights. Based on the inputs, this is implemented using Query, Key, and Value vectors. Here is a brief description of each vector. - **Query Vector**: It represents the word or token for which the attention weights are being calculated. The Query vector determines which parts of the input sequence should receive more attention. Multiplying word embeddings with the Query vector is like asking, "What should I pay attention to?" - **Key Vector**: It represents the set of words or tokens in the input sequence that are compared with the Query. The Key vector helps identify the relevant or essential information in the input sequence. Multiplying word embeddings with the Key vector is like asking, "What is important to consider?" - **Value Vector**: It contains the input sequence's associated information or features for each word or token. The Value vector provides the actual data that will be weighted and combined based on the attention weights calculated between the Query and Key. The Value vector answers the question, "What information do we have?" Before the advent of the transformer architecture, the attention mechanism was mainly utilized to compare two portions of texts. For example, the model could focus on different parts of the input article while generating the summary for a task like summarization. The self-attention mechanism enabled the models to highlight the important parts of the content for the task. It is helpful in encoder-only or decoder-only models to create a powerful representation of the input. The text can be transformed into embeddings for encoder-only scenarios, whereas the text is generated for decoder-only models. The effectiveness of the attention mechanism significantly increases when applied in a multi-head setting. In this configuration, multiple attention components process the same information, with each head learning to focus on distinct aspects of the text, such as verbs, nouns, numbers, and more, throughout the training process.


Understanding Hallucinations and Bias

# Understanding Transformers
## The Architecture In Action
This section will demonstrate the functioning of the above components from a pre-trained large language model, providing an insight into their inner workings using the `transformers` Hugging Face library. To begin, we load the model and tokenizer using `AutoModelForCausalLM` and `AutoTokenizer`, respectively. Then, we proceed to tokenize a sample phrase, which will serve as our input in the following steps. ```python from transformers import AutoModelForCausalLM, AutoTokenizer OPT = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b", load_in_8bit=True) tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b") inp = "The quick brown fox jumps over the lazy dog" inp_tokenized = tokenizer(inp, return_tensors="pt") print(inp_tokenized['input_ids'].size()) print(inp_tokenized) ``` ```python torch.Size([1, 10]) {'input_ids': tensor([[ 2, 133, 2119, 6219, 23602, 13855, 81, 5, 22414, 2335]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])} ``` We load Facebook's Open Pre-trained Transformer model with 1.3B parameters (`facebook/opt-1.3b`) in the 8-bit format, a memory-saving approach to efficiently utilize GPU resources. The `tokenizer` object loads the required vocabulary to interact with the model and will be used to convert the sample input (`inp` variable) to the token IDs and attention mask. Let’s look at the model’s architecture by accessing its `.model` method. ```python print(OPT.model) ``` ```python OPTModel( (decoder): OPTDecoder( (embed_tokens): Embedding(50272, 2048, padding_idx=1) (embed_positions): OPTLearnedPositionalEmbedding(2050, 2048) (final_layer_norm): LayerNorm((2048,), eps=1e-05, elementwise_affine=True) (layers): ModuleList( (0-23): 24 x OPTDecoderLayer( (self_attn): OPTAttention( (k_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) (v_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) (q_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) (out_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) ) (activation_fn): ReLU() (self_attn_layer_norm): LayerNorm((2048,), eps=1e-05, elementwise_affine=True) (fc1): Linear8bitLt(in_features=2048, out_features=8192, bias=True) (fc2): Linear8bitLt(in_features=8192, out_features=2048, bias=True) (final_layer_norm): LayerNorm((2048,), eps=1e-05, elementwise_affine=True) ) ) ) ) ``` The model is decoder-only, a common characteristic among transformer-based language models. Consequently, we must utilize the decoder key to access its inner components. Furthermore, the examination of the `layers` key reveals that the decoder component is composed of 24 stacked layers with the same architecture. To begin, we look at the embedding layer. ```python embedded_input = OPT.model.decoder.embed_tokens(inp_tokenized['input_ids']) print("Layer:\t", OPT.model.decoder.embed_tokens) print("Size:\t", embedded_input.size()) print("Output:\t", embedded_input) ``` ```python Layer: Embedding(50272, 2048, padding_idx=1) Size: torch.Size([1, 10, 2048]) Output: tensor([[[-0.0407, 0.0519, 0.0574, ..., -0.0263, -0.0355, -0.0260], [-0.0371, 0.0220, -0.0096, ..., 0.0265, -0.0166, -0.0030], [-0.0455, -0.0236, -0.0121, ..., 0.0043, -0.0166, 0.0193], ..., [ 0.0007, 0.0267, 0.0257, ..., 0.0622, 0.0421, 0.0279], [-0.0126, 0.0347, -0.0352, ..., -0.0393, -0.0396, -0.0102], [-0.0115, 0.0319, 0.0274, ..., -0.0472, -0.0059, 0.0341]]], device='cuda:0', dtype=torch.float16, grad_fn=<EmbeddingBackward0>) ``` The embedding layer is accessible through the `.embed_tokens` method under the decoder component and passes our tokenized inputs to the layer. As you can see, the embedding layer will transform a list of IDs with `[1, 10]` size to `[1, 10, 2048]`. This representation will then be used and passed through the decoder layers. Subsequently, the positional encoding component utilizes the attention masks to generate a vector that imparts a sense of positioning within the model. The following code uses the `.embed_positions` method from the decoder to generate the positional embeddings. As seen, the layer generates a distinct vector for each position, which is added to the output of the embedding layer. This process introduces supplementary positional information to the model. ```python embed_pos_input = OPT.model.decoder.embed_positions(inp_tokenized['attention_mask']) print("Layer:\t", OPT.model.decoder.embed_positions) print("Size:\t", embed_pos_input.size()) print("Output:\t", embed_pos_input) ``` ```python Layer: OPTLearnedPositionalEmbedding(2050, 2048) Size: torch.Size([1, 10, 2048]) Output: tensor([[[-8.1406e-03, -2.6221e-01, 6.0768e-03, ..., 1.7273e-02, -5.0621e-03, -1.6220e-02], [-8.0585e-05, 2.5000e-01, -1.6632e-02, ..., -1.5419e-02, -1.7838e-02, 2.4948e-02], [-9.9411e-03, -1.4978e-01, 1.7557e-03, ..., 3.7117e-03, -1.6434e-02, -9.9087e-04], ..., [ 3.6979e-04, -7.7454e-02, 1.2955e-02, ..., 3.9330e-03, -1.1642e-02, 7.8506e-03], [-2.6779e-03, -2.2446e-02, -1.6754e-02, ..., -1.3142e-03, -7.8583e-03, 2.0096e-02], [-8.6288e-03, 1.4233e-01, -1.9012e-02, ..., -1.8463e-02, -9.8572e-03, 8.7662e-03]]], device='cuda:0', dtype=torch.float16, grad_fn=<EmbeddingBackward0>) ``` Lastly, the self-attention component! We use the first layer’s self-attention component by indexing through the layers and accessing the `.self_attn` method. ```python embed_position_input = embedded_input + embed_pos_input hidden_states, _, _ = OPT.model.decoder.layers[0].self_attn(embed_position_input) print("Layer:\t", OPT.model.decoder.layers[0].self_attn) print("Size:\t", hidden_states.size()) print("Output:\t", hidden_states) ``` ```python Layer:


Understanding Hallucinations and Bias

# Understanding Transformers
## The Architecture In Action
OPTAttention( (k_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) (v_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) (q_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) (out_proj): Linear8bitLt(in_features=2048, out_features=2048, bias=True) ) Size: torch.Size([1, 10, 2048]) Output: tensor([[[-0.0119, -0.0110, 0.0056, ..., 0.0094, 0.0013, 0.0093], [-0.0119, -0.0110, 0.0056, ..., 0.0095, 0.0013, 0.0093], [-0.0119, -0.0110, 0.0056, ..., 0.0095, 0.0013, 0.0093], ..., [-0.0119, -0.0110, 0.0056, ..., 0.0095, 0.0013, 0.0093], [-0.0119, -0.0110, 0.0056, ..., 0.0095, 0.0013, 0.0093], [-0.0119, -0.0110, 0.0056, ..., 0.0095, 0.0013, 0.0093]]], device='cuda:0', dtype=torch.float16, grad_fn=<MatMul8bitLtBackward>) ``` The self-attention component comprises the mentioned query, key, and value layers, culminating in a final projection for the output. It takes the sum of the embedded input and the positional encoding vector as input. In a real-world example, the model also provides the attention mask to the component, enabling it to identify which portions of the input should be disregarded or ignored. (removed from the sample code for simplicity) The rest of the architecture applies non-linearity (e.g., RELU), feedforward, and batch normalization layers. <aside> 💡 If you are interested in learning the Transformer architecture in more detail and implement a GPT-like network from scratch, we recommend watching the following video from Andrej Karpathy: [https://www.youtube.com/watch?v=kCc8FmEb1nY](https://www.youtube.com/watch?v=kCc8FmEb1nY). </aside>


Understanding Hallucinations and Bias

# Understanding Transformers
## Conclusion
This lesson provides an overview of the transformer architecture and dives deeper into the model's structure by loading a pre-trained model and extracting its essential components. We also look into what occurs within an LLM under the surface. In particular, the attention mechanism serves as the core component of the model. In the next lesson, we will cover the diverse architectures of the transformer: encoder-decoder, decoder-only (like the GPTs), and decoder-only (like BERT). In this [Notebook](https://colab.research.google.com/drive/1FS9dRZh_ZemS4uQpKu9k1W_bDtP0DMoh?usp=sharing), you can find the code for this lesson.


Techniques for Fine-Tuning LLMs

# Fine-Tuning LLMs Module
## Fine-Tuning LLMs
Goals: Equip students with knowledge and practical skills in finetuning techniques accompanied by code examples. Discuss the approach to finetuning utilizing CPUs. The section centers around fine-tuning LLMs, addressing their various aspects and methodologies. As the module progresses, the focus will be given to specialized instruction tuning techniques, namely SFT and and LoRA. It will examine domain-specific applications, ensuring a holistic understanding of fine-tuning techniques and their real-world implications. - **Techniques for Finetuning LLMs:** The lesson highlights the challenges, particularly the resource intensity of traditional approaches. We will introduce instruction tuning methods like SFT, RLHF, and LoRA. - **Deep Dive into LoRA and SFT**: This lesson offers an in-depth exploration of LoRA and SFT techniques. We will uncover the mechanics and underlying principles of these methods. - **Finetuning using LoRA and SFT**: This lesson guides a practical application of LoRA and SFT to finetune an LLM to follow instructions, using data from the “LIMA: Less Is More for Alignment” paper. - **Finetuning using SFT for financial sentiment**. This lesson navigates the nuances of leveraging SFT to optimize LLMs, specifically tailored to capture and interpret sentiments within the financial domain. - **Fine-Tuning using Cohere for Medical Data.** In this lesson, we will adopt an entirely different method for fine-tuning a large language model, leveraging a service called [Cohere](https://cohere.com/). This lesson explores the procedure of fine-tuning a customized generative model using medical texts to extract information. The task, known as [Named Entity Recognition (NER)](https://en.wikipedia.org/wiki/Named-entity_recognition), empowers the models to identify various entities (such as names, locations, dates, etc.) within a text. The lessons have equipped students with both the knowledge and the practical skills needed for fine-tuning LLMs effectively. As they advance, they carry with them the capability to deploy and optimize LLM for various domains.


Deploying LLMs Module

# Deploying LLMs Module
## Deploying LLMs
Goals: Familiarize students with efficient LLM deployment techniques, emphasizing quantization and pruning. Offer hands-on experiences with deployments on platforms like GCP and Intel® CPUs. This module dives into deploying Large Language Models. Model quantization and pruning are central to these strategies, each serving as an effective tool to optimize model performance without compromising efficiency. With a foundation in research articles and real-world applications, this module also introduces participants to deployment on cloud platforms. - **Challenges of LLM deployment**: The lesson covers challenges during LLM deployment such as the sheer size of models, associated costs, and potential latency issues. We also provide a survey on optimizations, rooted in a research article on Transformer inference and a deepened perspective on potential solutions in addressing these challenges. - **Model Quantization**: This lesson centers on Quantization, highlighting its role in streamlining LLM deployments. We research the balance between model performance and efficiency by understanding its usefulness and various techniques. - **Model Pruning**: This module discusses model pruning, showcasing its place in LLM optimization. We will introduce various pruning techniques backed by recent research. - **Deploying an LLM on a Cloud CPU**: This module uncovers the advantages, considerations, and challenges of deploying large language models on cloud-based CPUs. This lesson needs a server instance equipped with an Intel® Xeon® processor. By the end of this module, students have gained a robust understanding of the intricacies involved in LLM deployment. The exploration of model quantization, pruning, and practical deployment strategies has provided them with the tools necessary to navigate real-world challenges. Moving beyond foundational concepts, the next section offers a deep dive into the advanced topics and future directions in the realm of LLMs. After navigating the diverse terrain of Transformers and LLMs, participants now deeply understand significant architectures like GPT and BERT. The sessions shed light on model evaluation metrics, advanced control techniques for optimal outputs, and the roles of pretraining and finetuning. The upcoming module dives into the complexities of deciding when to train an LLM from scratch, the operational necessities of LLMs, and the sequential steps crucial for the training process. --- *Intel, the Intel logo and Xeon are trademarks of Intel Corporation or its subsidiaries.*


Challenges of LLM Deployment

# Challenges of LLM Deployment
## Introduction
In this lesson, we will study the challenges of deploying Large Language Models, with a focus on the importance of latency and memory. We also explore optimization techniques with concepts like quantization and sparsity and how they can be applied using the Hugging Face Optimum and Intel® Neural Compressor libraries. We also discuss the role of Intel's® optimization technologies in efficiently running LLMs. This lesson will provide a deeper understanding of how to optimize LLMs for better performance and user experience.


Challenges of LLM Deployment

# Challenges of LLM Deployment
## Importance of Latency and Memory
Latency is the delay before a transfer of data begins following an instruction. It is a crucial factor in LLM applications. High latency in real-time or near-real-time applications can lead to a poor user experience. For instance, in a conversational AI application, a delay in response can disrupt the natural flow of conversation, leading to user dissatisfaction. Therefore, reducing latency is a critical aspect of LLM deployment. Considering an [average human reading speed](https://www.sciencedirect.com/science/article/abs/pii/S0749596X19300786) of ~250 words per minute (translated to ~312 tokens per minute) that is about 5 tokens per second, therefore a latency of 200ms per token. Usually, acceptable latency for near-real-time LLM applications is between 100ms and 200ms per token. Transformers can be computationally intensive and memory-demanding, due to their complex architecture and large size. However, several optimization techniques can be employed to enhance their efficiency without significantly compromising their performance.


Challenges of LLM Deployment

# Challenges of LLM Deployment
## Quantization
[Quantization](https://arxiv.org/pdf/2103.13630.pdf) is a technique used for compressing neural network models, including Transformers, by lowering the precision of model parameters and/or activations. This method can significantly reduce memory usage. It leverages low-bit precision arithmetic and decreases the size, latency, and energy consumption. However, it's important to strike a balance between performance gains through reduced precision and maintaining model accuracy. Techniques such as mixed-precision quantization, which assign higher bit precision to more sensitive layers, can mitigate accuracy degradation. We’ll learn different quantization methods later in the course.


Challenges of LLM Deployment

# Challenges of LLM Deployment
## Sparsity
[Sparsity](https://arxiv.org/abs/2102.00554), usually achieved by [pruning](https://en.wikipedia.org/wiki/Pruning_(artificial_neural_network)), is another technique for reducing the computational cost of LLMs by eliminating redundant or less important weights and activations. This method can significantly decrease off-chip memory consumption, the corresponding memory traffic, energy consumption, and latency. Pruning can be broadly divided into types: weight pruning and activation pruning. - **Weight pruning** can be further categorized into unstructured pruning and structured pruning. Unstructured pruning allows any sparsity pattern, and structured pruning imposes an additional constraint on the sparsity pattern. While structured pruning can provide benefits in terms of memory, energy consumption, and latency without additional hardware support, it is known to achieve a lower compression rate than unstructured pruning. - On the other hand, **activation pruning** prunes redundant activations during inference, which can be especially effective for Transformer models. However, this requires support to detect and zero out unimportant activations at run-time dynamically. We’ll study different pruning methods later in the course.


Challenges of LLM Deployment

# Challenges of LLM Deployment
## Utilizing Optimum and Intel® Neural Compressor Libraries
The [Hugging Face Optimum](https://github.com/huggingface/optimum) and the [Intel® Neural Compressor](https://github.com/intel/neural-compressor/tree/master) libraries provide a suite of tools helpful in optimizing models for inference, especially for Intel® architectures. - The Hugging Face Optimum library serves as an interface between the Hugging Face [transformers](https://github.com/huggingface/transformers) and [diffuser](https://github.com/huggingface/diffusers) libraries and the various tools provided by Intel®. - The Intel® Neural Compressor is an open-source library that facilitates the application of popular compression techniques such as quantization, pruning, and [knowledge distillation](https://github.com/intel/neural-compressor/blob/master/docs/source/distillation.md). It supports automatic accuracy-driven tuning strategies, enabling users to generate quantized models easily. This library allows users to apply static, dynamic, and aware-training quantization approaches while maintaining predefined accuracy criteria. It also supports different weight pruning techniques, allowing for the creation of pruned models that meet a predefined sparsity target. These libraries provide a practical application of the quantization and sparsity techniques, and their usage will be of great use in optimizing the deployment of LLMs.


Challenges of LLM Deployment

# Challenges of LLM Deployment
## Intel® Optimization Technologies for LLMs
Intel's® optimization technologies play a significant role in running LLMs efficiently on CPUs. The [4th Gen Intel® Xeon® Scalable processors](https://www.intel.com/content/www/us/en/developer/topic-technology/artificial-intelligence/platform.html) are equipped with AI-infused acceleration known as Intel® Advanced Matrix Extensions (Intel® AMX). These processors have built-in BF16 and INT8 GEMM (general matrix-matrix multiplication) accelerators in every core, which significantly accelerate deep learning training and inference workloads. The [Intel® Xeon® Proecssor Max Series](https://www.intel.com/content/www/us/en/products/details/processors/xeon/max-series.html) offers up to 128GB of high-bandwidth memory, which is particularly beneficial for LLMs, as these models are often memory-bandwidth bound. By (1) running model optimizations like quantization and pruning and (2) leveraging the Intel® hardware acceleration technologies, it’s possible to achieve a good latency for LLMs too. Take a look at [this page](https://github.com/intel/neural-compressor/blob/master/docs/source/validated_model_list.md#pytorch-models-with-torch-201cpu-in-ptq-mode) to see the performance improvements (better throughput, with less memory size) of several optimized models.


Challenges of LLM Deployment

# Challenges of LLM Deployment
## Conclusion
In this lesson, we have explored the challenges of deploying Large Language Models, with a particular focus on latency and memory. We also discussed optimization techniques like quantization and sparsity, which can significantly reduce LLMs' computational cost and memory usage. We introduced the Hugging Face Optimum and Intel® Neural Compressor libraries, which provide practical tools for applying these techniques. Furthermore, we have highlighted the role of Intel's® optimization technologies, such as the 4th Gen Intel® Xeon Scalable processors and the Intel® Xeon CPU Max Series, in efficiently running neural networks. By understanding and applying these concepts, we can optimize the deployment of LLMs, achieving better performance and user experience. --- *For more information on Intel® Accelerator Engines, visit [this resource page](https://download.intel.com/newsroom/2023/data-center-hpc/4th-Gen-Xeon-Accelerator-Fact-Sheet.pdf). Learn more about Intel® Extension for Transformers, an Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere [here](https://github.com/intel/intel-extension-for-transformers).* *Intel, the Intel logo, and Xeon are trademarks of Intel Corporation or its subsidiaries.*


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Introduction
The fine-tuning process has consistently proven to be a practical approach for enhancing the model's capabilities in new domains. Therefore, it is a valuable approach to adapt large language models while using a reasonable amount of resources. As mentioned earlier, the fine-tuning process builds upon the model's existing general knowledge, which means it doesn't need to learn everything from scratch. Consequently, it can grasp patterns from a relatively small number of samples and undergo a relatively short training process. In this lesson, we’ll see how to do SFT on an LLM using LoRA. We’ll use the dataset from the "[LIMA: Less Is More for Alignment](https://arxiv.org/pdf/2305.11206.pdf)" paper. According to their argument, a high-quality, hand-picked, small dataset with a thousand samples can replace the RLHF process, effectively enabling the model to be instructively fine-tuned. Their approach yielded competitive results compared to other language models, showcasing a more efficient fine-tuning process. However, it might not exhibit the same level of accuracy in domain-specific tasks, and it requires hand-picked data points. The [TRL library](https://github.com/huggingface/trl) has some classes for Supervised Fine-Tuning (SFT), making it accessible and straightforward. The classes permit the integration of LoRA configurations, facilitating its seamless adoption. It is worth highlighting that this process also serves as the first step for Reinforcement Learning with Human Feedback (RLHF), a topic we will explore in detail later in the course.


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Spinning Up a Virtual Machine for Finetuning on GCP Compute Engine
Cloud GPUs availability today is very scarse as they are used a lot for several deep learning applications. Few people know that CPUs can be actually used to finetune LLMs through various optimizations and that’s what we’ll be doing in these lessons when doing SFT. Let’s login to our Google Cloud Platform account and create a [Compute Engine](https://cloud.google.com/compute) instance (see the “Course Introduction” lesson for instructions). You can choose between different [machine types](https://cloud.google.com/compute/docs/cpu-platforms). In this lesson, we trained the model on the latest CPU generation from 4th Generation Intel® Xeon® Scalable Processors (formerly known as Intel® Sapphire Rapids). This architecture features an integrated accelerator designed to enhance the performance of training deep learning models. Intel® Advanced Matrix Extension (AMX) empowers the training of models with BF16 precision during the training process, allowing for half-precision training on the latest Xeon® Scalable processors. Additionally, it introduces an INT8 data type for the inference process, leading to a substantial acceleration in processing speed. Reports suggest a tenfold increase in performance when utilizing PyTorch for both training and inference processes. Follow the instructions in the course introduction to spin up a VM with Compute Engine with high-end Intel® CPUs. Once you have your virtual machine up, you can SSH into it. Incorporating CPUs for fine-tuning or inference processes presents an excellent choice, as renting alternate hardware is considerably less cost-effective. It worth mentioning that a minimum of 32GB of RAM is necessary to load the model and facilitate the experiment's training process. If there is an out-of-memory error, reduce arguments such as `batch_size` or `seq_length`. <aside> ⚠ Beware of costs when you spin up virtual machines. The total cost will depend on the machine type and the up time of the machine. Always remember to monitor your costs in the billing section of GCP and to spin off your virtual machines when you don’t use them. </aside> <aside> 💡 If you just want to replicate the code in the lesson spending very few money, you can just run the training in your virtual machine and stop it after a few iterations. </aside>


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Load the Dataset
The quality of a model is directly tied to the quality of the data it is trained on! The best approach is to begin the process with a dataset. Whether it is an open-source dataset or a custom one manually, planning and considering the dataset in advance is essential. In this lesson, we will utilize the dataset released with the LIMA research. It is publicly available with a non-commercial use license. The powerful feature of Deep Lake format enables seamless streaming of the datasets. There is no need to download and load the dataset into memory. The hub provides diverse datasets, including the LIMA dataset presented in the "LIMA: Less Is More for Alignment" paper. The Deep Lake Web UI not only aids in dataset exploration but also facilitates dataset visualization using the embeddings field, taking care of clustering the dataset and map it in 3D space. (We used Cohere embedding API to generate in this example) The enlarged image below illustrates one such cluster where data points in Portuguese language related to coding are positioned closely to each other. Note that Deep Lake Visualization Engine offers you the ability to pick the clustering algorithm. ![Deep Lake Visualization Engine 3D visualization feature.](Fine-Tuning%20using%20LoRA%20and%20SFT%207d646ca7e86c435b8827bfdb9d060c0e/Screenshot_2023-10-05_at_9.41.15_AM.png) Deep Lake Visualization Engine 3D visualization feature. The code below will create a loader object for the training and test sets. ```python import deeplake # Connect to the training and testing datasets ds = deeplake.load('hub://genai360/GAIR-lima-train-set') ds_test = deeplake.load('hub://genai360/GAIR-lima-test-set') print(ds) ``` ```python Dataset(path='hub://genai360/GAIR-lima-train-set', read_only=True, tensors=['answer', 'question', 'source']) ``` We can then utilize the `ConstantLengthDataset` class to bundle a number of smaller samples together, enhancing the efficiency of the training process. Furthermore, it also handles dataset formatting by accepting a template function and tokenizing the texts. To begin, we load the pre-trained tokenizer object for the [Open Pre-trained Transformer (OPT)](https://arxiv.org/abs/2205.01068) model using the Transformers library. We will load the model later. We are using OPT for convenience because it’s an open model with a relatively “small” amount of parameters. The same code in this lesson can be run in another model too, for example, using `meta-llama/Llama-2-7b-chat-hf` for [LLaMa 2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf). ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b") ``` Moreover, we need to define the formatting function called `prepare_sample_text`, which takes a row of data in Deep Lake format as input and formats it to begin with a question followed by the answer that is separated by two newlines. This formatting aids the model in learning the template and understanding that if a prompt starts with the `question` keyword, the most likely response would be to complete it with an answer. ```python def prepare_sample_text(example): """Prepare the text from a sample of the dataset.""" text = f"Question: {example['question'].text()}\n\nAnswer: {example['answer'].text()}" return text ``` Now, with all the components in place, we can initialize the dataset, which can be fed to the model for fine-tuning. We call the `ConstantLengthDataset` class using the combination of a tokenizer, deep lake dataset object, and formatting function. The additional arguments, such as `infinite=True` ensure that the iterator will restart when all data points have been used, but there are still training steps remaining. Alongside `seq_length`, which determines the maximum sequence length, it must be completed according to the model's configuration. In this scenario, it is possible to raise it to 2048, although we opted for a smaller value to manage memory usage better. Select a higher number if the dataset primarily comprises shorter texts. ```python from trl.trainer import ConstantLengthDataset train_dataset = ConstantLengthDataset( tokenizer, ds, formatting_func=prepare_sample_text, infinite=True, seq_length=1024 ) eval_dataset = ConstantLengthDataset( tokenizer, ds_test, formatting_func=prepare_sample_text, seq_length=1024 ) # Show one sample from train set iterator = iter(train_dataset) sample =


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Load the Dataset
next(iterator) print(sample) ``` ```python {'input_ids': tensor([ 2, 45641, 35, ..., 48443, 2517, 742]), 'labels': tensor([ 2, 45641, 35, ..., 48443, 2517, 742])} ``` As evidenced by the output above, the `ConstantLengthDataset` class takes care of all the necessary steps to prepare our dataset. <aside> 💡 If you use the iterator to print a sample from the dataset, remember to execute the following code to reset the iterator pointer. `train_dataset.start_iteration = 0` </aside>


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Initialize the Model and Trainer
As mentioned previously, we will be using the [OPT model](https://huggingface.co/facebook/opt-1.3b) with 1.3 billion parameters in this lesson, which has the `facebook/opt-1.3b` model id on the Hugging Face Hub. The LoRA approach is employed for fine-tuning, which involves introducing new parameters to the network while keeping the base model unchanged during the tuning process. This approach has proven to be highly efficient, enabling fine-tuning of the model by training less than 1% of the total parameters. (For more details, refer to the following [post](https://medium.com/@nlpiation/pre-trained-transformers-gpt-3-2-but-1000x-smaller-cafe4269a96c).) With the TRL library, we can seamlessly add additional parameters to the model by defining a number of configurations. The variable `r` represents the dimension of matrices, where lower values lead to fewer trainable parameters. `lora_alpha` serves as the scaling factor, while `bias` determines which bias parameters the model should train, with options of `none`, `all`, and `lora_only`. The remaining parameters are self-explanatory. ```python from peft import LoraConfig lora_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", ) ``` Next, we need to configure the `TrainingArguments`, which are essential for the training process. We have already covered some of the parameters in the training lesson, but note that the learning rate is higher when combined with higher weight decay, increasing parameter updates during fine-tuning. Furthermore, it is highly recommended to employ the argument `bf16=True` in order to minimize memory usage during the model's fine-tuning process. The utilization of the Intel® Xeon® 4s CPU empowers us to apply this optimization technique. This involves converting the numbers to a 16-bit precision, effectively reducing the RAM demand during fine-tuning. We will dive into other quantization methods as we progress through the course. We are also using a service called [Weights and Biases](https://wandb.ai/site), which is an excellent tool for training and fine-tuning any machine-learning model. They offer monitoring tools to record every facet of the process and various solutions for [prompt engineering](https://wandb.ai/site/prompts) and [hyperparameter sweep](https://docs.wandb.ai/guides/sweeps), among other functionalities. Simply installing the package and utilizing the `wandb` parameter for the `report_to` argument is all that's required. This will handle the logging process seamlessly. ```python from transformers import TrainingArguments training_args = TrainingArguments( output_dir="./OPT-fine_tuned-LIMA-CPU", dataloader_drop_last=True, evaluation_strategy="epoch", save_strategy="epoch", num_train_epochs=10, logging_steps=5, per_device_train_batch_size=8, per_device_eval_batch_size=8, learning_rate=1e-4, lr_scheduler_type="cosine", warmup_steps=10, gradient_accumulation_steps=1, bf16=True, weight_decay=0.05, run_name="OPT-fine_tuned-LIMA-CPU", report_to="wandb", ) ``` The final component we need is the pre-trained model. We will use the `facebook/opt-1.3b` key to load the model using the Transformers library. ```python from transformers import AutoModelForCausalLM import torch model = AutoModelForCausalLM.from_pretrained("facebook/opt-1.3b", torch_dtype=torch.bfloat16) ``` The subsequent code block will loop through the model parameters and revert the data type of specific layers (like LayerNorm and final language modeling head) to a 32-bit format. It will improve the fine-tuning stability. ```python import torch.nn as nn for param in model.parameters(): param.requires_grad = False # freeze the model - train adapters later if param.ndim == 1: # cast the small parameters (e.g. layernorm) to fp32 for stability param.data = param.data.to(torch.float32) model.gradient_checkpointing_enable() # reduce number of stored activations model.enable_input_require_grads() class CastOutputToFloat(nn.Sequential): def forward(self, x): return super().forward(x).to(torch.float32) model.lm_head = CastOutputToFloat(model.lm_head) ``` Finally, we can use the `SFTTrainer` class to tie all the components together. It accepts the model, training arguments, training dataset, and LoRA method configurations to construct the trainer object. The `packing` argument indicates that we used the `ConstantLengthDataset` class earlier to pack samples together. ```python from trl import SFTTrainer trainer = SFTTrainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, peft_config=lora_config, packing=True, ) ``` So, why did we use LoRA? Let's observe its impact in action by implementing a simple function that calculates the number of available parameters in the model and compares it with the trainable parameters. As a reminder, the trainable parameters refer to the ones that LoRA


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Initialize the Model and Trainer
added to the base model. ```python def print_trainable_parameters(model): """ Prints the number of trainable parameters in the model. """ trainable_params = 0 all_param = 0 for _, param in model.named_parameters(): all_param += param.numel() if param.requires_grad: trainable_params += param.numel() print( f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}" ) print( print_trainable_parameters(trainer.model) ) ``` ```python trainable params: 3145728 || all params: 1318903808 || trainable%: 0.23851079820371554 ``` As observed above, the number of trainable parameters is only 3 million. It accounts for only 0.2% of the total number of parameters that we would have had to update if we hadn't used LoRA! It significantly reduces the memory requirement. Now, it should be clear why using this approach for fine-tuning is advantageous. The trainer object is fully prepared to initiate the fine-tuning loop by calling the `.train()` method, as shown below. ```python print("Training...") trainer.train() ``` <aside> 💡 You can access the best checkpoint that we trained by using the following URL. Additionally, find more information about the fine-tuning process on the Weights and Biases [project page](https://wandb.ai/ala_/GenAI360/runs/uhe0kbku?workspace=user-ala_). </aside> [OPT-fine_tuned-LIMA-CPU.zip](Fine-Tuning%20using%20LoRA%20and%20SFT%207d646ca7e86c435b8827bfdb9d060c0e/OPT-fine_tuned-LIMA-CPU.zip)


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Merging LoRA and OPT
The final step involves merging the base model with the trained LoRA layers to create a standalone model. This can be achieved by loading the desired checkpoint from SFTTrainer, followed by the base model itself using the `PeftModel` class. Begin by loading the OPT-1.3B base model if using a fresh environment. ```python from transformers import AutoModelForCausalLM import torch model = AutoModelForCausalLM.from_pretrained( "facebook/opt-1.3b", return_dict=True, torch_dtype=torch.bfloat16 ) ``` The `PeftModel` class can merge the base model with the LoRA layers from the checkpoint specified using the `.from_pretrained()` method. We should then put the model in the evaluation mode. Upon execution, it will print out the model's architecture to observe the presence of the LoRA layers. ```python from peft import PeftModel # Load the Lora model model = PeftModel.from_pretrained(model, "./OPT-fine_tuned-LIMA-CPU/<desired_checkpoint>/") model.eval() ``` ```python PeftModelForCausalLM( (base_model): LoraModel( (model): OPTForCausalLM( (model): OPTModel( (decoder): OPTDecoder( (embed_tokens): Embedding(50272, 2048, padding_idx=1) (embed_positions): OPTLearnedPositionalEmbedding(2050, 2048) (final_layer_norm): LayerNorm((2048,), eps=1e-05, elementwise_affine=True) (layers): ModuleList( (0-23): 24 x OPTDecoderLayer( (self_attn): OPTAttention( (k_proj): Linear(in_features=2048, out_features=2048, bias=True) (v_proj): Linear( in_features=2048, out_features=2048, bias=True (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (q_proj): Linear( in_features=2048, out_features=2048, bias=True (lora_dropout): ModuleDict( (default): Dropout(p=0.05, inplace=False) ) (lora_A): ModuleDict( (default): Linear(in_features=2048, out_features=16, bias=False) ) (lora_B): ModuleDict( (default): Linear(in_features=16, out_features=2048, bias=False) ) (lora_embedding_A): ParameterDict() (lora_embedding_B): ParameterDict() ) (out_proj): Linear(in_features=2048, out_features=2048, bias=True) ) (activation_fn): ReLU() (self_attn_layer_norm): LayerNorm((2048,), eps=1e-05, elementwise_affine=True) (fc1): Linear(in_features=2048, out_features=8192, bias=True) (fc2): Linear(in_features=8192, out_features=2048, bias=True) (final_layer_norm): LayerNorm((2048,), eps=1e-05, elementwise_affine=True) ) ) ) ) (lm_head): Linear(in_features=2048, out_features=50272, bias=False) ) ) ) ``` Lastly, we can use the PEFT model’s `.merge_and_unload()` method to combine the base model and LoRA layers as a standalone object. It is possible to save the weights using the `.save_pretrained()` method for later usage. ```python model = model.merge_and_unload() model.save_pretrained("./OPT-fine_tuned-LIMA/merged") ``` <aside> 💡 Prior to progressing to the next section to observe the outcomes of the fine-tuned model, it's important to reiterate that the base model employed in this lesson is a relatively small language model with limited capabilities when compared with the state-of-the-art models we are accustomed to by now, such as ChatGPT. Remember that the insights gained from this lesson can be easily applied to train significantly larger variations of the models, leading to notably improved outcomes. (As highlighted in the lesson's introduction, modifying the key used for loading the tokenizer/model to models with any size like LLaMA2 is possible.) </aside>


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Inference
We can evaluate the fine-tuned model’s outputs by employing various prompts. The code below demonstrates how we can utilize Huggingface's `.generate()` method to interact with models effortlessly. Numerous arguments and decoding strategies exist that can enhance text generation quality; however, these are beyond the scope of this course. You can explore these techniques further in an informative [blog post](https://huggingface.co/blog/how-to-generate) by Huggingface. ```python inputs = tokenizer("Question: Write a recipe with chicken.\n\n Answer: ", return_tensors="pt") generation_output = model.generate(**inputs, return_dict_in_generate=True, output_scores=True, max_length=256, num_beams=1, do_sample=True, repetition_penalty=1.5, length_penalty=2.) print( tokenizer.decode(generation_output['sequences'][0]) ) ``` ``` Question: Write a recipe with chicken.\n\n Answer: \n* Chicken and rice is one of the most popular meals in China, especially during Chinese New Year celebrations when it's served as an appetizer or main course for dinner parties (or just to eat by yourself). It can be made from scratch using fresh ingredients like meatballs/chicken breasts if you have them on hand but otherwise use frozen ones that are already cooked so they don't need any additional cooking time before serving. You could also substitute some vegetables instead such as broccoli florets which would make this dish even more delicious! If your family doesn’t know how to cook well then I suggest making these recipes ahead of time because once done all you really do is reheat until hot again :)\n## Make homemade marinade\n1) Combine 1 tablespoon soy sauce, 2 tablespoons sesame oil, 3 teaspoons sugar, 4 cloves garlic minced into small pieces, 6-8 green onions chopped finely, 5 cups water, salt & pepper to taste, about 8 ounces boneless skinless chicken breast fillets cut up fine enough not to stick together while being mixed thoroughly - no bones needed here since there will only ever be two servings per person), ½ cup cornstarch dissolved in ¼... ``` To carry out further experimentation with the `OPT-fine_tuned-LIMA` model, we presented an identical prompt to both the vanilla base model and the fine-tuned version. This experiment aims to measure the degree to which each of these models can follow instructions. Below is a list of prompts. You can toggle the outputs by clicking on the right arrow icon. - 1. Write a recipe with chicken. ![Screenshot 2023-08-16 at 7.11.32 PM.png](Fine-Tuning%20using%20LoRA%20and%20SFT%207d646ca7e86c435b8827bfdb9d060c0e/Screenshot_2023-08-16_at_7.11.32_PM.png) - 2. Create a marketing plan for a coffee shop. ![Screenshot 2023-08-16 at 7.09.51 PM.png](Fine-Tuning%20using%20LoRA%20and%20SFT%207d646ca7e86c435b8827bfdb9d060c0e/Screenshot_2023-08-16_at_7.09.51_PM.png) - 3. Why does it rain? Explain your answer. ![Screenshot 2023-08-16 at 7.06.03 PM.png](Fine-Tuning%20using%20LoRA%20and%20SFT%207d646ca7e86c435b8827bfdb9d060c0e/Screenshot_2023-08-16_at_7.06.03_PM.png) - 4. What’s the Italian translation of the word ‘house’? ![Screenshot 2023-08-16 at 7.02.03 PM.png](Fine-Tuning%20using%20LoRA%20and%20SFT%207d646ca7e86c435b8827bfdb9d060c0e/Screenshot_2023-08-16_at_7.02.03_PM.png) The outcomes highlight the constraints and capabilities of both models. However, it is evident that the fine-tuned model learned to follow instructions better compared to the vanilla-based model. This effect would undoubtedly become more pronounced with the availability of resources to conduct the fine-tuning process for a large model.


Fine-Tuning using LoRA and SFT

# Fine-Tuning using LoRA and SFT
## Conclusion
During this lesson, we experimented with the fine-tuning process of Large Language models, utilizing the LoRA technique to achieve an efficient tuning process. During our exploration, we discovered the importance of the process. It can serve as a starting point for RLHF or be used for instruction tuning. In the upcoming lessons, we will experiment with the fine-tuning process for creating domain-specific models. --- >> [Notebook](https://colab.research.google.com/drive/1v7gtuE2CIosiF3nt4WSV9BGiQVGUWJRt?usp=sharing). >> [W&B Report](https://wandb.ai/ala_/GenAI360/runs/uhe0kbku?workspace=user-ala_). --- *For more information on Intel® Accelerator Engines, visit [this resource page](https://download.intel.com/newsroom/2023/data-center-hpc/4th-Gen-Xeon-Accelerator-Fact-Sheet.pdf). Learn more about Intel® Extension for Transformers, an Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere [here](https://github.com/intel/intel-extension-for-transformers).* *Intel, the Intel logo, and Xeon are trademarks of Intel Corporation or its subsidiaries.*


ReAct framework and ChatGPT plugins

# ReAct framework and ChatGPT plugins
## Introduction
Large Language Models have demonstrated great utility in diverse tasks, from coding assistance and content summarization to answering everyday questions. As our understanding of their strengths and limitations grows, numerous innovative methods for improvement and expanding their range of tasks have emerged in recent months. This lesson will delve into some of these advancements. We will learn about the [ReAct framework](https://arxiv.org/pdf/2210.03629.pdf), a **prompt-based paradigm** designed to synergize **reasoning** and **acting** in language models for **general task solving**. Additionally, this module will cover the latest upgrades to [ChatGPT](https://chat.openai.com/auth/login), including [plugins](https://openai.com/blog/chatgpt-plugins) integration. We will also explore new enhancements available through the OpenAI API, such as [function calling](https://openai.com/blog/function-calling-and-other-api-updates). This feature enables LLMs to produce structured outputs, further augmenting their reliability and utility in many applications.


ReAct framework and ChatGPT plugins

# ReAct framework and ChatGPT plugins
## Overview of ReAct
This framework aims to enhance the utility of language models by using them to create **autonomous agent systems**, which are systems that can operate and make decisions independently. A way to accomplish that is to make these models reason about an input question and context, create an action plan, and execute it. In the **[ReAct framework](https://arxiv.org/pdf/2210.03629.pdf)**, language models are prompted to generate **verbal reasoning traces,** which are detailed records of the model's thought process, and **actions** interleaved when accomplishing a task. The process is done iteratively until the answer to a question is found. The verbal reasoning step in ReAct allows the model to dynamically create, maintain, and adjust high-level plans for acting, which refers to the execution of specific tasks or actions. When acting, the model can also interact with external environments (like Wikipedia) to incorporate additional information into reasoning. As a side note, allowing these models to access sources of information like Wikipedia can also reduce the number of [hallucinations and biases](https://www.notion.so/Understanding-Hallucinations-and-Bias-e417eb8cce7849a58b2c8a4b0ef0f6f1?pvs=21). To better understand the framework, let’s look at the example below from the paper. ![From “[ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/pdf/2210.03629.pdf)” paper](ReAct%20framework%20and%20ChatGPT%20plugins%20806d77f2ff6c4a16be58c011998b358e/Untitled.png) From “[ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/pdf/2210.03629.pdf)” paper The figure above provides a comparative analysis of different methods a language model is utilized to accomplish a specific task: identifying a device that can control the same program the Apple Remote was first designed to interact with. - (1a) The model is asked to answer the question directly. - (1b) Uses **[Chain-of-Thought](https://arxiv.org/pdf/2201.11903.pdf)** prompting, which asks the LLM to reason about the question before answering. - (1c) An act-only process where the LLM is not prompted to reason about the question. - (1d) Using the ReAct framework, the LLM is prompted to reason about the question and perform a specific action to find the answer. The authors of ReAct note that the framework recipe accomplishes various tasks. The main steps include: 1. **Thought Step**: The LLM is prompted to think critically about the task. Given the question, it evaluates which actions might lead to finding the answer. 2. **Action Steps**: In this phase, the LLM interacts with an external environment. It can utilize external APIs to acquire necessary information if needed. 3. **Observation Step**: After taking action, the LLM receives a result from the external environment. These observations are crucial for the LLM to determine the effectiveness of the action and plan the next steps. 4. **Next Thought Step**: Equipped with the information from the action and observation, the LLM reevaluates the situation. This evaluation allows the model to consider and decide on the subsequent action. This sequential process continues until the LLM successfully finds the answer. ### ReAct framework in code Code implementations of the ReAct framework are available for those interested in creating autonomous agents. For a practical demonstration, consider looking at the **[author's implementation](https://github.com/ysymyth/ReAct/blob/master/hotpotqa.ipynb)**. This link directs you to a notebook showcasing an example of utilizing **`text-davinci-002`** to create an agent that answers questions using Wikipedia as a source of information. There is also a **[LangChain](https://python.langchain.com/docs/modules/agents/agent_types/react.html#using-lcel)** implementation of ReAct, enabling you to create a capable agent in less time as they have many available agent [tools](https://python.langchain.com/docs/integrations/tools/).


ReAct framework and ChatGPT plugins

# ReAct framework and ChatGPT plugins
## OpenAI Function calling
OpenAI has recently introduced a [function calling](https://openai.com/blog/function-calling-and-other-api-updates) feature for their language models through their API. In an API call, you can describe functions to **`gpt-3.5-turbo-0613`** and **`gpt-4-0613`**, and have the model output a JSON object containing arguments to call those functions. Be aware that the Chat completions API **does not call the function**; instead, the model generates a JSON object that you can use to **call the function in your code**. To include this feature, OpenAI fine-tuned the models **`gpt-3.5-turbo-0613`** and **`gpt-4-0613`** to detect when a function should be called (depending on the user input) and to respond with a JSON object that adheres to the function signature. It’s not disclosed how they implement this, but they may be using a form of prompt engineering similar to ReAct for this. With this feature, you can more easily create: - Chatbots that answer questions by calling external tools, such as sending an email. - Convert natural language into API calls or database queries so the model can answer questions such as “Who are my top ten customers this month?” - Extract structured data from text, such as extracting the names of all the locations mentioned in a Wikipedia article. Check out this [example](https://platform.openai.com/docs/guides/gpt/function-calling) in the OpenAI documentation to learn how to set up function calling. If your application makes use of LangChain, there is also a way to use function calling; take a look at the documentation [here](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent).