# Fine-Tuning using SFT for Financial Sentiment
## Inference
We randomly selected four previously unseen examples from the dataset and provided them as input to both the vanilla base model (OPT-1.3B) and the fine-tuned model in order to contrast their respective responses. The code is relatively straightforward when utilizing the `.generate()` method from the Transformers library. ```python inputs = tokenizer("""What is the sentiment of this news? Please choose an answer from {strong negative/moderately negative/mildly negative/neutral/mildly positive/moderately positive/strong positive}, then provide some short reasons.\n\n Content: UPDATE 1-AstraZeneca sells rare cancer drug to Sanofi for up to S300 mln.\n\nSentiment: """, return_tensors="pt").to("cuda:0") generation_output = model.generate(**inputs, return_dict_in_generate=True, output_scores=True, max_length=256, num_beams=1, do_sample=True, repetition_penalty=1.5, length_penalty=2.) print( tokenizer.decode(generation_output['sequences'][0]) ) ``` ``` What is the sentiment of this news? Please choose an answer from {strong negative/moderately negative/mildly negative/neutral/mildly positive/moderately positive/strong positive}, then provide some short reasons. Content: UPDATE 1-AstraZeneca sells rare cancer drug to Sanofi for up to S300 mln. Sentiment: positive ``` Observing the samples, we see that the model fine-tuned on financial tweets for the specific domain exhibits good performance in terms of adhering to instructions and comprehending the task at hand. Below, find a list of prompts to toggle the outputs by clicking on the right arrow icon. - 1. UPDATE 1-AstraZeneca sells rare cancer drug to Sanofi for up to S300 mln. ***Correct Answer: Positive*** ![Screenshot 2023-08-21 at 9.43.51 AM.png](Fine-Tuning%20using%20SFT%20for%20Financial%20Sentiment%20f744059ba2f8490b92b5c5a641c1e96a/Screenshot_2023-08-21_at_9.43.51_AM.png) - 2. SABMiller revenue hit by weaker EM currencies ***Correct Answer: Negative*** ![Screenshot 2023-08-21 at 9.45.08 AM.png](Fine-Tuning%20using%20SFT%20for%20Financial%20Sentiment%20f744059ba2f8490b92b5c5a641c1e96a/Screenshot_2023-08-21_at_9.45.08_AM.png) - 3. Buffett's Company Reports 37 Percent Drop in 2Q Earnings ***Correct Answer: Negative*** ![Screenshot 2023-08-21 at 9.48.49 AM.png](Fine-Tuning%20using%20SFT%20for%20Financial%20Sentiment%20f744059ba2f8490b92b5c5a641c1e96a/Screenshot_2023-08-21_at_9.48.49_AM.png) - 4. For a few hours this week, the FT gained access to Poly, the university where students in Hong Kong have been trapped… ***Correct Answer: Neutral*** ![Screenshot 2023-08-21 at 9.47.23 AM.png](Fine-Tuning%20using%20SFT%20for%20Financial%20Sentiment%20f744059ba2f8490b92b5c5a641c1e96a/Screenshot_2023-08-21_at_9.47.23_AM.png) These instances demonstrate that the vanilla model primarily focuses on the default language modeling task, which involves predicting the next word based on the input. In contrast, the fine-tuned model comprehends the instruction and generates the requested content.


# Fine-Tuning using SFT for Financial Sentiment
## Conclusion
This tutorial illustrated the procedure of leveraging publicly accessible datasets or curated data from your organization to develop a personalized model that caters to personalized requirements. More powerful base models, such as LLaMA2 can be employed, by modifying the model id to load both the model and its tokenizer. However, it needs more resources to initiate the fine-tuning process. We also showcased the feasibility of conducting the fine-tuning process using 4th Generation Intel® Xeon® Scalable Processors for both the fine-tuning and inference stages. The Intel® [oneAPI](https://www.intel.com/content/www/us/en/developer/tools/oneapi/toolkits.html#gs.5qwv2n) Math Kernel Library (Intel® MKL) toolkits offer a suite of utilities aimed at boosting the efficiency of various applications, including those in the field of Artificial Intelligence. It's intriguing to anticipate how the latest CPU architectures like Emerald Rapids, Sierra Forest, and Granite Rapids will shape and potentially transform the deep learning landscape. In the upcoming lessons, we will integrate GPUs to fine-tune a model using the RLHF (Reinforcement Learning from Human Feedback) approach. --- >> [notebook](https://colab.research.google.com/drive/1fRLsayydg2GYNZBWyJwgcxQL0h6CU0Ac?usp=sharing). >> [Weights and Biases report](https://wandb.ai/ala_/GenAI360/runs/p08s2n5f?workspace=user-ala_). --- *For more information on Intel® Accelerator Engines, visit [this resource page](https://download.intel.com/newsroom/2023/data-center-hpc/4th-Gen-Xeon-Accelerator-Fact-Sheet.pdf). Learn more about Intel® Extension for Transformers, an Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere [here](https://github.com/intel/intel-extension-for-transformers).* *Intel, the Intel logo, and Xeon are trademarks of Intel Corporation or its subsidiaries.*


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## **Introduction**
In this lesson, we’ll dive into the most popular proprietary LLMs, such as GPT-4, Claude, and Cohere LLMs. The debate between open-source or proprietary models is multifaceted when discussing aspects like customizations, development speed, control, regulation, and quality.


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## Proprietary LLMs
Proprietary models like GPT-4 and PaLM are developed and controlled by specific organizations, in contrast to open-source LLMs such as BigScience’s [Bloom](https://arxiv.org/pdf/2211.05100.pdf) and various community-driven projects, which are freely available for developers to use, modify, and distribute. As a result, these models may offer advanced features and customization options tailored to specific use cases. Organizations can fine-tune these models to meet their exact requirements, providing a competitive edge in the market. Since proprietary models are developed and controlled by specific organizations, they have complete control over the model's development, deployment, and updates. This level of control allows organizations to protect their intellectual property and maintain a competitive advantage. Organizations offering proprietary models often provide commercial support and service level agreements (SLAs) to ensure reliability and performance guarantees. This level of professional support can be crucial for some use cases. Using proprietary LLMs can be cost-effective in most use cases. LLMs are large and need several GPUs to operate at low latency (and the right competencies); therefore, they benefit from economies of scale. Let’s see now a list of popular proprietary models (as of July 2023).


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## **Cohere LLMs**
Cohere’s models are categorized into two main types: Generative and Embeddings. The Generative models, also known as command models, are trained on a large corpus of data from the Internet. They are continually developed and updated, with improvements released weekly. You can [register for a Cohere account](https://dashboard.cohere.ai/welcome/register?ref=txt.cohere.com) and get a free trial API key. There is no credit or time limit associated with a trial key; calls are rate-limited to 100 calls per minute, which is typically enough for an experimental project. Save your key in your `.env` file as follows. ```python COHERE_API_KEY="<YOUR-COHERE-API-KEY>" ``` Then, install the `cohere` package with this command. ```python pip install cohere ``` You can now generate text with Cohere as follows. ```python from dotenv import load_dotenv load_dotenv() import cohere import os co = cohere.Client(os.environ["COHERE_API_KEY"]) response = co.generate( prompt='Please briefly explain to me how Deep Learning works using at most 100 words.', max_tokens=200 ) print(response.generations[0].text) ``` ``` Deep Learning is a subfield of artificial intelligence and machine learning that is based on artificial neural networks with many layers, inspired by the structure and function of the human brain. These networks are trained on large amounts of data and algorithms to identify patterns and learn from experience, enabling them to perform complex tasks such as image and speech recognition, language translation, and decision-making. The key components of Deep Learning are neural networks with many layers, large amounts of data, and algorithms for training and optimization. Some of the applications of Deep Learning include autonomous vehicles, natural language processing, and speech recognition. ``` On the other hand, the embedding models are multilingual and can support over 109 languages. These models are designed for large enterprises whose end users are spread worldwide. Developers can also build classifiers on top of Cohere's language models to automate language-based tasks and workflows. The Cohere service provides a variety of models such as *Command* (`command`) for dialogue-like interactions, *Generation*`base`) for generative tasks, *Summarize*`summarize-xlarge`) for generating summaries, [and more](https://docs.cohere.com/docs/models).


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## **OpenAI's GPT-3.5**
**GPT-3.5** is a language model developed by OpenAI. Its turbo version, **GPT-3.5-turbo** (recommended by OpenAI over [other variants](https://platform.openai.com/docs/models/gpt-3-5)), offers a more affordable option for generating human-like text through an API accessible via OpenAI endpoints. The model is optimized for chat applications while remaining powerful on other generative tasks and can process 96 languages. GPT-3.5-turbo comes in two variants: one with a 4k tokens context length and the other with 16k tokens. The [Azure Chat Solution Accelerator](https://github.com/microsoft/azurechat), powered by Azure Open AI Service, offers enterprises a robust platform to host OpenAI chat models with enhanced moderation and safety features. This solution enables organizations to establish a dedicated chat environment within their Azure Subscription, ensuring a secure and tailored user experience. One of the key advantages is its privacy aspect, as it's deployed within your Azure tenancy, allowing for complete isolation and control over your chat services. In the first lesson, “What are Large Language Models,” we saw how to use GPT-3.5-turbo via API, so refer to that lesson for a code snippet on how to use it with Python and get an API key. Additionally, we have recently introduced the "[**LangChain & Vector Databases in Production**](https://learn.activeloop.ai/courses/langchain)" free course, aimed at assisting you in getting the most out of LLMs and enhancing their functionality. The course encompasses fundamental topics such as initiating prompts and addressing hallucination, as well as delving into advanced areas like using LangChain to give memory to LLMs and developing agents for interaction with the real world.


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## **OpenAI's GPT-4**
OpenAI's **GPT-4** is a multimodal model with an undisclosed number of parameters or training procedures. It is the latest and most powerful model published by OpenAI, and the multi-modality enables the model to process both text and image as input. It can be accessed by submitting your early access request through the OpenAI platform (as of July 2023). The two variants of the model are `gpt-4` and `gpt-4-32k` with different context lengths, 8k and 32k tokens, respectively.


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## **Anthropic’s Claude**
Anthropic, an AI safety and research company, is a significant player in the AI landscape. It has secured substantial funding and partnered with Google for cloud computing access, mirroring OpenAI's trajectory in recent years. Anthropic's flagship product, Claude 2, is an LLM with a context size of 100k tokens. Anthropic has ambitious growth plans and aims to compete with top players like OpenAI and Deepmind, working with similarly advanced models. Claude 2 is trained to be a helpful assistant in a conversational tone, similar to ChatGPT. Its beta, unfortunately, is open only to people in the US or UK (as of July 2023). If you're in the US or UK, you can sign up for free on Anthropic's website. Just click "Talk to Claude," and you'll be prompted to provide an email address. You'll be ready to go after you confirm the email address. The API is made available via the web [Console](https://console.anthropic.com/). First, read here, [Getting Access to Claude](https://docs.anthropic.com/claude/docs/getting-access-to-claude), for how to apply for access. Once you have access to the Console, you can generate API keys via your [Account Settings](https://console.anthropic.com/account/keys).


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## Google’s PaLM
Google's Pathways Language Model, or PaLM, is a next-generation artificial intelligence model optimized for various developer use cases, particularly in the realm of NLP. Its primary applications include the development of chatbots, text summarization, question-answering systems, and document search through its text embedding service. PaLM 2, the upgraded version of the model, is renowned for its ease of use and precision in following instructions. It features variants that are specifically trained for text and chat generation, as well as text embeddings, allowing for a broad range of use cases. Access to PaLM is exclusively through the PaLM API. Read the [Setup process](https://developers.generativeai.google/tutorials/setup), and please note that as of July 2023, the PaLM API is only available after being selected from a waitlist. [Google's PaLM 2 showcases](https://ai.google/static/documents/palm2techreport.pdf) significant advancements, including multilingual training for superior foreign language performance, enhanced logical reasoning, and the ability to generate and debug code. It integrates seamlessly into services like Gmail. PaLM 2 can be fine-tuned for specific domains such as cybersecurity vulnerability detection (Sec-PaLM) and medical query responses (Med-PaLM).


The Most Popular Proprietary LLMs

# The Most Popular Proprietary LLMs
## Conclusion
The choice between proprietary and open-source AI models depends on the specific needs and resources of the user or organization, and the decision should be based on a careful evaluation of all factors. The next lesson will cover the most popular open-source LLMs.


Going at Scale with LLM Training

# Going at Scale with LLM Training
## Introduction
In this lesson, we will share some tips for training LLMs at scale, focusing on the Zero Redundancy Optimizer (ZeRO) and its implementation in DeepSpeed. We explore how ZeRO optimizes memory and computational resources, its various stages of operation, and the benefits of DeepSpeed. We also touch on the Hugging Face Accelerate library. Finally, we will discuss the importance of maintaining a logbook of training runs to manage potential challenges and instabilities during the training process.


Going at Scale with LLM Training

# Going at Scale with LLM Training
## The Zero Redundancy Optimizer (ZeRO)
Training Large Language Models can be a formidable task due to the immense computational and memory requirements. However, the introduction of the [Zero Redundancy Optimizer (ZeRO)](https://arxiv.org/abs/1910.02054), implemented in DeepSpeed, has made it possible to train these models with lower hardware requirements. ZeRO is a parallelized optimizer that drastically reduces the resources required for model and data parallelism while significantly increasing the number of parameters that can be trained. ZeRO is designed to make the most of data parallelism's computational and memory resources, reducing the memory and compute requirements of each device (GPU) used for model training. It achieves this by distributing the various model training states (weights, gradients, and optimizer states) across the available devices (GPUs and CPUs) in the distributed training hardware. As long as the aggregated device memory is large enough to share the model states, ZeRO-powered data parallelism can accommodate models of any size.


Going at Scale with LLM Training

# Going at Scale with LLM Training
## The Stages of ZeRO
ZeRO operates in three main optimization stages, where the enhancements in earlier stages are available in the later stages. The stages are partitioning optimizer states, gradients, and parameters. - **Stage 1 - Optimizer State Partitioning**: Shards optimizer states across data parallel workers/GPUs. This results in a 4x memory reduction, with the same communication volume as data parallelism. For example, this stage can be used to train a 1.5 billion parameter GPT-2 model on eight V100 GPUs. - **Stage 2 - Gradient Partitioning**: Shards optimizer states and gradients across data parallel workers/GPUs. This leads to an 8x memory reduction, with the same communication volume as data parallelism. For example, this stage can be used to train a 10 billion parameter GPT-2 model on 32 V100 GPUs. - **Stage 3 - Parameter Partitioning**: Shards optimizer states, gradients, and model parameters across data parallel workers/GPUs. This results in a linear memory reduction with the data parallelism degree. ZeRO can train a trillion-parameter model on about 512 NVIDIA GPUs with all three stages. - **Stage 3 Extra - Offloading to CPU and NVMe memory**: In addition to these stages, ZeRO-3 includes the infinity offload engine to form [ZeRO-Infinity,](https://arxiv.org/abs/2104.07857) which can offload to both CPU and [NVMe memory](https://www.pogolinux.com/blog/why-leverage-nvme-ssds-on-premise-artificial-intelligence-machine-learning/) for significant memory savings. This technique allows you to train even larger models that wouldn't fit into GPU memory. It offloads optimizer states, gradients, and parameters to the CPU, allowing you to train models with billions of parameters on a single GPU.


Going at Scale with LLM Training

# Going at Scale with LLM Training
## DeepSpeed
DeepSpeed is a high-performance library for accelerating distributed deep learning training. It incorporates ZeRO and other state-of-the-art training techniques, such as distributed training, mixed precision, and checkpointing, through lightweight APIs compatible with PyTorch. DeepSpeed excels in four key areas: 1. **Scale**: DeepSpeed's ZeRO stage one provides system support to run models up to 100 billion parameters, which is 10 times larger than the current state-of-the-art large models. 2. **Speed**: DeepSpeed combines ZeRO-powered data parallelism with model parallelism to achieve up to five times higher throughput over the state-of-the-art across various hardware. 3. **Cost**: The improved throughput translates to significantly reduced training costs. For instance, to train a model with 20 billion parameters, DeepSpeed requires three times fewer resources. 4. **Usability**: Only a few lines of code changes are needed to enable a PyTorch model to use DeepSpeed and ZeRO. DeepSpeed does not require a code redesign or model refactoring, and it does not put limitations on model dimensions, batch size, or any other training parameters.


Going at Scale with LLM Training

# Going at Scale with LLM Training
## Accelerate and DeepSpeed ZeRO
The [Hugging Face Accelerate](https://huggingface.co/docs/accelerate/index) library allows you to leverage DeepSpeed's ZeRO features by making very few code changes. By using Accelerate and DeepSpeed ZeRO, we can significantly increase the maximum batch size that our hardware can handle without running into OOM errors.


Going at Scale with LLM Training

# Going at Scale with LLM Training
## Logbook of Training Runs
Despite these libraries, there are still unexpected obstacles in the training runs. This is because there may be instabilities during training that are hard to recover from, such as spikes in the loss function. For example, here’s a [logbook](https://github.com/huggingface/m4-logs/blob/master/memos/README.md) of the training of reproduction of [Flamingo (by Google Deepmind)](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model), an 80B parameters vision and language model, done by Hugging Face. In the following image, the second chart shows the loss function of the final model as the training progresses. Some of these spikes rapidly recovered to the original loss level, and some others diverged and never recovered. To stabilize and continue the training, the authors usually applied a rollback, i.e., a re-start from a checkpoint a few hundred steps prior to the spike/divergence, sometimes with a decrease in the learning rate (shown in the first chart of the image). ![Image from [https://github.com/huggingface/m4-logs/blob/master/memos/README.md](https://github.com/huggingface/m4-logs/blob/master/memos/README.md).](Going%20at%20Scale%20with%20LLM%20Training%20a64bc649f30d4bbb8451cff9d2fef3bf/training_loss_without_rollbacks.png) Image from [https://github.com/huggingface/m4-logs/blob/master/memos/README.md](https://github.com/huggingface/m4-logs/blob/master/memos/README.md). Other times, it may be possible for the model to be stuck in a local optimum, thus requiring other rollbacks. Sometimes, memory errors may require a manual inspection. Have a look at this [114-page logbook](https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf) made by Meta during the training of the OPT 175B model.


Going at Scale with LLM Training

# Going at Scale with LLM Training
## Conclusion
This lesson covered a few tips for training Large Language Models at scale, focusing on the Zero Redundancy Optimizer (ZeRO) and its implementation in DeepSpeed. We won’t cover them in more detail in the course, so if you want to deep dive into them you can read the resources linked in this page. We learned how ZeRO optimizes memory and computational resources across different stages, enabling the training of models with billions of parameters. We also explored DeepSpeed, a high-performance library that incorporates ZeRO and other state-of-the-art training techniques, providing scalability, speed, cost-effectiveness, and usability. We touched on the Hugging Face Accelerate library, which simplifies the application of DeepSpeed's ZeRO features. Lastly, we highlighted the importance of maintaining a logbook of training runs to manage potential challenges and instabilities during the training process.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## Introduction
In this lesson, we will explore the diverse applications and use cases of LLMs and generative AI across various industries. We dive into how LLMs are revolutionizing healthcare and medical research by improving diagnosis, drug discovery, and patient care. Additionally, we will uncover their impact on finance, copywriting, education, programming, and the legal industry. While LLMs offer immense potential, we will also address the risks and ethical considerations associated with their deployment in real-world scenarios, emphasizing the importance of responsible AI implementation and human oversight.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## **Healthcare and Medical Research**
Generative AI offers promising applications that can enhance patient care, drug discovery, and operational efficiency in the industry. Generative AI is being utilized for diagnosis, patient monitoring, and resource optimization. By incorporating large language models into digital pathology, accuracy for detecting diseases such as cancer has improved significantly. Furthermore, the technology aids in automating administrative tasks, which streamlines workflows and allows clinical staff to focus on more critical aspects of patient care. In the pharmaceutical industry, generative AI has become a game-changer in drug discovery. It accelerates the process and improves precision in medicine therapies, leading to shorter drug development timelines and reduced costs. This advancement paves the way for more personalized treatments and targeted therapies, ultimately benefiting patients. Medtech companies are exploring the potential of generative AI to create personalized devices for patient-centered care. Integrating generative AI into the design process optimizes medical devices for specific patient needs, improving treatment outcomes and increasing patient satisfaction. For example, [Med-PaLM](https://sites.research.google/med-palm/) is a large language model designed by Google to provide high quality answers to medical questions. It functions as a multimodal generative model capable of processing diverse biomedical data such as clinical text, medical images, and genomics, all using the same set of model parameters. Another example is [BioMedLM](https://www.mosaicml.com/blog/introducing-pubmed-gpt), a domain-specific LLM for biomedical text, made by the Stanford Center for Research on Foundation Models (CRFM) and MosaicML.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## **Finance**
LLMs like GPT have proven to be powerful tools for analyzing and processing financial data, revolutionizing how financial institutions interact with their clients and manage risks. One of the key applications of LLMs in finance is in customer interactions with digital platforms, where models can be utilized to enhance user experience through chatbots or AI-based apps. These applications enable seamless and efficient customer support, providing real-time responses to queries and concerns. The analysis of financial time-series data is another area where LLMs and generative AI have proven worthy. By leveraging large datasets of stock exchange information, these models can offer valuable insights for macroeconomic analysis and stock exchange prediction. Predicting market trends and identifying potential investment opportunities are crucial for making informed financial decisions. LLMs play a significant role in this aspect. For example, Bloomberg trained an LLM on a mix of general purpose and domain specific documents, calling it [BloombergGPT](https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/). BloombergGPT outperforms similarly-sized open models on financial NLP tasks, without sacrificing performance on general LLM benchmarks.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## Copywriting
Large Language Models and generative AI are influencing the field of copywriting by providing powerful tools for creating content. The applications of generative AI in copywriting are diverse. It can be utilized to speed up the writing process, overcome writer's block, and reduce costs by improving overall productivity. Additionally, generative AI helps maintain a consistent brand image by learning a company's language patterns and style, ensuring cohesive marketing activities. Some prominent use cases include generating website content and blog posts, crafting social media posts, creating product descriptions, and optimizing content for SEO. Generative AI can also contribute to developing content for mobile apps, tailoring it to suit different platforms and user experiences. A popular copywriting tool that uses LLMs is [Jasper](https://www.jasper.ai/), which makes it easy to generate diverse kinds of content using generative AI.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## **Education**
LLMs can help a lot in online learning and personalized tutoring. By analyzing individual learning progress, LLMs offer personal feedback, adaptive testing, and personalized interventions. These models can address the challenges of teacher shortages by providing scalable solutions such as virtual teachers or supporting para-teachers with advanced tools. They empower educators to become mentors and guides, offering personalized support and interactive learning experiences. AI can analyze individual student performance and personalize the learning experience. For example, an application of LLMs for education is [Khanmigo](https://support.khanacademy.org/hc/en-us/articles/13888935335309-How-do-the-Large-Language-Models-powering-Khanmigo-work-) of [Khan Academy](https://www.khanacademy.org/). LLMs serve as virtual tutors, offering explanations and examples for better subject understanding. LLMs aid language learning, generating sentences for grammar and vocabulary practice.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## Programming
LLMs and generative AI can significantly help in coding by providing powerful tools for developers. LLMs like GPT-4 and its predecessors can generate code snippets based on natural language prompts, significantly enhancing programmers' efficiency. These models are trained on vast corpora of code samples and can understand context, enabling them to generate more relevant and accurate code over time. The applications of LLMs for coding are diverse. They can assist in code completion by suggesting code snippets as developers type, saving time and reducing errors. Additionally, LLMs are employed for unit test generation, automating the creation of test cases. This not only enhances code quality but also assists in software maintenance. However, the use of generative AI in coding also presents challenges. While it can boost productivity, developers must exercise caution and review the generated code, as it may contain errors or security vulnerabilities. Furthermore, the potential for model biases and "hallucinations" (fabricating incorrect information) necessitates careful scrutiny. A popular product using LLMs for programming is [GitHub Copilot](https://github.com/features/copilot), which is trained on billions of lines of code. Copilot can turn natural language prompts into coding suggestions across dozens of languages.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## **Legal Industry**
LLMs and generative AI have emerged as powerful tools for the legal industry, offering a range of applications and use cases. These models can be designed to handle the complexities of legal language, interpretations, and the dynamic nature of law. LLMs have the potential to assist legal practitioners in various tasks, such as providing legal advice, understanding complex legal documents, and analyzing court case texts. One key application is reducing hallucinations, a common challenge with early legal LLMs. These models can produce more accurate and reliable results by integrating domain-specific knowledge through reference modules and reliable data from knowledge bases. They can also identify legal feature words within users' input and quickly analyze legal situations.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## **Risks and Ethical Considerations of Using LLMs in the Real World**
As we learned in previous lessons, deploying Large Language Models (LLMs) in real-world applications poses certain risks and ethical concerns. One significant risk is "**hallucinations**," where the LLM generates false but plausible-sounding information. This could lead to serious consequences, particularly in critical domains like healthcare, finance, and law. Another concern is "**bias,**" as LLMs can inadvertently perpetuate societal biases present in their training data. This could result in unfair treatment in areas such as healthcare and finance. Addressing bias requires rigorous data evaluation, inclusivity efforts, and continuous improvement in fairness. **Data privacy and security** are crucial as LLMs might memorize sensitive information, potentially leading to privacy breaches. Organizations must implement measures like data anonymization and strict access controls. Additionally, the impact on employment should be considered, balancing automation and human involvement to preserve human expertise. Dependence on LLMs without human judgment can be risky, necessitating a responsible approach that combines AI benefits with human oversight.


Applications and Use-Cases of LLMs

# Applications and Use-Cases of LLMs
## Conclusion
This lesson explored the wide-ranging applications and use cases of LLMs and generative AI across diverse industries. From healthcare and medical research to finance, copywriting, education, programming, and the legal industry, LLMs are powerful tools with immense potential. However, alongside their benefits, we must be mindful of the risks and ethical considerations associated with their deployment. Addressing issues like hallucinations, bias, data privacy, and the impact on human employment is crucial for responsible AI implementation. In the next module, we will study Transformer architectures, the backbone of LLMs, to gain insights into their working principles and understand how these models process and generate language with great accuracy and efficiency.


When to Train an LLM from Scratch

# When to Train an LLM from Scratch
## Introduction
The increasing popularity of LLMs has led businesses to integrate them for task handling and employee productivity enhancement. There are several ways to use LLMs in daily activities, such as incorporating proprietary models via APIs, deploying pre-trained open-source options, or developing one's own language model. Of course, the trade-offs are between quality, costs, and ease of use. In this lesson, we will discuss different approaches and what might be the best solution for your use case.


When to Train an LLM from Scratch

# When to Train an LLM from Scratch
## Few-Shot (In-Context) Learning
Up to 2020, language models were already good at picking up patterns from the data. However, teaching them new knowledge from a different domain was difficult. The only solution was to finetune them by adjusting the weights. **What are the characteristics that set LLMs apart from the previous language models?** Few-shot learning (also called In-Context learning) enables the LLMs to learn from the examples provided to them. For instance, it is possible to show a couple of examples of JSON-formatted responses to receive the model’s output in JSON format. It means that the models can learn from examples and follow directions without changing weights or repeating the training process. There are multiple use cases where this approach could be the best option. The model can adapt to a writing style, set specific formatting guidelines, or provide additional context for answering questions. LLMs are able to answer questions using external knowledge bases through in-context learning. Let’s think about how we could create a Q&A chatbot leveraging an LLM. The LLM has a cut-off training date, so it can’t access the information or events after that date. Also, they tend to hallucinate, which refers to generating non-factual responses based on their limited knowledge. As a solution, it is possible to provide additional context to the LLM through the Internet (e.g., Google search) or retrieve it from a database and include it in the prompt so that the model can leverage it to generate the correct response. It is like taking an open-book exam! The beauty of this approach is that the model does not need domain-specific knowledge. Instead, it can extract information or patterns from the provided context. Creating applications, such as chatbots, becomes more accessible and faster. Whether you are utilizing proprietary APIs or open-source models, this approach offers a budget-friendly solution for many use cases.


When to Train an LLM from Scratch

# Fine-Tuning
The fine-tuning method proves valuable when adapting the model to a more complex use case. This technique can improve model understanding by providing more examples and adjusting weights based on errors, for tasks like classification or summarization. There are different approaches to doing this. We could either adjust the weights with a small learning rate to minimally affect the model’s current abilities, or a more recent technique is to freeze the network and introduce new weights for fine-tuning. The latter approach (like LoRA) is a great alternative for fine-tuning models with hundreds of billions of parameters since we will deal with a much smaller number of parameters. (~100x less) The fine-tuning approach is an excellent option for creating a model with task-specific knowledge and building on top of the available powerful LLMs. However, before considering this option, it is essential to acknowledge the associated costs and required resource implications.


When to Train an LLM from Scratch

# Training
Lastly, let's talk about training your own model from scratch! Among the approaches mentioned earlier, this option stands out as the most demanding and challenging. Of course, the scale of requirements depends on the model size. However, acquiring several millions of data points, such as web pages, books, and articles, not to mention the task-specific documents held by your organization (if you want to train a domain-specific LLM), is essential. Furthermore, completing the training process could cost upward of several hundreds of thousands of dollars. The training costs of these models are rarely revealed by the organizations that publish them. Nevertheless, considering the hardware utilized, speculations have estimated the training expenses for the GPT-3 model to be approximately $4.6 million. However, the more critical aspect of training from scratch is curating the dataset. While the intention is to train a domain-specific model, the training loop that processes vast quantities of general documents, such as web pages, articles, and books, empowers LLMs' language understanding capabilities. Therefore, to create a model that excels in a specific domain, it is essential to have a sizable dataset comprising top-quality samples from that particular domain. An example of this approach is the [BloombergGPT](https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/) 50B model, which is specifically designed for the finance industry. They used a dataset of 708 billion tokens for training, consisting of 51.2% (363 billion tokens) domain-specific resources and the rest general resources. Training a model from scratch demands substantial resources, including hardware and dataset resources, and expertise within the organization to train and maintain these models.


When to Train an LLM from Scratch

# Training
## Main Takeaways
- **Few-Shot Learning**: The LLMs are able to learn from the examples given to them, allowing them to handle more complicated tasks without the need for training or fine-tuning. This method is significantly less expensive than other options, as it only requires the cost of adding examples to each prompt. If your task can be solved just with few-shot learning, then it’s always the most efficient approach. - **Fine-Tuning**: If few-shot learning is not effective for your task, an alternative method is fine-tuning. This involves using some data points to create a task-specific model. Although finetuning can be challenging when acquiring new knowledge, it is more effective in adapting to different styles, and tones, or incorporating new vocabulary. - **Training From Scratch**: If fine-tuning is not effective, consider training a model from scratch with domain-specific data. However, this requires significant resources, such as cost, dataset availability, and expertise.


When to Train an LLM from Scratch

# Training
## Conclusion
We have explored various methods to harness the capabilities of large language models within your organization and highlighted the advantages and disadvantages of each approach. Picking the best practice depends on your organization’s use case and the resources at hand. This course aims to equip you with the necessary knowledge to make informed decisions about which approach best suits your needs. It will also guide on maximizing the benefits of large language models and mastering the process involved.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## Introduction
In this lesson, we will view the next challenges in large language model research, covering various facets such as model performance, data and training, language and tokenization, hardware and infrastructure, usability and application, and learning and preferences. We will explore pressing issues such as mitigating hallucinations, optimizing context, managing massive datasets, improving tokenization, and developing alternatives to GPUs. We will also discuss the need to make agents usable, detect LLM-generated text, and improve learning from human preference.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## ****Model Performance and Efficiency****
- **Mitigating and Measuring Hallucinations**: One of the significant challenges in LLM research is hallucinations. This phenomenon occurs when an AI model generates information that isn't based on the input data, essentially making things up. While this can be beneficial for creative applications, it is generally considered a drawback for most use cases. The challenge lies in reducing and developing metrics to measure these hallucinations accurately. - **Optimizing Context Length and Construction**: Context plays a crucial role in the performance of LLMs. The challenge here is to optimize the context length and how it is constructed. This is particularly important for applications like [Retrieval Augmented Generation (RAG)](https://huggingface.co/docs/transformers/model_doc/rag), where the model's response quality depends on the amount and efficiency of the context it can use. - **Making LLMs Faster and Cheaper**: With the advent of models like GPT-3.5, concerns about latency and cost have become more prominent. The challenge lies in developing models that offer similar performance but with a smaller memory footprint and lower costs. Faster inference is especially important for real-time applications like online customer service assistants. - **Designing New Model Architectures**: Transformer architecture has been dominant in the field since 2017. However, the need for a new model architecture that can outperform the Transformer is becoming increasingly apparent. The challenge is to develop an architecture that performs well on current hardware and scales to meet modern requirements. - **Addressing High Inference Latency**: LLMs often exhibit high inference latencies due to low parallelizability and large memory footprints. The task at hand is to develop models and techniques that can reduce this latency, making LLMs more efficient and practical for real-time applications. - **Overcoming Tasks Not Solvable By Scale**: The rapid advancements in LLM capabilities have led to astonishing improvements in performance. However, some tasks seem resistant to further scaling of data or model sizes. The existence of such tasks is speculative, but their potential presence poses a significant challenge. The research community needs to identify these tasks and devise strategies to overcome them, pushing the boundaries of what LLMs can achieve. Read about the [Inverse Scaling Prize](https://www.lesswrong.com/posts/DARiTSTx5xDLQGrrz/inverse-scaling-prize-second-round-winners) competition to know more about this.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## ****Data and Training****
- **Incorporating Other Data Modalities**: The ability to incorporate other data modalities into LLMs is another significant research direction. Multimodality, the ability to understand and process different types of data, can enhance the performance of LLMs and extend their applicability to various industries. - **Understanding and Managing Huge Datasets**: The sheer size of modern pre-training datasets makes it nearly impossible for individuals to read or conduct quality assessments on all the documents. This lack of clarity about the data on which the model has been trained poses a significant challenge. Researchers need to devise strategies to comprehend these vast datasets better and ensure the quality of the data used for training. - **Reducing High Pre-Training Costs**: Training a single LLM can require substantial computational resources, translating into high costs and significant energy consumption. The challenge here is to find ways to reduce these pre-training costs without compromising the performance of the model. This could involve optimizing the training process or developing more efficient model architectures.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## ****Language and Tokenization****
- **Building LLMs for Non-English Languages**: There is a pressing need to develop LLMs for non-English languages. This complex challenge involves dealing with low-resource languages and ensuring that the models are practical and efficient. - **Overcoming Tokenizer-Reliance**: Tokenization, the process of breaking down text into smaller units, is crucial for feeding data into the model. However, this necessity comes with drawbacks, such as computational overhead, language dependence, handling of novel words, fixed vocabulary size, information loss, and low human interpretability. The challenge lies in developing more effective tokenization methods or alternatives that can mitigate these issues. - **Improving Tokenization for Multilingual Settings**: Tokenization schemes that work well in a multilingual setting, particularly with non-space-separated languages such as Chinese or Japanese, remain challenging. The challenge is to improve these schemes to ensure fair and efficient tokenization across all languages.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## ****Hardware and Infrastructure****
- **Developing Alternatives to GPUs**: GPUs have been the primary hardware for deep learning for nearly a decade. However, there is a growing need for alternatives that can offer better performance or efficiency. This includes exploring technologies like quantum computing and photonic chips. There’s also currently a problem of availability of GPUs in the global market, therefore having alternatives would make this problem more manageable.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## ****Usability and Application****
- **Making Agents Usable**: Agents are LLMs that can perform actions like browsing the internet or sending emails. The challenge here is to make these agents reliable and performant enough to be trusted with these tasks. Examples of agents frameworks are [LangChain](https://python.langchain.com/docs/get_started/introduction) and [LlamaIndex](https://www.llamaindex.ai/). - **Detecting LLM-generated Text**: As LLMs become more sophisticated, distinguishing between human-written and LLM-generated text becomes increasingly challenging. This detection is crucial for various reasons, such as preventing the spread of misinformation, plagiarism, impersonation, automated scams, and the inclusion of inferior generated text in future models' training data. The challenge lies in developing robust detection mechanisms that can keep up with the improving fluency of LLMs.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## ****Learning and Preferences****
- **Improving Learning from Human Preference**: Reinforcement Learning from Human Preference (RLHF) is a promising approach but has its challenges. These include defining and mathematically representing human preferences and dealing with the diversity of human preferences.


Next Challenges in LLM Research

# Next Challenges in LLM Research
## Conclusion
In this lesson, we explored the challenges in large language models research. We've examined the need for improved model performance and efficiency, including mitigating hallucinations, optimizing context, and designing new model architectures. We've also discussed the complexities of managing vast datasets and the importance of incorporating other data modalities. We highlighted the necessity for better tokenization methods, especially for non-English and non-space-separated languages. We have also underscored the urgency of developing alternatives to GPUs and the need to make LLM agents more reliable. Lastly, we've touched upon the challenge of detecting LLM-generated text and the intricacies of learning from human preference. Each of these challenges presents an exciting opportunity for researchers to push the boundaries of what LLMs can achieve, making them more efficient, inclusive, and beneficial for a wide array of applications.


Deep Dive into RLHF

# Deep Dive into RLHF
## Introduction
In this lesson, we will dive deeper into Reinforcement Learning from Human Feedback (RLHF), a method that combines human feedback and reinforcement learning to enhance the alignment and efficiency of Large Language Models. We explore the RLHF training process, compare it with Supervised Fine-Tuning (SFT), and discuss its alternatives, such as Direct Preference Optimization (DPO) and Reinforced Self-Training (ReST). By the end of this lesson, you'll have a comprehensive understanding of how RLHF and its alternatives are used to improve the performance and safety of LLMs.


Deep Dive into RLHF

# Deep Dive into RLHF
## Understanding RLHF
[Reinforcement Learning from Human Feedback (RLHF)](https://arxiv.org/abs/1909.08593) is a method that integrates human feedback and reinforcement learning into LLMs, enhancing their alignment with human objectives and improving their efficiency. RLHF has shown significant promise in making LLMs safer and more helpful. It was used for the first time for creating [InstructGPT](https://openai.com/research/instruction-following), a finetuned version of GPT3 for following instructions, and it’s used nowadays in the last OpenAI models ChatGPT (GPT-3.5-turbo) and GPT-4. RLHF leverages human-curated rankings that act as a signal to the model, directing it to favor specific outputs over others, thereby encouraging the production of more reliable, secure responses that align with human expectations. All of this is done with the help of a reinforcement learning algorithm, namely [PPO](https://openai.com/research/openai-baselines-ppo), that optimizes the underlying LLM model, leveraging the human-curated rankings.


Deep Dive into RLHF

# Deep Dive into RLHF
## RLHF Training Process
RLHF can be useful in guiding LLMs to generate appropriate texts by treating text generation as a reinforcement learning problem. In this approach, the language model serves as the RL agent, the possible language outputs represent the action space, and the reward is based on how well the LLM's response aligns with the context of the application and the user's intent. RLHF must be done on an already pretrained LLM. A language model must be trained in advance on a large corpus of text data collected from the internet. The RLHF training process can then be broken down into the following steps. - **(Optional) Finetune the LLM by following instructions**: This is an optional step, but some sources recommend fine-tuning raw LLM in advance by following instructions, using a specialized dataset for it. This step should make the following RL finetuning of RLHF converge faster. - **RLHF dataset creation**: The LLM is used to generate a lot of text completions from a set of instructions. For each instruction, we collect multiple completions from the model. - **Collecting human feedback**: Human labelers then rank the generated completions to the same instruction from best to worst. Humans can be asked to rank the completions, keeping into account several aspects, such as completeness, relevancy, accuracy, toxicity, bias, etc. It’s possible to convert these ranks into scores that we can assign to the text completions in our dataset, where a high score means that the completion is good. - **Training a Reward Model**: The RLHF dataset is used to train a reward model, which means a model that, when provided with an instruction and a text completion, assigns a score to the completion. In this context, a high score indicates that the completion is good. The reward model does a very similar job to what human labelers did on the dataset. The reward model is expected to learn, from the RLHF dataset, how to assign scores according to all the aspects taken into account during the labeling process (completeness, relevancy, accuracy, toxicity, bias, etc.). - **Fine-tuning the Language Model with Reinforcement Learning and the Reward Model**: Starting from a random instruction, our pretrained LLM generates multiple completions. These completions are then assigned scores by the reward model, and these scores are utilized by a reinforcement learning algorithm (PPO) to update the parameters of the LLM. This process aims to make the LLM more likely to produce completions with higher scores. To prevent the LLM from forgetting helpful information during fine-tuning, the RLHF fine-tuning process also aims to maintain a small Kullback-Leibler (KL) divergence between the fine-tuned LLM and the original LLM. This ensures that the token distribution predicted by it remains relatively consistent. After repeating this process for several iterations, we will have our final, finalized LLM. ![Visual illustration of RLHF. Image from [https://openai.com/research/instruction-following](https://openai.com/research/instruction-following).](Deep%20Dive%20into%20RLHF%209551cbead6e041258a32eb7deac5989a/Screenshot_2023-08-28_at_16.05.19.png) Visual illustration of RLHF. Image from [https://openai.com/research/instruction-following](https://openai.com/research/instruction-following).


Deep Dive into RLHF

# Deep Dive into RLHF
## RLHF vs SFT
As seen in the previous lessons, aligning LLM to follow instructions with human values is possible by doing simple SFT (with or without LoRA) with a high-quality dataset ([see the LIMA paper](https://arxiv.org/abs/2305.11206)). So, what’s the tradeoff between RLHF and SFT? In reality, it's still an open question. Empirically, it seems that RLHF can better teach the "human alignment" aspects of its dataset if it's sufficiently large and of high quality. However, in contrast, it's more expensive and time-consuming. Reinforcement learning, in this context, is still quite unstable, meaning that the results are very sensitive to the initial model parameters and training hyperparameters. It often falls into local optima, and the loss diverges multiple times, necessitating multiple restarts. This makes it less straightforward than plain SFT with LoRA.


Deep Dive into RLHF

# Deep Dive into RLHF
## Alternatives to RLHF
Over time, several alternatives to RLFH have been researched. Here are the most popular of them. ### Direct Preference Optimization [Direct Preference Optimization (DPO)](https://arxiv.org/pdf/2305.18290.pdf) is a novel method for finetuning LLMs as an alternative to RLHF. Unlike RLHF, which requires complex reward functions and careful balance to ensure sensible text generation, DPO simplifies the process by directly optimizing the language model using a binary cross-entropy loss. It bypasses the need for a reward model and RL-based optimization. Instead, it directly optimizes the language model on preference data. This is accomplished through an analytical mapping from the reward function to the optimal RL policy. It involves directly transforming the RL loss, which typically involves the reward and reference models, into a loss over the reference model. As a result, DPO potentially simplifies the fine-tuning process of LLMs by eliminating the need for complex RL techniques or a reward model. ![DPO optimizes for human preferences while avoiding reinforcement learning. Existing methods for fine-tuning language models with human feedback first fit a reward model to a dataset of prompts and human preferences over pairs of responses. Then, RL is used to find a policy that maximizes the learned reward. In contrast, DPO directly optimizes the policy to best satisfy the preferences with a simple classification objective, without an explicit reward function or RL. Image from [https://arxiv.org/pdf/2305.18290.pdf](https://arxiv.org/pdf/2305.18290.pdf).](Deep%20Dive%20into%20RLHF%209551cbead6e041258a32eb7deac5989a/Screenshot_2023-08-28_at_16.13.03.png) DPO optimizes for human preferences while avoiding reinforcement learning. Existing methods for fine-tuning language models with human feedback first fit a reward model to a dataset of prompts and human preferences over pairs of responses. Then, RL is used to find a policy that maximizes the learned reward. In contrast, DPO directly optimizes the policy to best satisfy the preferences with a simple classification objective, without an explicit reward function or RL. Image from [https://arxiv.org/pdf/2305.18290.pdf](https://arxiv.org/pdf/2305.18290.pdf). ### Reinforced Self-Training [Google DeepMind's Reinforced Self-Training (ReST)](https://arxiv.org/pdf/2308.08998.pdf) is a more cost-effective alternative to Reinforcement Learning from Human Feedback. The ReST algorithm operates in a cyclical manner, involving two main steps that are repeated iteratively. 1. The first step, referred to as the 'Grow' step, involves the use of an LLM to generate multiple output predictions for each context. These predictions are then used to augment a training dataset. 2. Following this, the 'Improve' step comes into play. In this phase, the augmented dataset is ranked and filtered using a reward model that has been trained based on human preferences. Subsequently, the LLM is fine-tuned on this filtered dataset using an offline reinforcement learning objective. The fine-tuned LLM is then used in the subsequent Grow step. ![ReST method. During the Grow step, a policy generates a dataset. The filtered dataset is used to fine-tune the policy in the Improve step. Both steps are repeated. The improvement step is repeated more frequently to amortize the dataset creation cost. Image from [https://arxiv.org/pdf/2308.08998.pdf](https://arxiv.org/pdf/2308.08998.pdf).](Deep%20Dive%20into%20RLHF%209551cbead6e041258a32eb7deac5989a/new_image.png) ReST method. During the Grow step, a policy generates a dataset. The filtered dataset is used to fine-tune the policy in the Improve step. Both steps are repeated. The improvement step is repeated more frequently to amortize the dataset creation cost. Image from [https://arxiv.org/pdf/2308.08998.pdf](https://arxiv.org/pdf/2308.08998.pdf). The ReST methodology offers several advantages over RLHF. - It significantly reduces the computational load compared to online reinforcement learning. This is achieved by leveraging the output of the Grow step across multiple Improve steps. - The quality of the policy is not limited by the quality of the original dataset, as is the case with offline reinforcement learning. This is because new training data is sampled from an improved policy during the Grow step. - Decoupling the Grow and Improve steps allows for easy inspection of data


Deep Dive into RLHF

# Deep Dive into RLHF
## Alternatives to RLHF
quality and potential diagnosis of alignment issues, such as reward hacking. - The ReST approach is straightforward and stable and only requires tuning a small number of hyperparameters, making it a user-friendly and efficient tool in the machine learning toolkit. ### Reinforcement Learning from AI Feedback (RLAIF) Another innovative alternative to RLHF is [Reinforcement Learning from AI Feedback (RLAIF)](https://arxiv.org/abs/2212.08073). Developed by Anthropic, RLAIF aims to address some of the limitations of RLHF, particularly concerning the subjectivity and scalability of human feedback. In RLAIF, instead of relying on human feedback, an AI Feedback Model is used to provide feedback for training the AI assistant. This Feedback Model is guided by a constitution provided by humans, outlining the essential principles for the model's judgment. This approach allows for a more objective and scalable supervision technique, as it is not dependent on a small pool of human preferences. The RLAIF process begins with the creation of a dataset of ranked preferences generated automatically by the AI Feedback Model. This dataset is then used to train a Reward Model similar to RLHF. The Reward Model serves as the reward signal in a reinforcement learning schema for an LLM. ![A diagram depicting RLAIF (top) vs. RLHF (bottom). Image from [https://arxiv.org/pdf/2309.00267.pdf](https://arxiv.org/pdf/2309.00267.pdf).](Deep%20Dive%20into%20RLHF%209551cbead6e041258a32eb7deac5989a/Screenshot_2023-09-04_at_11.53.01.png) A diagram depicting RLAIF (top) vs. RLHF (bottom). Image from [https://arxiv.org/pdf/2309.00267.pdf](https://arxiv.org/pdf/2309.00267.pdf). RLAIF offers several advantages over RLHF. Firstly, it maintains the helpfulness of RLHF models while making improvements in terms of harmlessness. Secondly, it reduces subjectivity as the AI assistant's behavior is not solely dependent on a small pool of humans and their particular preferences. Lastly, RLAIF is significantly more scalable as a supervision technique, making it a promising alternative for the future development of safer and more efficient LLMs. [A recent paper from Google](https://arxiv.org/pdf/2309.00267.pdf) did more experiments with RLAIF and found that humans prefer both RLAIF and RLHF to standard SFT at almost equal rates, indicating that they could be alternatives.


Deep Dive into RLHF

# Deep Dive into RLHF
## Conclusion
This lesson provided a more in-depth exploration of Reinforcement Learning from Human Feedback, a method that combines human feedback and reinforcement learning to enhance the performance and safety of Large Language Models. We covered the RLHF training process, highlighting its steps and how it leverages human-curated rankings and reinforcement learning to finetune the LLM. We also compared RLHF with Supervised Fine-Tuning (SFT), discussing the trade-offs between the two. Furthermore, we explored alternatives to RLHF, such as Direct Preference Optimization (DPO) and Reinforced Self-Training (ReST), which offer different approaches to fine-tuning LLMs. As we continue to refine these techniques, we move closer to our goal of creating LLMs that are more aligned with human values, efficient, and safer to use.


# Understanding Transformers and GPTs Introduction
## Understanding Transformers and GPTs
Goals: Equip students with foundational theoretical knowledge of transformers and GPTs, ensuring a robust understanding beneficial for effective LLM training and utilization. This section comprehensively examines the inner workings and components surrounding Transformers and GPT. Participants begin with a detailed study of the Transformer architecture, understanding its foundational concepts and essential parts. The progression covers evaluations of LLMs, their control mechanisms, and the nuances of prompting, pretraining, and finetuning. Each lesson is designed to impart intricate details, practical knowledge, and a tiered understanding of these technologies. - **Understanding Transformers**: This lesson provides an in-depth look at Transformers, breaking down their complex components and the network's essential mechanics. We begin by examining the paper "Attention is all you need.” We conclude by highlighting the use of these components in Hugging Face's transformers library. - **Transformers Architectures**: This chapter is a concise guide to Transformer architectures. We will first dissect the encoder-decoder framework, which is pivotal for sequence-to-sequence tasks. Next, we provide a high-level overview of the GPT model, known for its language generation capabilities. We also spotlight BERT, emphasizing its significance in understanding the context within textual data. - **Deep Dive on the GPT architecture**: This section explores the GPT architecture. We shed light on the structural specifics, the objective function, and the principles of causal modeling. This technical session is designed for individuals seeking an in-depth understanding of the intricate details and mathematical foundations of GPT. - **Evaluating LLM Performance**: This lesson explores the nuances of evaluating Large Language Model performance. We differentiate between objective functions and metrics and transition into perplexity, BLEU, and ROUGE metrics. We also provide an overview of popular benchmarks in the domain. - **Controlling LLM Outputs**: This lesson delves into decoding techniques like Greedy and Beam Search, followed by concepts such as Temperature and the use of stop sequences. We will also discuss the importance of these methods, with references to frameworks like ReAct. It also presents concepts like Frequency and Presence Penalties. - **Prompting and few-shot prompting**: This lesson provides an overview of how carefully crafted prompts can guide LLMs in tasks like answering questions and generating text. We will progress from zero-shot prompting, where LLMs operate without specific examples, to in-context and few-shot prompting, teaching the model to manage intricate tasks with sparse training data. - **Pretraining and Finetuning**: This module examines the foundational concepts of pretraining and finetuning in the context of Large Language Models. In subsequent chapters, we will discern the differences between pretraining, finetuning, and instruction tuning, setting the stage for deeper dives. While the lesson touches upon various types of instruction tuning, detailed exploration of specific methods like SFT and RLHF will be reserved for later sessions, ensuring a progressive understanding of the topic. After navigating the diverse terrain of Transformers and LLMs, participants now deeply understand significant architectures like GPT and BERT. The sessions shed light on model evaluation metrics, advanced control techniques for optimal outputs, and the roles of pretraining and finetuning. The upcoming module dives into the complexities of deciding when to train an LLM from scratch, the operational necessities of LLMs, and the sequential steps crucial for the training process.


Pretraining and Fine-Tuning of LLMs

# Pretraining and Fine-Tuning of LLMs
## Introduction
This lesson will explore how pretrained LLMs learn from vast amounts of text, becoming great at language tasks. Then, we'll discover the power of finetuning, a process that molds these models into specialized experts, enabling them to tackle complex tasks. We'll also cover instruction finetuning, where we guide the models with explicit instructions, making them versatile and responsive to our needs. This lesson introduces several fine-tuning techniques we will use to finetune models later in the course.


Pretraining and Fine-Tuning of LLMs

# Pretraining and Fine-Tuning of LLMs
## **Pretraining LLMs**
Pretrained LLMs have catalyzed a paradigm shift in AI. These models are trained on massive text corpora sourced from the Internet, honing their linguistic knowledge through the prediction of the following words within sentences. By training on billions of sentences, these models acquire an excellent grasp of grammar, context, and semantics, enabling them to capture the nuances of language effectively. Aside from being good at generating text, pretrained LLMs are also good at other tasks, as was found in 2020 with the GPT3 paper “[Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165).” The paper showed that big enough LLMs are “few-shot learners”; that is, they are able to perform other tasks aside from text generation with the help of just a few examples of that task (hence the name “few-shot learners”). With those examples, the LLM is able to understand the logic behind what the user wants. This was a huge step forward in the field, where each NLP task had very different models for each task. Now, a single model can do several of them and do them well.


Pretraining and Fine-Tuning of LLMs

# Pretraining and Fine-Tuning of LLMs
## **The Power of Finetuning**
Finetuning complements pretraining for specialized tasks. While pretrained LLMs are undeniably impressive, their true potential is unlocked through finetuning. Although pretrained models possess a deep understanding of language, they require further adaptation to excel in complex tasks. For example, if the task is to answer questions about medical texts, the model would be finetuned on a dataset of medical question-answer pairs. Finetuning helps these models become specialized. Finetuning exposes pretrained models to task-specific datasets, enabling them to recalibrate internal parameters and representations to align with the intended task. This adaptation enhances their ability to handle domain-specific challenges effectively. The necessity for finetuning arises from the inherent non-specificity of pretrained models. While they possess a wide-ranging grasp of language, they lack task-specific context. For instance, finetuning is essential when tackling sentiment analysis of financial news. In the early days of GPT3 in 2020 and 2021, finetuning also allowed an LLM to be tuned for a specific task without the need for multiple few-shot examples in the prompt.


Pretraining and Fine-Tuning of LLMs

# Pretraining and Fine-Tuning of LLMs
## **Instruction Finetuning: Making General-Purpose Assistants out of LLMs**
Instruction finetuning adds precise control over model behavior, making it a general-purpose assistant. The goal of instruction finetuning is to obtain an LLM that interprets prompts as instructions instead of text. It’s just a special type of finetuning. For example, consider the following prompt. > What is the capital of France? > An LLM with instruction finetuning would likely interpret the prompt as an instruction, giving the following answer. > Paris. > However, a plain LLM without instruction finetuning could think that we are writing a list of exercises for our geography students, therefore merely continuing the text with a new question. > What is the capital of Italy? > Instruction finetuning takes things up a notch. Imagine giving precise instructions to our model: "Analyze the sentiment of this text and tell us if it's positive.” It's like coaching the model to paint exactly what you envision. With instruction finetuning, we provide explicit guidance, shaping the model's behavior to match our intentions. Instruction tuning offers several advantages. It trains models on a collection of tasks described via instructions, granting LLMs the capacity to generalize to new tasks prompted by additional instructions. This sidesteps the need for vast amounts of task-specific data and instead uses textual instructions to guide learning. While traditional finetuning acquaints models with task-specific data, instruction finetuning adds an extra layer by incorporating explicit instructions to guide model behavior. This approach empowers developers to shape desired outputs, encourage specific behaviors, and steer model responses.


Pretraining and Fine-Tuning of LLMs

# Pretraining and Fine-Tuning of LLMs
## Finetuning Techniques
There are several finetuning methods. We’ll learn more about them later in the course. There are multiple methods in fine-tuning with a focus on the number of parameters, such as: - **Full Finetuning:** This method is based on adjusting all the parameters in the pretrained LLM models in order to adapt to a specific task. However, this method is relatively resource-intensive, requiring extensive computational power. - **Low-Rank Adaptation (LoRA):** LoRA aims to adapt LLMs to specific tasks and datasets while simultaneously reducing computational resources and costs. By applying low-rank approximations to the downstream layers of LLMs, LoRA significantly reduces the number of parameters to be trained, thereby lowering the GPU memory requirements and training costs. Multiple methods are focusing on the learning algorithm used for finetuning, such as: - **Supervised Finetuning (SFT):** SFT involves doing standard supervised finetuning with a pretrained LLM on a small amount of demonstration data. - **Reinforcement Learning from Human Feedback (RLHF):** RLHF is a training methodology where models are trained to follow human feedback over multiple iterations. Later in this course, we’ll see how to finetune a model using SFT and RLHF, both using LoRA.


Pretraining and Fine-Tuning of LLMs

# Pretraining and Fine-Tuning of LLMs
## Conclusion
In this lesson, we covered the pretraining and finetuning of LLMs. Pretraining equips LLMs with a profound grasp of language by immersing them in vast text corpora. Finetuning then bridges the gap between general understanding and specialized knowledge, allowing LLMs to perform well in specialized domains. Instruction finetuning makes LLMs become versatile assistants, enabling precise control over their behavior through explicit guidance. From full finetuning to the resource-efficient Low-Rank Adaptation (LoRA) and from Supervised Finetuning (SFT) to Reinforcement Learning from Human Feedback (RLHF), we also learned about the most popular finetuning techniques. In the next module, we’ll do some hands-on exercises up to launch a pretraining of an LLM on the cloud.


# The Evolution of Language Modeling up to LLMs
## Introduction
In this lesson, we’ll see the most popular models used for language modeling, starting from statistical ones up to the first Large Language Models (LLMs). This lesson is meant to be more like a narrative on the evolution of the models rather than a technical explanation. Therefore, don’t worry if you can’t understand every model in detail.


# The Evolution of Language Modeling up to LLMs
## The Evolution of Language Modeling
The evolution of NLP models has been a remarkable journey marked by continuous innovation and improvement. It began with the Bag of Words model in 1954, which simply counted word occurrences in documents. This was followed by TF-IDF in 1972, which adjusted these scores based on the rarity or commonality of words. The advent of Word2Vec in 2013 marked a significant leap forward, introducing the concept of word embeddings that captured semantic relationships between words. This was then further enhanced by Recurrent Neural Networks (RNNs), which could learn sequence patterns and handle documents of any length. The introduction of the Transformer architecture in 2017 revolutionized the field, with its attention mechanism allowing the model to focus on the most relevant parts of the input when generating output. This was the foundation for BERT in 2018, which used bidirectional Transformers to achieve impressive results in traditional NLP tasks. The subsequent years saw a flurry of advancements, with models like RoBERTa, XLM, ALBERT, and ELECTRA each introducing their own improvements and optimizations.


# The Evolution of Language Modeling up to LLMs
## Model’s Timeline
- **[1954]** [Bag of Words (BOW)](https://en.wikipedia.org/wiki/Bag-of-words_model) BOW is a simple model that counts word occurrences in documents, using these counts as features. It was a basic yet effective way to analyze text. However, it did not account for word order or context. - **[1972]** [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) TF-IDF enhanced BOW by giving more weight to rare words and less to common ones. This improved the model's ability to discern document relevance. However, it still did not account for word context. - **[2013]** [Word2Vec](https://arxiv.org/abs/1301.3781) Word2Vec introduced word embeddings, high-dimensional vectors that capture semantic relationships. These embeddings were learned by a neural network trained on a large corpus of text. This model marked a significant advancement in capturing semantic meaning in text. - **[2014]** [RNNs in Encoder-Decoder architectures](https://en.wikipedia.org/wiki/Recurrent_neural_network) RNNs (Recurrent Neural Networks) compute document embeddings, leveraging word context in sentences, which was not possible with word embeddings alone. Later evolved with **[LSTM](http://www.bioinf.jku.at/publications/older/2604.pdf)** [1997] to capture long-term dependencies and to **[Bidirectional RNN](https://ieeexplore.ieee.org/document/650093)** [1997] to capture left-to-right and right-to-left dependencies. Eventually, **[Encoder-Decoder RNNs](https://proceedings.neurips.cc/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf)** [2014] emerged, where an RNN creates a document embedding (i.e., the encoder), and another RNN decodes it into text (i.e., the decoder). - **[2017]** [Transformer](https://arxiv.org/abs/1706.03762) The Transformer is an encoder-decoder model that leverages attention mechanisms to compute better embeddings and to align output better to input. This model marked a significant advancement in NLP tasks. - **[2018]** [BERT](https://arxiv.org/abs/1810.04805) BERT is a bidirectional Transformer pre-trained using a combination of Masked Language Modeling and Next Sentence Prediction objectives. It uses global attention. - **[2018]** [GPT](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) GPT is the first autoregressive model based on the Transformer architecture. Later, it evolved into **[GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)** [2019], a bigger and optimized version of GPT pre-trained on WebText, and **[GPT-3](https://arxiv.org/abs/2005.14165)** [2020], a further bigger and optimized version of GPT-2, pre-trained on Common Crawl. - **[2019]** [CTRL](https://arxiv.org/abs/1909.05858) CTRL, similar to GPT, introduced control codes for conditional text generation. This allowed for more control over the generated text. - **[2019]** [Transformer-XL](https://arxiv.org/abs/1901.02860) Transformer-XL reused previously computed hidden states to attend to a longer context. This allowed the model to handle longer sequences of text. - **[2019]** [ALBERT](https://arxiv.org/abs/1909.11942) ALBERT is a lighter version of BERT where (1) Next Sentence Prediction is replaced by Sentence Order Prediction, and (2) parameter-reduction techniques are used for lower memory consumption and faster training. - **[2019]** [RoBERTa](https://arxiv.org/abs/1907.11692) RoBERTa is a better version of BERT, where (1) the Masked Language Modeling objective is dynamic, (2) the Next Sentence Prediction objective is dropped, (3) the BPE tokenizer is employed, and (4) better hyperparameters are used. - **[2019]** [XLM](https://arxiv.org/abs/1901.07291) XLM, a multilingual Transformer, was pre-trained using objectives like Causal Language Modeling, Masked Language Modeling, and Translation Language Modeling. - **[2019]** [XLNet](https://arxiv.org/abs/1906.08237) It’s a Transformer-XL with a generalized autoregressive pre-training method that enables learning bidirectional dependencies. - **[2019]** [PEGASUS](https://arxiv.org/abs/1912.08777) PEGASUS, a bidirectional encoder and left-to-right decoder, was pre-trained with Masked Language Modeling and Gap Sentence Generation objectives. - **[2019]** [DistilBERT](https://arxiv.org/abs/1910.01108) It is the same as BERT but smaller and faster while preserving over 95% of BERT’s performances. Trained by distillation of the pre-trained BERT model. - **[2019]** [XLM-RoBERTa](https://arxiv.org/pdf/1911.02116.pdf) XLM-RoBERTa is a multilingual version of RoBERTa, trained on a multilanguage corpus with the Masked Language Modeling objective. - **[2019]** [BART](https://arxiv.org/abs/1910.13461) BART, a bidirectional encoder and left-to-right decoder, was trained by corrupting text with an arbitrary noising function and learning a model to reconstruct the original text. - **[2019]** [ConvBERT](https://arxiv.org/abs/2008.02496) ConvBERT replaced self-attention blocks with new ones that leveraged convolutions to better model global and local contexts. - **[2020]** [Funnel Transformer](https://arxiv.org/abs/2006.03236) It’s a type of Transformer that gradually compresses the sequence of hidden states to a shorter one, reducing the computation cost. - **[2020]** [Reformer](https://arxiv.org/abs/2001.04451) Reformer


# The Evolution of Language Modeling up to LLMs
## Model’s Timeline
is a more efficient Transformer thanks to local-sensitive hashing attention, axial position encoding, and other optimizations. - **[2020]** [T5](https://arxiv.org/abs/1910.10683) T5, a bidirectional encoder and left-to-right decoder, was pre-trained on a mix of unsupervised and supervised tasks. - **[2020]** [Longformer](https://arxiv.org/abs/2004.05150) Longformer replaced the attention matrices with sparse matrices for higher training efficiency. This made the model faster and more memory-efficient. - **[2020]** [ProphetNet](https://arxiv.org/abs/2001.04063) ProphetNet was trained with the Future N-gram Prediction objective and with a novel self-attention mechanism. - **[2020]** [ELECTRA](https://arxiv.org/abs/2003.10555) Lighter and better than BERT, ELECTRA was trained with the Replaced Token Detection objective. This made the model more efficient and improved its performance on NLP tasks. - **[2021]** [Switch Transformers](https://arxiv.org/abs/2101.03961) Switch Transformers introduced a sparsely-activated expert Transformer model, aiming to simplify and improve over Mixture of Experts. This allowed the model to handle a wider range of tasks. The years 2020 and 2021 are the ones where Large Language Models truly arose. Up to 2020, most language models were able to generate good-looking texts. After this date, the best language models could follow instructions and solve various tasks aside from simple text generation.


# The Evolution of Language Modeling up to LLMs
## The Transformer
The most crucial model of the previous pipeline is, without doubt, the Transformer, introduced in the very popular paper “[Attention Is All You Need](https://arxiv.org/abs/1706.03762).” The Transformer is a type of neural network that is used today by all of the best Large Language Models like GPT-4, Claude, and LLaMA. Central to Transformers is the encoder-decoder structure, which excels at modeling long-range dependencies and capturing contextual information. **The Encoder** processes the input text, identifying key elements and creating word embeddings based on their relevance to other words in the sentence. In the original Transformer architecture, designed for text translation, the attention mechanism was employed in two distinct ways: encoding the source language and decoding the encoded embedding back into the target language. On the other hand, **the Decoder** takes the encoder's output, an embedding, and transforms it back into text. Some models may opt to use only the decoder, bypassing the encoder entirely. The decoder's attention mechanism differs slightly from the encoder's, functioning more like a conventional language model by focusing on previous words during text processing. This approach is particularly useful for tasks like language generation, which is why models like GPT, primarily designed for text generation in response to an input text sequence, utilize the decoder part of the Transformer. Later in the course, we’ll learn more about the Transformer architecture. ![ Image from the paper **[Attention is All You Need](https://arxiv.org/abs/1706.03762)**.](The%20Evolution%20of%20Language%20Modeling%20up%20to%20LLMs%20d011060497644ebeba93113a55433606/attention.png) Image from the paper **[Attention is All You Need](https://arxiv.org/abs/1706.03762)**.


# The Evolution of Language Modeling up to LLMs
## Scaling Transformers: What Lead to Large Language Models
The effectiveness of the Transformer models was further improved by **scaling**, i.e., increasing the number of parameters and training on more data. This scaling led to models with more than 100B parameters that could perform tasks using few-shot or zero-shot approaches, eliminating the need for fine-tuning specific tasks. The increase in the size of these models and the datasets used for training them (and thus the associated costs) led to the large language models that we see today, like Cohere Command, GPT-4, and LLaMA.


# The Evolution of Language Modeling up to LLMs
## Conclusion
In this lesson, we navigated through the rich history of Natural Language Processing, tracing the path from the rudimentary Bag of Words model to the advanced Transformer family. This timeline underscored the continuous innovation in NLP, spotlighting the progression of models in sophistication and proficiency. In the next lesson, we’ll continue the timeline of popular models from 2020 (with GPT-3) up to today.


# Fine-Tuning using Cohere for Medical Data
## Introduction
In this lesson, we will adopt an entirely different method for fine-tuning a large language model, leveraging a platform called [Cohere](https://cohere.com/). This approach allows you to craft a personalized model with just providing sample inputs and outputs, while the service handles the fine-tuning process in the background. Essentially, you supply a set of examples and, in return, obtain a fine-tuned model. For instance, in the context of a classification model, a sample entry would consist of a pair containing <text, label>. Cohere utilizes a collection of exclusive models to execute various functions like [summarization](https://cohere.com/summarize), [embedding](https://docs.cohere.com/docs/multilingual-language-models), [chat](https://cohere.com/chat), and more, all accessible through APIs. Additionally, they empower us to enhance their models by customizing them to suit our precise use case through fine-tuning. It is possible to create [custom models](https://docs.cohere.com/docs/training-custom-models) for 3 distinct objectives: 1) Generative task where we expect the model to generate a text as the output, 2) Classifier which the model will categorizes the text in different categories, or 3) Rerank to enhance semantic search ([What is Semantic Search?](https://docs.cohere.com/docs/what-is-semantic-search)) results. This lesson explores the procedure of fine-tuning a customized generative model using medical texts to extract information. The task, known as [Named Entity Recognition (NER)](https://en.wikipedia.org/wiki/Named-entity_recognition), empowers the models to identify various entities (such as names, locations, dates, etc.) within a text. Large Language Models simplify the process of instructing a model to locate desired information within content. In the following sections we will delve into the procedure of fine-tuning a model to extract diseases, chemicals, and their relationships from a paper abstract.


# Fine-Tuning using Cohere for Medical Data
## Cohere API
The Cohere service offers a range of robust base models tailored for various objectives. Since our focus is on generative tasks, you have the option to select either base models for faster performance or command models for enhanced capability. Both variants also include a "light" version, which is a smaller-sized model, providing you with additional choices. To access the API, you must first [create an account](https://dashboard.cohere.com/welcome/register) on the Cohere platform. You can then proceed to the "API Keys" page, where you will find a Trial key available for free usage. Note that the trial key has rate limitations and cannot be used in a production environment. Nonetheless, there is a valuable opportunity to utilize the models and conduct your experiments prior to submitting your application for production deployment. Now, let's install the Cohere Python package to seamlessly use their API. You should run the following command in terminal. ```bash pip install cohere ``` Next, you'll need to create a Cohere object, which requires your API key and a prompt to generate a response for your request. You can utilize the following code, but please remember to replace the API placeholder with your own key. ```python import cohere co = cohere.Client("<API_KEY>") prompt = """The following article contains technical terms including diseases, drugs and chemicals. Create a list only of the diseases mentioned. Progressive neurodegeneration of the optic nerve and the loss of retinal ganglion cells is a hallmark of glaucoma, the leading cause of irreversible blindness worldwide, with primary open-angle glaucoma (POAG) being the most frequent form of glaucoma in the Western world. While some genetic mutations have been identified for some glaucomas, those associated with POAG are limited and for most POAG patients, the etiology is still unclear. Unfortunately, treatment of this neurodegenerative disease and other retinal degenerative diseases is lacking. For POAG, most of the treatments focus on reducing aqueous humor formation, enhancing uveoscleral or conventional outflow, or lowering intraocular pressure through surgical means. These efforts, in some cases, do not always lead to a prevention of vision loss and therefore other strategies are needed to reduce or reverse the progressive neurodegeneration. In this review, we will highlight some of the ocular pharmacological approaches that are being tested to reduce neurodegeneration and provide some form of neuroprotection. List of extracted diseases:""" response = co.generate( model='command', prompt = prompt, max_tokens=200, temperature=0.750) base_model = response.generations[0].text print(base_model) ``` ``` - glaucoma - primary open-angle glaucoma ``` The provided code utilizes the `cohere.Client()` method to input your API key. Subsequently, the `prompt` variable will contain the model's instructions. In this case, we want the model to read a scientific paper's abstract from the [PubMed website](https://pubmed.ncbi.nlm.nih.gov/) and extract the list of diseases it can identify. Finally, we employ the `cohere` object's `.generate()` method to specify the model type and provide the prompts, along with certain control parameters. The `max_tokens` parameter determines the maximum number of new tokens the model can produce, while the `temperature` parameter governs the level of randomness in the generated results. As you can see, the command model is robust enough to identify diseases without the need for any examples or additional information. In the upcoming sections, we will explore the fine-tuning feature to assess whether we can enhance the model performance even further.


# Fine-Tuning using Cohere for Medical Data
## The Dataset
Before delving into the details of fine-tuning, let's begin by introducing the dataset we are utilizing and clarifying the objective. We will be utilizing the dataset known as [BC5CDR](https://paperswithcode.com/dataset/bc5cdr) which is short for BioCreative V Chemical Disease Relation. This dataset comprises 1,500 PubMed research papers that have been manually annotated by human experts with structured information. The data has been divided into training, validation, and testing sets, with each set containing 500 samples. Our goal is to fine-tune the model to enable it to identify and extract the names of various diseases/chemicals and their relationship from text. This is very useful because the information about the relationships between chemicals and diseases are usually specified in the paper abstracts, but in this form it’s not actionable. That is, it’s not possible to search for “all the chemicals that influence the disease X”, because we’d have to read all the papers mentioning the “disease X” to do it. If we had an accurate way of extracting this structured information from the unstructured texts of the papers, it would be useful for doing these searches. Now, let's perform some preprocessing on the dataset to transform it into a suitable format for the Cohere service. They support files in three formats: CSV, JSONL, or plain text files. We will use the JSONL format, which should align with the following template. ```json {"prompt": "This is the first prompt", "completion": "This is the first completion"} {"prompt": "This is the second prompt", "completion": "This is the second completion"} ``` You can download the dataset in JSON format here as follows. [bc5cdr.json](Fine-Tuning%20using%20Cohere%20for%20Medical%20Data%207aaba9f14aa7475b8dc4f5331866abb5/bc5cdr.json) Then, we can open file using the code below. <aside> 💡 The provided code is an example showcasing the extraction of disease names. Nevertheless, our final dataset will encompass not only diseases but also chemicals and their corresponding relationships. We present a single step aimed at minimizing the repetition of code. For the complete preprocessing procedure and the resulting dataset, please refer to the notebook. </aside> ```python with open('bc5cdr.json') as json_file: data = json.load(json_file) print(data[0]) ``` ``` {'passages': [{'document_id': '227508', 'type': 'title', 'text': 'Naloxone reverses the antihypertensive effect of clonidine.', 'entities': [{'id': '0', 'offsets': [[0, 8]], 'text': ['Naloxone'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D009270'}]}, {'id': '1', 'offsets': [[49, 58]], 'text': ['clonidine'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D003000'}]}], 'relations': [{'id': 'R0', 'type': 'CID', 'arg1_id': 'D008750', 'arg2_id': 'D007022'}]}, {'document_id': '227508', 'type': 'abstract', 'text': 'In unanesthetized, spontaneously hypertensive rats the decrease in blood pressure and heart rate produced by intravenous clonidine, 5 to 20 micrograms/kg, was inhibited or reversed by nalozone, 0.2 to 2 mg/kg. The hypotensive effect of 100 mg/kg alpha-methyldopa was also partially reversed by naloxone. Naloxone alone did not affect either blood pressure or heart rate. In brain membranes from spontaneously hypertensive rats clonidine, 10(-8) to 10(-5) M, did not influence stereoselective binding of [3H]-naloxone (8 nM), and naloxone, 10(-8) to 10(-4) M, did not influence clonidine-suppressible binding of [3H]-dihydroergocryptine (1 nM). These findings indicate that in spontaneously hypertensive rats the effects of central alpha-adrenoceptor stimulation involve activation of opiate receptors. As naloxone and clonidine do not appear to interact with the same receptor site, the observed functional antagonism suggests the release of an endogenous opiate by clonidine or alpha-methyldopa and the possible role of the opiate in the central control of sympathetic tone.', 'entities': [{'id': '2', 'offsets': [[93, 105]], 'text': ['hypertensive'], 'type': 'Disease', 'normalized': [{'db_name': 'MESH', 'db_id': 'D006973'}]}, {'id': '3', 'offsets': [[181, 190]], 'text': ['clonidine'], 'type': 'Chemical', 'normalized': [{'db_name': 'MESH', 'db_id': 'D003000'}]}, {'id': '4', 'offsets': [[244, 252]], 'text': ['nalozone'], 'type': 'Chemical', 'normalized': []}, {'id': '5', 'offsets': [[274, 285]], 'text': ['hypotensive'], 'type': 'Disease', 'normalized':