Momento | 🦜🔗 Langchain

Main:  On this page #Momento [Momento Cache](https://docs.momentohq.com/) is the world's first truly serverless caching service, offering instant elasticity, scale-to-zero capability, and blazing-fast performance.  [Momento Vector Index](https://docs.momentohq.com/vector-index) stands out as the most productive, easiest-to-use, fully serverless vector index.  For both services, simply grab the SDK, obtain an API key, input a few lines into your code, and you're set to go. Together, they provide a comprehensive solution for your LLM data needs.  This page covers how to use the [Momento](https://gomomento.com) ecosystem within LangChain.  ##Installation and Setup[](#installation-and-setup) - Sign up for a free account [here](https://console.momentohq.com) to get an API key - Install the Momento Python SDK with pip install momento ##Cache[](#cache) Use Momento as a serverless, distributed, low-latency cache for LLM prompts and responses. The standard cache is the primary use case for Momento users in any environment.  To integrate Momento Cache into your application:  from langchain.cache import MomentoCache  Then, set it up with the following code:  from datetime import timedelta from momento import CacheClient, Configurations, CredentialProvider from langchain.globals import set_llm_cache  # Instantiate the Momento client cache_client = CacheClient(     Configurations.Laptop.v1(),     CredentialProvider.from_environment_variable("MOMENTO_API_KEY"),     default_ttl=timedelta(days=1))  # Choose a Momento cache name of your choice cache_name = "langchain"  # Instantiate the LLM cache set_llm_cache(MomentoCache(cache_client, cache_name))  ##Memory[](#memory) Momento can be used as a distributed memory store for LLMs.  ###Chat Message History Memory[](#chat-message-history-memory) See [this notebook](/docs/integrations/memory/momento_chat_message_history) for a walkthrough of how to use Momento as a memory store for chat message history.  ##Vector Store[](#vector-store) Momento Vector Index (MVI) can be used as a vector store.  See [this notebook](/docs/integrations/vectorstores/momento_vector_index) for a walkthrough of how to use MVI as a vector store.


MongoDB Atlas | 🦜🔗 Langchain

Main:  On this page #MongoDB Atlas [MongoDB Atlas](https://www.mongodb.com/docs/atlas/) is a fully-managed cloud database available in AWS, Azure, and GCP. It now has support for native Vector Search on the MongoDB document data.  ##Installation and Setup[](#installation-and-setup) See [detail configuration instructions](/docs/integrations/vectorstores/mongodb_atlas).  We need to install pymongo python package.  pip install pymongo  ##Vector Store[](#vector-store) See a [usage example](/docs/integrations/vectorstores/mongodb_atlas).  from langchain.vectorstores import MongoDBAtlasVectorSearch


Motherduck | 🦜🔗 Langchain

Main:  On this page #Motherduck [Motherduck](https://motherduck.com/) is a managed DuckDB-in-the-cloud service.  ##Installation and Setup[](#installation-and-setup) First, you need to install duckdb python package.  pip install duckdb  You will also need to sign up for an account at [Motherduck](https://motherduck.com/)  After that, you should set up a connection string - we mostly integrate with Motherduck through SQLAlchemy. The connection string is likely in the form:  token="..."  conn_str = f"duckdb:///md:{token}@my_db"  ##SQLChain[](#sqlchain) You can use the SQLChain to query data in your Motherduck instance in natural language.  from langchain.llms import OpenAI, SQLDatabase, SQLDatabaseChain db = SQLDatabase.from_uri(conn_str) db_chain = SQLDatabaseChain.from_llm(OpenAI(temperature=0), db, verbose=True)  From here, see the [SQL Chain](/docs/use_cases/tabular/sqlite) documentation on how to use.  ##LLMCache[](#llmcache) You can also easily use Motherduck to cache LLM requests. Once again this is done through the SQLAlchemy wrapper.  import sqlalchemy from langchain.globals import set_llm_cache eng = sqlalchemy.create_engine(conn_str) set_llm_cache(SQLAlchemyCache(engine=eng))  From here, see the [LLM Caching](/docs/modules/model_io/llms/how_to/llm_caching) documentation on how to use.


Motörhead | 🦜🔗 Langchain

Main:  On this page #Motörhead [Motörhead](https://github.com/getmetal/motorhead) is a memory server implemented in Rust. It automatically handles incremental summarization in the background and allows for stateless applications.  ##Installation and Setup[](#installation-and-setup) See instructions at [Motörhead](https://github.com/getmetal/motorhead) for running the server locally.  ##Memory[](#memory) See a [usage example](/docs/integrations/memory/motorhead_memory).  from langchain.memory import MotorheadMemory


MyScale | 🦜🔗 Langchain

Main:  On this page #MyScale This page covers how to use MyScale vector database within LangChain. It is broken into two parts: installation and setup, and then references to specific MyScale wrappers.  With MyScale, you can manage both structured and unstructured (vectorized) data, and perform joint queries and analytics on both types of data using SQL. Plus, MyScale's cloud-native OLAP architecture, built on top of ClickHouse, enables lightning-fast data processing even on massive datasets.  ##Introduction[](#introduction) [Overview to MyScale and High performance vector search](https://docs.myscale.com/en/overview/)  You can now register on our SaaS and [start a cluster now!](https://docs.myscale.com/en/quickstart/)  If you are also interested in how we managed to integrate SQL and vector, please refer to [this document](https://docs.myscale.com/en/vector-reference/) for further syntax reference.  We also deliver with live demo on huggingface! Please checkout our [huggingface space](https://huggingface.co/myscale)! They search millions of vector within a blink!  ##Installation and Setup[](#installation-and-setup) - Install the Python SDK with pip install clickhouse-connect ###Setting up environments[](#setting-up-environments) There are two ways to set up parameters for myscale index.  - Environment Variables  Before you run the app, please set the environment variable with export: export MYSCALE_HOST='<your-endpoints-url>' MYSCALE_PORT=<your-endpoints-port> MYSCALE_USERNAME=<your-username> MYSCALE_PASSWORD=<your-password> ...  You can easily find your account, password and other info on our SaaS. For details please refer to [this document](https://docs.myscale.com/en/cluster-management/) Every attributes under MyScaleSettings can be set with prefix MYSCALE_ and is case insensitive.  - Create MyScaleSettings object with parameters  ```python from langchain.vectorstores import MyScale, MyScaleSettings config = MyScaleSetting(host="<your-backend-url>", port=8443, ...) index = MyScale(embedding_function, config) index.add_documents(...) ```  ##Wrappers[](#wrappers) supported functions:  - add_texts - add_documents - from_texts - from_documents - similarity_search - asimilarity_search - similarity_search_by_vector - asimilarity_search_by_vector - similarity_search_with_relevance_scores - delete ###VectorStore[](#vectorstore) There exists a wrapper around MyScale database, allowing you to use it as a vectorstore, whether for semantic search or similar example retrieval.  To import this vectorstore:  from langchain.vectorstores import MyScale  For a more detailed walkthrough of the MyScale wrapper, see [this notebook](/docs/integrations/vectorstores/myscale)


Neo4j | 🦜🔗 Langchain

Main:  On this page #Neo4j This page covers how to use the Neo4j ecosystem within LangChain.  What is Neo4j?  Neo4j in a nutshell:  - Neo4j is an open-source database management system that specializes in graph database technology. - Neo4j allows you to represent and store data in nodes and edges, making it ideal for handling connected data and relationships. - Neo4j provides a Cypher Query Language, making it easy to interact with and query your graph data. - With Neo4j, you can achieve high-performance graph traversals and queries, suitable for production-level systems. - Get started quickly with Neo4j by visiting [their website](https://neo4j.com/). ##Installation and Setup[](#installation-and-setup) - Install the Python SDK with pip install neo4j ##Wrappers[](#wrappers) ###VectorStore[](#vectorstore) There exists a wrapper around Neo4j vector index, allowing you to use it as a vectorstore, whether for semantic search or example selection.  To import this vectorstore:  from langchain.vectorstores import Neo4jVector  For a more detailed walkthrough of the Neo4j vector index wrapper, see [documentation](/docs/integrations/vectorstores/neo4jvector)  ###GraphCypherQAChain[](#graphcypherqachain) There exists a wrapper around Neo4j graph database that allows you to generate Cypher statements based on the user input and use them to retrieve relevant information from the database.  from langchain.graphs import Neo4jGraph from langchain.chains import GraphCypherQAChain  For a more detailed walkthrough of Cypher generating chain, see [documentation](/docs/use_cases/graph/graph_cypher_qa)  ###Constructing a knowledge graph from text[](#constructing-a-knowledge-graph-from-text) Text data often contain rich relationships and insights that can be useful for various analytics, recommendation engines, or knowledge management applications. Diffbot's NLP API allows for the extraction of entities, relationships, and semantic meaning from unstructured text data. By coupling Diffbot's NLP API with Neo4j, a graph database, you can create powerful, dynamic graph structures based on the information extracted from text. These graph structures are fully queryable and can be integrated into various applications.  from langchain.graphs import Neo4jGraph from langchain_experimental.graph_transformers.diffbot import DiffbotGraphTransformer  For a more detailed walkthrough generating graphs from text, see [documentation](/docs/use_cases/graph/diffbot_graphtransformer)


NLPCloud | 🦜🔗 Langchain

Main:  On this page #NLPCloud [NLP Cloud](https://docs.nlpcloud.com/#introduction) is an artificial intelligence platform that allows you to use the most advanced AI engines, and even train your own engines with your own data.  ##Installation and Setup[](#installation-and-setup) - Install the nlpcloud package. pip install nlpcloud  - Get an NLPCloud api key and set it as an environment variable (NLPCLOUD_API_KEY) ##LLM[](#llm) See a [usage example](/docs/integrations/llms/nlpcloud).  from langchain.llms import NLPCloud  ##Text Embedding Models[](#text-embedding-models) See a [usage example](/docs/integrations/text_embedding/nlp_cloud)  from langchain.embeddings import NLPCloudEmbeddings


Notion DB | 🦜🔗 Langchain

Main:  On this page #Notion DB [Notion](https://www.notion.so/) is a collaboration platform with modified Markdown support that integrates kanban boards, tasks, wikis and databases. It is an all-in-one workspace for notetaking, knowledge and data management, and project and task management.  ##Installation and Setup[](#installation-and-setup) All instructions are in examples below.  ##Document Loader[](#document-loader) We have two different loaders: NotionDirectoryLoader and NotionDBLoader.  See a [usage example for the NotionDirectoryLoader](/docs/integrations/document_loaders/notion).  from langchain.document_loaders import NotionDirectoryLoader  See a [usage example for the NotionDBLoader](/docs/integrations/document_loaders/notiondb).  from langchain.document_loaders import NotionDBLoader


Nuclia | 🦜🔗 Langchain

Main:  On this page #Nuclia [Nuclia](https://nuclia.com) automatically indexes your unstructured data from any internal and external source, providing optimized search results and generative answers. It can handle video and audio transcription, image content extraction, and document parsing.  Nuclia Understanding API document transformer splits text into paragraphs and sentences, identifies entities, provides a summary of the text and generates embeddings for all the sentences.  ##Installation and Setup[](#installation-and-setup) We need to install the nucliadb-protos package to use the Nuclia Understanding API.  pip install nucliadb-protos  To use the Nuclia Understanding API, we need to have a Nuclia account. We can create one for free at [https://nuclia.cloud](https://nuclia.cloud), and then [create a NUA key](https://docs.nuclia.dev/docs/docs/using/understanding/intro).  To use the Nuclia document transformer, we need to instantiate a NucliaUnderstandingAPI tool with enable_ml set to True:  from langchain.tools.nuclia import NucliaUnderstandingAPI  nua = NucliaUnderstandingAPI(enable_ml=True)  ##Document Transformer[](#document-transformer) See a [usage example](/docs/integrations/document_transformers/nuclia_transformer).  from langchain.document_transformers.nuclia_text_transform import NucliaTextTransformer


Obsidian | 🦜🔗 Langchain

Main:  On this page #Obsidian [Obsidian](https://obsidian.md/) is a powerful and extensible knowledge base that works on top of your local folder of plain text files.  ##Installation and Setup[](#installation-and-setup) All instructions are in examples below.  ##Document Loader[](#document-loader) See a [usage example](/docs/integrations/document_loaders/obsidian).  from langchain.document_loaders import ObsidianLoader


OpenLLM | 🦜🔗 Langchain

Main:  On this page #OpenLLM This page demonstrates how to use [OpenLLM](https://github.com/bentoml/OpenLLM) with LangChain.  OpenLLM is an open platform for operating large language models (LLMs) in production. It enables developers to easily run inference with any open-source LLMs, deploy to the cloud or on-premises, and build powerful AI apps.  ##Installation and Setup[](#installation-and-setup) Install the OpenLLM package via PyPI:  pip install openllm  ##LLM[](#llm) OpenLLM supports a wide range of open-source LLMs as well as serving users' own fine-tuned LLMs. Use openllm model command to see all available models that are pre-optimized for OpenLLM.  ##Wrappers[](#wrappers) There is a OpenLLM Wrapper which supports loading LLM in-process or accessing a remote OpenLLM server:  from langchain.llms import OpenLLM  ###Wrapper for OpenLLM server[](#wrapper-for-openllm-server) This wrapper supports connecting to an OpenLLM server via HTTP or gRPC. The OpenLLM server can run either locally or on the cloud.  To try it out locally, start an OpenLLM server:  openllm start flan-t5  Wrapper usage:  from langchain.llms import OpenLLM  llm = OpenLLM(server_url='http://localhost:3000')  llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")  ###Wrapper for Local Inference[](#wrapper-for-local-inference) You can also use the OpenLLM wrapper to load LLM in current Python process for running inference.  from langchain.llms import OpenLLM  llm = OpenLLM(model_name="dolly-v2", model_id='databricks/dolly-v2-7b')  llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")  ###Usage[](#usage) For a more detailed walkthrough of the OpenLLM Wrapper, see the [example notebook](/docs/integrations/llms/openllm)


OpenSearch | 🦜🔗 Langchain

Main:  On this page #OpenSearch This page covers how to use the OpenSearch ecosystem within LangChain. It is broken into two parts: installation and setup, and then references to specific OpenSearch wrappers.  ##Installation and Setup[](#installation-and-setup) - Install the Python package with pip install opensearch-py ##Wrappers[](#wrappers) ###VectorStore[](#vectorstore) There exists a wrapper around OpenSearch vector databases, allowing you to use it as a vectorstore for semantic search using approximate vector search powered by lucene, nmslib and faiss engines or using painless scripting and script scoring functions for bruteforce vector search.  To import this vectorstore:  from langchain.vectorstores import OpenSearchVectorSearch  For a more detailed walkthrough of the OpenSearch wrapper, see [this notebook](/docs/integrations/vectorstores/opensearch)


OpenWeatherMap | 🦜🔗 Langchain

Main:  On this page #OpenWeatherMap [OpenWeatherMap](https://openweathermap.org/api/) provides all essential weather data for a specific location:  - Current weather - Minute forecast for 1 hour - Hourly forecast for 48 hours - Daily forecast for 8 days - National weather alerts - Historical weather data for 40+ years back This page covers how to use the OpenWeatherMap API within LangChain.  ##Installation and Setup[](#installation-and-setup) - Install requirements with pip install pyowm  - Go to OpenWeatherMap and sign up for an account to get your API key [here](https://openweathermap.org/api/) - Set your API key as OPENWEATHERMAP_API_KEY environment variable ##Wrappers[](#wrappers) ###Utility[](#utility) There exists a OpenWeatherMapAPIWrapper utility which wraps this API. To import this utility:  from langchain.utilities.openweathermap import OpenWeatherMapAPIWrapper  For a more detailed walkthrough of this wrapper, see [this notebook](/docs/integrations/tools/openweathermap).  ###Tool[](#tool) You can also easily load this wrapper as a Tool (to use with an Agent). You can do this with:  from langchain.agents import load_tools tools = load_tools(["openweathermap-api"])  For more information on tools, see [this page](/docs/modules/agents/tools/).


Outline | 🦜🔗 Langchain

Main:  On this page #Outline [Outline](https://www.getoutline.com/) is an open-source collaborative knowledge base platform designed for team information sharing.  ##Setup[](#setup) You first need to [create an api key](https://www.getoutline.com/developers#section/Authentication) for your Outline instance. Then you need to set the following environment variables:  import os  os.environ["OUTLINE_API_KEY"] = "xxx" os.environ["OUTLINE_INSTANCE_URL"] = "https://app.getoutline.com"  ##Retriever[](#retriever) See a [usage example](/docs/integrations/retrievers/outline).  from langchain.retrievers import OutlineRetriever


Petals | 🦜🔗 Langchain

Main:  On this page #Petals This page covers how to use the Petals ecosystem within LangChain. It is broken into two parts: installation and setup, and then references to specific Petals wrappers.  ##Installation and Setup[](#installation-and-setup) - Install with pip install petals - Get a Hugging Face api key and set it as an environment variable (HUGGINGFACE_API_KEY) ##Wrappers[](#wrappers) ###LLM[](#llm) There exists an Petals LLM wrapper, which you can access with  from langchain.llms import Petals


Understanding Hallucinations and Bias

# Understanding Hallucinations and Bias
## **Introduction**
In this lesson, we'll cover the concept of **hallucinations** in LLMs, highlighting their influence on AI applications and demonstrating how to mitigate them using techniques like the retriever's architectures. We'll also explore **bias** within LLMs with examples.


Understanding Hallucinations and Bias

# Understanding Hallucinations and Bias
## Hallucinations in LLMs
In Large Language Models, hallucinations refer to cases where the model produces text that's incorrect and not based on reality. An AI hallucination is a confident response by the model that cannot be grounded in any of its training data. There are several possible reasons for these types of hallucinations: - An LLM could be trained on a dataset that doesn’t have the knowledge required to answer a question. - An LLM does not have a reliable way to check the factual accuracy of its responses. It just outputs a response that may be wholly incorrect yet convincing. - The training dataset used to train the LLM may include fictional content and subjective content, like opinions and beliefs. - LLMs are not generally optimized to say “I don’t know” when they don’t have enough information. Therefore, when the LLM has no answer, it generates whatever is the most probable sequence of text as a response. But note that “most probable” does not necessarily mean “true.” Hallucination is the result of this misalignment. Note that the goal of language modeling is learning the underlying distribution of the words in a language, not to distinguish what is true and what isn’t. Let’s see a clear example of a hallucination. On April 12th, 2023, Databricks announced their “Dolly 2.0” model, an open-source 12B parameter language model fine-tuned on a human-generated instruction-following dataset. GPT-3.5 has been trained on a dataset with a cutoff year of 2021. Therefore, it can’t know about Dolly 2. When GPT-3.5 is asked: “Does Databricks Dolly provide value to the open source community?” it responds with something like this: > **GPT 3.5:** *Databricks Dolly is a tool developed by Databricks, which is designed to help users migrate their data from one platform to another, particularly from on-premise data centers to cloud-based platforms such as AWS or Azure. While Databricks Dolly is not an open source tool, it can still provide value to the open source community in a few ways.[…]* > Except for “Databricks Dolly is a tool developed by DataBricks,” the rest of the text is entirely incorrect! Dolly is not a tool to migrate data; it is open-source, contrary to the response. Still, GPT-3.5 knows about Databricks as a company, and therefore, it generates a plausible description of Dolly 2 as a generic product of Databricks. While this example shows OpenAI GPT-3.5 hallucinating, it’s important to note that this phenomenon applies to other similar LLMs like Bard or LLama. Strategies to mitigate hallucinations include tuning the text generation parameters, cleaning up the training data, precisely defining prompts (prompt engineering), and using retriever architectures to ground responses in specific retrieved documents. ### **Misinformation Spreading** One significant risk associated with hallucinations in LLMs is their potential to generate content that, while appearing credible, is factually incorrect. Due to their limited capacity to understand the context and verify facts, LLMs can unintentionally spread misinformation. There's the potential for individuals with malicious intent to exploit LLMs to spread disinformation deliberately, creating and promoting false narratives. [A study by Blackberry](https://www.prnewswire.com/news-releases/chatgpt-may-already-be-used-in-nation-state-cyberattacks-say-it-decision-makers-in-blackberry-global-research-301737059.html) found that nearly half of the respondents (49%) believed that GPT-4 could be used to spread misinformation. The unrestricted spread of such false information via LLMs can lead to widespread negative impacts across societal, cultural, economic, and political landscapes. It's crucial to address these issues related to LLM hallucinations to ensure the ethical use of these models. ### Tuning the Text Generation Parameters The generated output of LLMs is greatly influenced by various model parameters, including temperature, frequency penalty, presence penalty, and top-p. We’ll learn more about them in a later lesson in


Understanding Hallucinations and Bias

# Understanding Hallucinations and Bias
## Hallucinations in LLMs
the course. Higher temperature values promote randomness and creativity, while lower values make the output more deterministic. Increasing the frequency penalty value encourages the model to use repeated tokens more conservatively. Similarly, a higher presence penalty value increases the likelihood of generating tokens not yet included in the generated text. The “top-p” parameter controls response diversity by setting a cumulative probability threshold for word selection. ### Leveraging External Documents with Retrievers Architectures Response accuracy can be improved by providing domain-specific knowledge to the LLM in the form of external documents. Augmenting the **knowledge base** with domain-specific information allows the model to ground its responses in the knowledge base. After a question from a user, we could retrieve documents relevant to the questions (leveraging a module called “retriever”) and use them in a prompt to produce the answer. This type of process is implemented into architectures typically called “retrievers architectures”. In these architectures: 1. When a user poses a question, the system computes an embedding representation of it. 2. The embedding of the question is then used for executing a **semantic search** in the database of documents (by comparing their embeddings and computing similarity scores). 3. The top-ranked documents are used by the LLM as context to give the final answer. Usually, the LLM is asked to extract the answer from those context passages precisely and not to write anything that can’t be inferred from them. > Retrieval-augmented generation (RAG) is a technique that enhances language model capabilities by sourcing data from external resources and integrating it with the context provided in the model's prompt. > Providing access to external data sources during the prediction process enriches the model’s knowledge and grounding. By leveraging external knowledge, the model can generate more accurate, contextually appropriate responses and be less prone to hallucination.


Understanding Hallucinations and Bias

# Understanding Hallucinations and Bias
## Bias in LLMs
Large language models like GPT-3.5 and GPT-4 have raised serious privacy and ethical concerns. Research has shown that these models are prone to inherent bias, leading to the generation of prejudiced or hateful language, intensifying the concerns regarding their use and governance. > Biases in LLMs arise from various sources: the data, the annotation process, the input representations, the models, and the research design. > For instance, training data that don't represent the diversity of language can lead to demographic biases, resulting in a model's inability to understand and accurately represent certain user groups. Misrepresentation can vary from mild inconveniences to more covert, gradual declines in performance, which can unfairly impact certain demographic groups. LLMs can unintentionally intensify harmful biases through their hallucinations, creating prejudiced and offensive content. The data used to train LLMs frequently includes stereotypes, which the models may unknowingly reinforce. This imbalance can lead the models to generate prejudiced content that discriminates against underrepresented groups, potentially targeting them based on factors like race, gender, religion, and ethnicity. This can be exemplified when an LLM produces content that presents women as inferior or portrays certain ethnicities as intrinsically violent or unreliable. Also, if a model is trained on data biased towards a younger, technologically savvy demographic, it may generate outputs that overlook older individuals or those from less technologically equipped regions. If the model is steeped in data from sources promoting hate speech or toxic content, it might produce damaging and prejudiced outputs, amplifying the diffusion of harmful stereotypes and biases. These examples underscore the urgent need for constant monitoring and ethical management in the use of these models.


Understanding Hallucinations and Bias

# Understanding Hallucinations and Bias
## Constitutional AI
Constitutional AI' is a conceptual framework crafted by researchers at Anthropic. It aims to align AI systems with human values, ensuring that they become beneficial, safe, and trustworthy. In the beginning, the model is trained to self-review and modify its responses based on a set of predetermined principles and a small set of process examples. The next phase involves reinforcement learning training. At this point, the model leans on AI-generated feedback, grounded in the given principles, as opposed to human feedback, to choose the least harmful response. > Constitutional AI employs methodologies like **self-supervision training**. These techniques allow the AI to learn to conform to its constitution, without the need for explicit human labeling or supervision. > The approach also includes developing constrained optimization techniques. These ensure that the AI pursues helpfulness within the boundaries set by its constitution rather than pursuing unbounded optimization, potentially forgetting helpful knowledge.


Understanding Hallucinations and Bias

# Understanding Hallucinations and Bias
## Conclusion
The risks of hallucinations and biases in LLMs present significant issues in producing reliable and accurate outputs. The presence of biases can further damage the accuracy and fairness of the outputs, resulting in the ongoing progression of harmful stereotypes and misinformation. It's imperative to formulate strategies to mitigate these risks. Such strategies should incorporate pre-processing and input control measures, model configuration adjustments, improvement mechanisms, and context and knowledge enhancement techniques. Integrating the ethical guidelines is essential to ensure that the models generate fair and trustworthy outputs, ultimately achieving responsible use of these powerful technologies.


Introduction to LLMs Module

# Introduction to LLMs Module
## Introduction to LLMs
This module uncovers the core principles of Large Language Models (LLMs), zooming in on their foundational underpinnings. We provide a historical perspective, highlighting the emergence of Transformers and the difference between proprietary and open-source LLMs. Key attention is on recognizing and mitigating inherent issues like hallucinations and biases within these models. The module is structured as follows, concisely describing each lesson. - **What are Large Language Models?** This lesson dives into the core principles of LLMs, highlighting the capabilities of notable models such as GPT-3 and GPT-4. We introduce the concepts of tokens, few-shot learning, emergent abilities, and the significance of scaling laws. As we explore the functions and outputs of these models, we emphasize the potential challenges of hallucinations and biases. Additionally, we also discuss the context size limitation in LLMs. - **The Evolution of LLMs and Transformers**: This lesson provides a chronological narrative of the progression in language modeling techniques. Starting with the foundational Bag of Words model from 1954, we navigate through significant milestones like TF-IDF, the groundbreaking Word2Vec with its semantic-rich word embeddings, and the sequence-processing capabilities of RNNs. Central to our exploration is the 2017 Transformer architecture, which set the stage for powerhouses like BERT, RoBERTa, and ELECTRA. This lesson only offers an overview without the deep technical intricacies, presenting a panoramic view of the model evolution in NLP. - **A timeline of Large Language Models**: This lesson steers towards a comprehensive overview of the advancements in the Large Language Models landscape, spotlighting models that marked distinct milestones such as GPT-3, PaLM, and Galactica. Recognizing the significant role of techniques like scaling and alignment tuning in the unprecedented capabilities exhibited by LLMs, we untangle the principles that steer these giants. From exploring the enigmatic emergent abilities to decoding the scaling laws, this lesson explores the phenomena driving the potency and performance of LLMs. - **Emergent Abilities in LLMs**: This lesson covers the unexpected skills that surface in Large Language Models as they grow beyond certain thresholds. As models expand, they exhibit unique capabilities influenced by factors like training compute. These emergent skills indicate performance leaps in LLMs as they scale, revealing unforeseen learning beyond what was initially anticipated. - **Proprietary LLMs**: This lesson introduces prominent proprietary Large Language Models such as GPT-4, ChatGPT, and Cohere, among others. We'll weigh the advantages and drawbacks of proprietary models against open-source counterparts. Practical demonstrations will guide students in executing API calls for select models. - **Open-Source LLMs**: This lesson offers insights into open-source Large Language Models, with a focus on LLaMA 2, Open Assistant, Dolly, and Falcon. We will explore their unique features, capabilities, and licensing details. Additionally, we'll discuss potential commercial uses and emphasize any restrictions within their licenses. - **Understanding Hallucinations and Bias in LLMs**: This lesson focuses on the challenges posed by hallucinations and biases in Large Language Models. We'll define hallucinations, provide examples, and discuss their impact on LLM use cases. We'll also explore methods to minimize these issues, such as retriever architectures. The session also covers the concept of bias, its origin in LLMs, and potential mitigation strategies, including approaches like constitutional AI. - **Applications and Use-Cases of LLMs:** This lesson highlights the leading applications and emerging trends of Large Language Models across industries. By referencing real-world news and examples, we illustrate the transformative impact of LLMs across sectors. While emphasizing the vast potential benefits, the module also underscores the importance of recognizing LLMs' limitations and potential challenges. This section provided a comprehensive overview of Large Language Models, highlighting their evolution and significant milestones. Topics ranged from understanding emergent abilities in LLMs to


Introduction to LLMs Module

# Introduction to LLMs Module
## Introduction to LLMs
discerning between proprietary and open-source models. Critical challenges like hallucinations and biases were also addressed.


Datasets for Training LLMs

# Datasets for Training LLMs
## Introduction
In this lesson, we talk about the datasets that fuel LLMs pretraining. We'll explore popular datasets like Falcon RefinedWeb, The Pile, Red Pajama Data, and Stack Overflow Posts, understanding their composition, sources, and usage. We'll also discuss the emerging trend of prioritizing data quality over quantity in pretraining LLMs.


Datasets for Training LLMs

# Datasets for Training LLMs
## Popular Datasets for Training LLMs
In recent times, a variety of open-source datasets have been employed for pre-training Large Language Models. Some of the notable datasets include **"Falcon RefinedWeb,” "The Pile,” "Red Pajama Data,”** and **"Stack Overflow Posts,"** among others**.** Assembling such datasets typically involves collecting and cleaning vast volumes of text data. ### Falcon RefinedWeb The [Falcon RefinedWeb dataset](https://huggingface.co/datasets/tiiuae/falcon-refinedweb) is a large-scale English web dataset developed by [TII](https://www.tii.ae/) and released under the ODC-By 1.0 license. It was created through rigorous filtering and extensive deduplication of [CommonCrawl](https://commoncrawl.org/), resulting in a dataset that has shown comparable or superior performance to models trained on curated datasets, relying solely on web data. The dataset is designed to be "multimodal-friendly,” as it includes links and alt texts for images in the processed samples. Depending on the tokenizer used, the public extract of this dataset ranges from 500-650GT and requires about 2.8TB of local storage when unpacked. Falcon RefinedWeb has been primarily used for training [Falcon LLM models](https://falconllm.tii.ae/), including the Falcon-7B/40B and Falcon-RW-1B/7B models. The dataset is primarily in English, and each data instance corresponds to a unique web page that has been crawled, processed, and deduplicated. It contains around 1 billion instances. The dataset was constructed using the [Macrodata Refinement Pipeline](https://huggingface.co/datasets/tiiuae/falcon-refinedweb#curation-rationale), which includes content extraction, filtering heuristics, and deduplication. The design philosophy of RefinedWeb prioritizes scale, strict deduplication, and neutral filtering. The dataset was iteratively refined by measuring the zero-shot performance of models trained on development versions of the dataset and manually auditing samples to identify potential filtering improvements. ### The Pile [The Pile](https://pile.eleuther.ai/) is a comprehensive, open-source dataset of English text designed specifically for training LLMs. Developed by [EleutherAI](https://www.eleuther.ai/) in 2020, it's a massive 886.03GB dataset comprising 22 smaller datasets, 14 of which are new. Prior to the Pile's creation, most LLMs were trained using data from the Common Crawl. However, the Pile offers a more diverse range of data, enabling LLMs to handle a broader array of situations post-training. The Pile is a carefully curated collection of data handpicked by EleutherAI's researchers to include information they deemed necessary for language models to learn. The Pile covers a wide range of topics and writing styles, including academic writing, a style that models trained on other datasets often struggle with. All data used in the Pile was sourced from publicly accessible resources and filtered to remove duplicates and non-textual elements like HTML formatting and links. However, individual documents within the sub-datasets were not filtered to remove non-English, biased, or profane text, nor was consent considered in the data collection process. Originally developed for EleutherAI's GPT-Neo models, the Pile has since been used to train a variety of other models. ### RedPajama Dataset The [RedPajama dataset](https://together.ai/blog/redpajama) is a comprehensive, open-source dataset that emulates the LLaMa dataset. It comprises 2084 jsonl files, which can be accessed via HuggingFace or directly downloaded. The dataset is primarily in English but includes multiple languages in its Wikipedia section. The dataset is structured into text and metadata, including the URL, timestamp, source, language, and more. It also specifies the subset of the RedPajama dataset it belongs to, such as Commoncrawl, C4, GitHub, Books, ArXiv, Wikipedia, or StackExchange. The dataset is sourced from various platforms: - Commoncrawl data is processed through the official [cc_net](https://github.com/facebookresearch/cc_net) pipeline, deduplicated, and filtered for quality. - [C4](https://huggingface.co/datasets/c4) data is obtained from HuggingFace and formatted to suit the dataset's structure. - GitHub data is sourced from Google BigQuery, deduplicated, and filtered for quality, with only MIT, BSD, or Apache-licensed projects included. - The Wikipedia data is sourced from HuggingFace and is based on a 2023 dump, with hyperlinks, comments, and other formatting


Datasets for Training LLMs

# Datasets for Training LLMs
## Popular Datasets for Training LLMs
removed. - Gutenberg and Books3 data are also downloaded from HuggingFace, with near duplicates removed using simhash. - ArXiv data is sourced from Amazon S3, with only latex source files included and preambles, comments, macros, and bibliographies removed. - Lastly, StackExchange data is sourced from the [Internet Archive](https://archive.org/download/stackexchange), with only the posts from the 28 largest sites included, HTML tags removed, and posts grouped into question-answer pairs. The RedPajama dataset encompasses 1.2 trillion tokens, making it a substantial resource for various language model training and research purposes. ### Stack Overflow Posts If you’re interested more in a specific domain like coding, there are massive datasets available for that, too. The [Stack Overflow Posts](https://huggingface.co/datasets/mikex86/stackoverflow-posts) dataset comprises approximately 60 million posts submitted to StackOverflow prior to June 14, 2023. The dataset, sourced from the [Internet Archive StackExchange Data Dump](https://archive.org/download/stackexchange), is approximately 35GB in size and contains around 65 billion text characters. Each record in the dataset represents a post type and includes fields such as Id, PostTypeId, Body, and ContentLicense, among others.


Datasets for Training LLMs

# Datasets for Training LLMs
## Data Quality vs. Data Quantity in Pretraining
As we just saw, many of the most used pretraining datasets today are cleaned and more complete versions of other past datasets. There’s recently been a shift in focus from increasing dataset sizes to “increasing dataset size AND dataset quality.” The paper "[Textbooks Are All You Need](https://arxiv.org/abs/2306.11644)," published in June 2023, shows this trend. It introduces Phi-1, an LLM designed for code. Phi-1 is a Transformer-based model with 1.3 billion parameters, trained over a period of four days on eight A100s. Despite its relatively smaller scale, it exhibits remarkable accuracy on benchmarks like [HumanEval](https://github.com/openai/human-eval) and [MBPP](https://paperswithcode.com/dataset/mbpp). How? It’s been trained on high-quality data (i.e., textbook-quality data; that’s why the paper name is “Textbooks are all you need”). The training data for Phi-1 comprises 6 billion tokens of "textbook quality" data from the web and 1 billion tokens from synthetically generated textbooks using GPT-3.5. Although Phi-1's specialization in Python coding and lack of domain-specific knowledge somewhat limit its versatility, these limitations are not inherent and can be addressed to enhance its capabilities. Despite its smaller size, the model's success in coding benchmarks demonstrates the significant impact of high-quality and coherent data on the proficiency of language models, thereby shifting the focus from quantity to quality of data.


Datasets for Training LLMs

# Datasets for Training LLMs
## Creating Your Own Dataset
Creating your own dataset would involve a whole lesson for it and we won’t cover it in this course in detail. However, if you’re interested in doing so, you can study the creation process of the datasets listed in the above sections, as it’s often a publicly disclosed process.


Datasets for Training LLMs

# Datasets for Training LLMs
## Conclusion
This lesson provides a comprehensive overview of the datasets that fuel the pretraining of LLMs. We delved into popular datasets such as Falcon RefinedWeb, The Pile, Red Pajama Data, and Stack Overflow Posts, understanding their composition, sources, and usage. Often derived from larger, less refined datasets, these datasets have been meticulously cleaned and curated to provide high-quality data for training LLMs. We also discussed the emerging trend of prioritizing data quality over quantity in pretraining LLMs, as exemplified by the Phi-1 model. Despite its smaller scale, Phi-1's high performance on benchmarks underscores the significant impact of high-quality and coherent data on the proficiency of language models. This shift in focus from data quantity to quality is an exciting development in the field of LLMs, highlighting the importance of dataset refinement in achieving superior model performance.


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Introduction
In this lesson, we'll guide you through the step-by-step process of training a large language model from the ground up. Our primary focus will be on conducting the pre-training process in the cloud. Nevertheless, it's worth noting that all the concepts covered here can be transferable if you want to train a model locally and have enough resources on your local machine (only for small language models). When embarking on model training, three key components must be taken into account. The process begins with selecting an appropriate dataset that aligns with your specific use case. Next, configure the architecture of the model, making adjustments based on the resources at your disposal. Finally, execute the training loop, bringing everything together to train the model effectively. We integrate well-known libraries like Deep Lake Datasets and Transformers into our implementation to build a smooth pipeline. The initial step to initiate the process involves selecting the dataset.


Train an LLM in the Cloud

# Train an LLM in the Cloud
## GPU Cloud - Lambda
In this lesson, we’ll leverage Lambda, the GPU cloud designed by ML engineers for training LLMs & Generative AI. We can create an account on it, link a billing account, and then [rent one instance of the following GPU servers with associated costs](https://lambdalabs.com/service/gpu-cloud/pricing). Please follow the instructions in the course logistics section to open a Lambda account. The cost of your instance is based on its duration, not just the time spent training your model; so remember to turn your instance off. For this lesson, we rented an 8x NVIDIA A100 instance comprising 40GB of memory for $8.80/h. If you're using the Lambda-provided cloud credit for the course, be aware that you still need to register a credit card. The credit will cover costs up to $75, but you must have a card on file. If you spend more money than allocated by the credit (more than $75), you will have to cover those costs yourself. You can find the code of this lesson in this [Notebook](https://colab.research.google.com/drive/1MVeH4vbbVZnZwVhEKxq1RNhUKP245Udt?usp=sharing). <aside> ⚠ Beware of costs when you borrow cloud GPUs. The total cost will depend on the machine type and the up time of the instance. Always remember to monitor your costs in the billing section of Lambda Labs and to spin off your instances when you don’t use them. </aside> <aside> 💡 If you just want to replicate the code in the lesson spending very few money, you can just run the training in your instance and stop it after a few iterations. </aside>


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Training Monitoring - Weights and Biases
As we’re going to spend a lot of money in training our LLM and ensure that everything is progressing smoothly, we’ll log the training metrics to Weights and Biases, allowing us to see the metrics in real time in a suitable dashboard.


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Load the Dataset
During the pre-training process, we utilize the Activeloop datasets to stream the samples seamlessly, batch by batch. This approach proves beneficial for resource management as loading the entire dataset directly into memory is unnecessary. Consequently, it greatly helps in optimizing resource usage. You can quickly load the dataset, and it automatically handles the streaming process without requiring any special configurations. You can load the datasets in just one line of code and visualize their content for analysis. The [library seamlessly integrates with PyTorch and TensorFlow](https://docs.deeplake.ai/en/latest/Pytorch-and-Tensorflow-Support.html), which are considered two of the most powerful frameworks for implementing AI applications. You can head out to [datasets.activeloop.ai](https://datasets.activeloop.ai/docs/ml/datasets/) to see the complete list of available datasets. Porting your datasets to the hub is also achievable with minimal effort. Let’s start by loading the `openwebtext` [dataset](https://app.activeloop.ai/activeloop/openwebtext-train), a collection of Reddit posts with at least three upvotes. This dataset is well-suited for acquiring broad knowledge to build a foundational model for general purposes. The Deep Lake web UI simplifies dataset exploration through its table view and empowers you to query the data using [TQL](https://docs.activeloop.ai/performance-features/querying-datasets) (Tensor Query Language). You can notice that it's possible to quickly inspect dataset details, even when dealing with a sizable dataset containing 8 million rows. This comes thanks to Deep Lake's format, that enables rapid data streaming straight to your browser. ![Deep Lake Visualization Engine table view.](Train%20an%20LLM%20in%20the%20Cloud%209de9852bd1654b219cd67299d66a1761/Screenshot_2023-10-05_at_9.06.07_AM.png) Deep Lake Visualization Engine table view. ```python import deeplake ds = deeplake.load('hub://activeloop/openwebtext-train') ds_val = deeplake.load('hub://activeloop/openwebtext-val') print(ds) print(ds[0].text.text()) ``` ```python Dataset(path='hub://activeloop/openwebtext-train', read_only=True, tensors=['text', 'tokens']) "An in-browser module loader configured to get external dependencies directly from CDN. Includes babel/typescript. For quick prototyping, code sharing, teaching/learning - a super simple web dev environment without node/webpack/etc.\n\nAll front-end libraries\n\nAngular, React, Vue, Bootstrap, Handlebars, and jQuery are included. Plus all packages from cdnjs.com and all of NPM (via unpkg.com). Most front-end libraries should work out of the box - just use import / require() . If a popular library does not load, tell us and we’ll try to solve it with some library-specific config.\n\nWrite modern javascript (or typescript)\n\nUse latest language features or JSX and the code will be transpiled in-browser via babel or typescript (if required). To make it fast the transpiler will start in a worker thread and only process the modified code. Unless you change many files at once or open the project for the first time, the transpiling should be barely noticeable as it runs in parallel with loading a..." ``` The provided code will instantiate a dataset object capable of retrieving the data points for both training and validation sets. Afterward, we can print the variable to examine the dataset's characteristics. It consists of two tensors: `text` containing the textual input and `tokens` representing the tokenized version of the content (which we won't be utilizing). We can also index through the dataset and access each column by using `.text` and convert the row to textual format by calling `.text()` method. The next step involves crafting a PyTorch Dataset class that leverages the loader object and ensures compatibility with the framework. The Dataset class handles both dataset formatting and any desired preprocessing steps to be applied. In this instance, our objective is to tokenize the samples. We will load the GPT-2 tokenizer model from the Transformers library to achieve this. For this specific model, we need to set a padding token (which may not be required for other models), and for this specific purpose, we have chosen to utilize the end of sentence `eos_token` to set the loaded tokenizer’s `pad_token` method. ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") tokenizer.pad_token = tokenizer.eos_token ``` Next, we create dataloaders from


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Load the Dataset
the Deep Lake datasets. In doing so, we also specify a `transform` that tokenizes the texts of the dataset on the fly. ```python # define transform to tokenize texts def get_tokens_transform(tokenizer): def tokens_transform(sample_in): tokenized_text = tokenizer( sample_in["text"], truncation=True, max_length=512, padding='max_length', return_tensors="pt" ) tokenized_text = tokenized_text["input_ids"][0] return { "input_ids": tokenized_text, "labels": tokenized_text } return tokens_transform # create data loaders ds_train_loader = ds.dataloader()\ .batch(32)\ .transform(get_tokens_transform(tokenizer))\ .pytorch() ds_eval_train_loader = ds_val.dataloader()\ .batch(32)\ .transform(get_tokens_transform(tokenizer))\ .pytorch() ``` Please note that we have formatted the dataset so that each sample is comprised of two components: `input_ids` and `labels`. `input_ids` are the tokens the model will use as inputs, while `labels` are the tokens the model will try to predict. Currently, both keys contain the same tokenized text. However, the trainer object from the Transformers library will automatically shift the labels by one token, preparing them for training.


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Initialize the Model
As the scope of this course does not include building the architecture from scratch, we won't be implementing it. We have already covered the details of the Transformer architecture in a previous lesson and provided additional resources for those who are interested in a more in-depth implementation. To accelerate the process, we will leverage an existing publicly available implementation of the transformer architecture. This approach allows us to scale the model quickly using available hyperparameters, including the number of layers, embedding dimension, and attention heads. Additionally, we will capitalize on the success of established architectures while maintaining the flexibility to modify the model size to accommodate our available resources. We opted to utilize the GPT-2 pre-trained model. Nonetheless, there is an option to utilize any other available model from the Huggingface hub; the approach presented here can be easily adapted to work with various architectures. Initially, we examine the default hyperparameters by loading the configuration file and reviewing the choices made in the architecture design. ```python from transformers import AutoConfig config = AutoConfig.from_pretrained("gpt2") print(config) ``` ```python GPT2Config { "_name_or_path": "gpt2", "activation_function": "gelu_new", "architectures": [ "GPT2LMHeadModel" ], "attn_pdrop": 0.1, "bos_token_id": 50256, "embd_pdrop": 0.1, "eos_token_id": 50256, "initializer_range": 0.02, "layer_norm_epsilon": 1e-05, "model_type": "gpt2", "n_ctx": 1024, "n_embd": 768, "n_head": 12, "n_inner": null, "n_layer": 12, "n_positions": 1024, "reorder_and_upcast_attn": false, "resid_pdrop": 0.1, "scale_attn_by_inverse_layer_idx": false, "scale_attn_weights": true, "summary_activation": null, "summary_first_dropout": 0.1, "summary_proj_to_labels": true, "summary_type": "cls_index", "summary_use_proj": true, "task_specific_params": { "text-generation": { "do_sample": true, "max_length": 50 } }, "transformers_version": "4.30.2", "use_cache": true, "vocab_size": 50257 } ``` It is apparent that we have the ability to exert significant control over almost every aspect of the network by manipulating the configuration settings. Specifically, we focus on the following parameters: `n_layer`, which indicates the number of stacking decoder components and defines the embedding layer’s hidden dimension; `n_positions` and `n_ctx`, to represent the maximum number of input tokens; and `n_head` to change the number of attention heads in each attention component. You can read the [documentation](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2Config) to gain a more comprehensive understanding of the remaining parameters. We can start by initializing the model using the default configuration and then count the number of parameters it contains, which will serve as a baseline. To achieve this, we utilize the `GPT2LMHeadModel` class, which takes the `config` variable as input and then proceeds to loop through the parameters, summing them up accordingly. ```python from transformers import GPT2LMHeadModel model = GPT2LMHeadModel(config) model_size = sum(t.numel() for t in model.parameters()) print(f"GPT-2 size: {model_size/1e6:.1f}M parameters") ``` ``` GPT-2 size: 124.4M parameters ``` As shown, the GPT-2 model is relatively small (124M) when compared to the current state-of-the-art large language models. We’re going to pre-train a 124-million-parameter model, which we refer to as `GPT2-scratch-openwebtext`. We chose this size so that a part of its training can be easily replicated by any reader within a reasonable price (~$100). If you wanted to train a larger model, you could modify the architecture to scale it up slightly. As we previously described the selected parameters, we can create a network with 32 layers and an embedding size of 1600. It is worth noting that if not specified, the hidden dimensionality of the linear layers will be `4 × n_embd`. ```python config.n_layer = 32 config.n_embd = 1600 config.n_positions = 512 config.n_ctx = 512 config.n_head = 32 ``` Now, we proceed to load the model with the updated hyperparameters. ```python model_1b = GPT2LMHeadModel(config) model_size = sum(t.numel() for t in model_1b.parameters()) print(f"GPT2-1B size: {model_size/1e6:.1f}M parameters") ``` ``` GPT2-1B size: 1065.8M parameters ``` The modifications led to a model with 1 billion parameters. It is possible to scale the network further to be more in line with


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Initialize the Model
the newest state-of-the-art models, which often have more than 80 layers. However, let’s continue with this lesson's 124M parameters model.


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Training Loop
The final step in the process involves initializing the training loop. We utilize the Transformers library's `Trainer` class, which takes the necessary parameters for training the model. However, before proceeding, we need to create a `TrainingArguments` object that defines all the essential arguments. ```python from transformers import Trainer, TrainingArguments args = TrainingArguments( output_dir="GPT2-scratch-openwebtext", evaluation_strategy="steps", save_strategy="steps", eval_steps=500, save_steps=500, num_train_epochs=2, logging_steps=1, per_device_train_batch_size=1, per_device_eval_batch_size=1, gradient_accumulation_steps=1, weight_decay=0.1, warmup_steps=100, lr_scheduler_type="cosine", learning_rate=5e-4, bf16=True, ddp_find_unused_parameters=False, run_name="GPT2-scratch-openwebtext", report_to="wandb" ) ``` Note that we set the `per_device_train_batch_size` and the `per_device_eval_batch_size` variables to `1` as the batch size is already specified by the dataloader we created earlier. There are over 90 parameters available for adjustment. Find a comprehensive list with explanations in the [documentation](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments). Please note that if there is an "out of memory" error while attempting to train, a smaller `batch_size` can be used. Additionally, the `bf16` flag, which trains the model using lower precision floating numbers, is only available on high-end GPU devices. If unavailable, it can be substituted with the argument `fp16=True`. Notice also that we set the parameter `report_to` to `wandb`; that is, we are sending the training metrics to [Weights and Biases](https://wandb.ai/site) so that we can see a real-time report of how the training is going. Next, we define the `TrainerWithDataLoaders` class, a subclass of `Trainer` where we override the `get_train_dataloader` and `get_eval_dataloader` methods to return our previously defined data loaders. ```python from transformers import Trainer class TrainerWithDataLoaders(Trainer): def __init__(self, *args, train_dataloader=None, eval_dataloader=None, **kwargs): super().__init__(*args, **kwargs) self.train_dataloader = train_dataloader self.eval_dataloader = eval_dataloader def get_train_dataloader(self): return self.train_dataloader def get_eval_dataloader(self, dummy): return self.eval_dataloader ``` The process initiates with a call to the `.train()` method. ```python trainer = TrainerWithDataLoaders( model=model, args=args, train_dataloader=ds_train_loader, eval_dataloader=ds_eval_train_loader, ) trainer.train() ``` The `Trainer` object will handle model evaluation during training, as specified in the `eval_steps` argument, and save checkpoints based on the previously defined in `save_steps`. Here’s the final trained model after about 45 hours of training on 8x NVIDIA A100 ****on Lambda Labs. [GPT2-scratch-openwebtext.zip](Train%20an%20LLM%20in%20the%20Cloud%209de9852bd1654b219cd67299d66a1761/GPT2-scratch-openwebtext.zip) As the [hourly cost of 8x NVIDIA A100 on Lambda Labs is $8.80](https://lambdalabs.com/service/gpu-cloud#pricing), the total cost is $ 400. You can stop your pretraining earlier if you want to spend less money on that. Here’s the training report on [Weights and Biases](https://wandb.ai/ala_/GenAI360/runs/dng49avj?workspace=user-ala_). The following report shows that the training loss decreased relatively smoothly as iterations passed. ![Screenshot 2023-09-04 at 15.31.08.png](Train%20an%20LLM%20in%20the%20Cloud%209de9852bd1654b219cd67299d66a1761/Screenshot_2023-09-04_at_15.31.08.png)


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Inference
Once the pre-training process is complete, we proceed with the inference stage to observe our model in action and evaluate its capabilities. As specified, the `Trainer` will store the intermediate checkpoints in a designated directory called `./GPT2-scratch-openwebtext`. The most efficient approach to utilize the model involves leveraging the Transformers pipeline functionality, which automatically loads both the model and tokenizer, making them ready for text generation. Below is the code snippet that establishes a pipeline object utilizing the pre-trained model alongside the tokenizer we defined in the preceding section. This pipeline enables text generation. ```python from transformers import pipeline pipe = pipeline("text-generation", model="./GPT2-scratch-openwebtext", tokenizer=tokenizer, device="cuda:0") ``` The pipeline object leverages the powerful Transformers `.generate()` method internally, offering exceptional flexibility in managing the text generation process. ([documentation](https://huggingface.co/docs/transformers/v4.18.0/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate)) We can use methods like `min_length` to define a minimum number of tokens to be generated, `max_length` to limit the newly generated tokens, `temperature` to control the generation process between randomness and most likely, and lastly, `do_sample` to modify the completion process, switching between a greedy approach that always selects the most probable token and other sampling methods, such as beam search or diverse search. We only set the `num_return_sequences` to limit the number of generated sequences. ```python txt = "The house prices dropped down" completion = pipe(txt, num_return_sequences=1) print(completion) ``` ``` [{'generated_text': 'The house prices dropped down to 3.02% last year. While it was still in development, the housing market was still down. The recession hit on 3 years between 1998 and 2011. In fact, it slowed the amount of housing from 2013 to 2013'}] ``` The code will attempt to generate a completion for the given input sequence using the knowledge it has acquired from the training dataset. It aims to finish the following sequence: `The house prices dropped down` while being relevant and contextually appropriate. Even with a brief training period, the model exhibits a good grasp of the language, generating grammatically correct and contextually coherent sentences.


Train an LLM in the Cloud

# Train an LLM in the Cloud
## Conclusion
Throughout this lesson, we gained an understanding of the fundamental steps required to train your own language model. The steps involve loading the relevant training data, defining the architecture, scaling it up as per your requirements, and, finally, commencing the training process. As previously discussed, there is no need to train a language model from scratch in many cases. In the upcoming module, we will cover the fine-tuning process in greater detail, enabling you to harness the capabilities of existing powerful models for specific use cases. --- - **Resources** - [Notebook](https://colab.research.google.com/drive/1MVeH4vbbVZnZwVhEKxq1RNhUKP245Udt?usp=sharing). *(Inference results at the end)* - [Weight and Biases Report](https://wandb.ai/ala_/GenAI360/runs/dng49avj?workspace=user-ala_). - Requirements [requirements-train.txt](Train%20an%20LLM%20in%20the%20Cloud%209de9852bd1654b219cd67299d66a1761/requirements-train.txt) *(The provided file is a snapshot of all the packages on the server; not all of these packages are necessary for you)* ---


# How to train NanoGPT with Deep Lake streamable dataloader
In this project lesson, we walk through a training example of Andrej Karpathy’s **[NanoGPT](https://github.com/karpathy/nanoGPT/blob/master/train.py)**. This is the easiest, swiftest repository for training and fine-tuning medium-sized GPTs. The code to train the model is just a 300-line boilerplate training loop and a 300-line GPT model definition completing GPT2. While NanoGPT was designed to be run locally, we built on this and overcame the speed constraints of local training by replacing the local data loader with Deep Lake’s streamable data loader. [https://www.activeloop.ai/resources/generative-ai-data-infrastructure-how-to-train-large-language-models-ll-ms-with-deep-lake/](https://www.activeloop.ai/resources/generative-ai-data-infrastructure-how-to-train-large-language-models-ll-ms-with-deep-lake/)


Open-Source LLMs

# Open-Source LLMs
## **Introduction**
In this lesson, we will discuss several open-source LLMs and their features, capabilities, and licenses. This overview will cover LLaMA 2, Open Assistant, Dolly (by Databricks), and Falcon as the most used LLMs. We will also explore the licenses and potential commercial usage of these models. Additionally, we will discuss limitations or restrictions that may be present in their licenses.


Open-Source LLMs

# Open-Source LLMs
## LLaMA 2
[LLaMA 2](https://ai.meta.com/llama/) is a cutting-edge large language model developed by Meta, released on July 18, 2023, with an open license for both research and commercial use. The architecture of LLaMA 2 is described in great detail in the 77-page [paper](https://arxiv.org/abs/2307.09288), making it easier for data scientists to recreate and fine-tune the models for their specific needs. The model's training data comprises an impressive 2 trillion tokens. It has been trained on a massive scale, outperforming all open-source benchmarks and demonstrating performance comparable to GPT3.5 in terms of human evaluation. LLaMA 2 is available in three parameter variations: 7B, 13B, and 70B, and there are also instruction-tuned versions known as LLaMA-Chat. The fine-tuning process is done through Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF), using a novel approach to segment data based on helpfulness and safety prompts. The reward models are crucial to LLaMA 2's performance, allowing it to balance safety and helpfulness effectively. The safety reward model and helpfulness reward model are trained to evaluate the quality of generated responses. The impact of LLaMA 2 in Generative AI is substantial, outperforming other open innovation models like Falcon or Vicuna. You can find the LLaMA 2 models on the Hugging Face Hub [here](https://huggingface.co/meta-llama). Here, we test the `meta-llama/Llama-2-7b-chat-hf` model. For this, you’ll first have to request access to the model on [this page](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf). First, let’s download the model. It takes some time as the model weighs about 14GB. ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch # download model model_id = "meta-llama/Llama-2-7b-chat-hf" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, torch_dtype=torch.bfloat16 ) ``` Then, we generate a completion with it. This step will take a lot of time if you’re generating text using CPUs instead of GPUs! ```python # generate answer prompt = "Translate English to French: Configuration files are easy to use!" inputs = tokenizer(prompt, return_tensors="pt", return_token_type_ids=False) outputs = model.generate(**inputs, max_new_tokens=100) # print answer print(tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]) ```


Open-Source LLMs

# Open-Source LLMs
## **Falcon**
The [Falcon](https://falconllm.tii.ae/) models, developed and trained by the Technology Innovation Institute (TII) of Abu Dhabi, have gained significant attention since their release in May 2023. These models are causal large language models (LLM), similar to GPT, and are also known as "decoder-only" models. They excel in predicting the next token in a sequence of tokens with their attention focused solely on the left context during training, while the right context remains masked. The Falcon models are distributed under the Apache 2.0 License, allowing even commercial use. The largest of these models, Falcon-40B, has shown great performance, outperforming other causal LLMs like LLaMa-65B and MPT-7B. Falcon-7B, a slightly smaller version, was designed to be fine-tuned on consumer hardware and has half the number of layers and embedding dimensions compared to Falcon-40B. The training data for Falcon models primarily comes from the “Falcon RefinedWeb dataset,” which is meticulously curated and multimodal-friendly, preserving links and alt texts of images. This dataset and curated corpora make up 75% of the pre-training data for the Falcon models. While it primarily covers English, additional versions like "RefinedWeb-Europe" have been prepared to include several European languages. The instruct versions of Falcon-40B and Falcon-7B perform even better, with fine-tuning done on a mixture of chat/instruct datasets sourced from various places, including [GPT4all](https://gpt4all.io/index.html) and [GPTeacher](https://github.com/teknium1/GPTeacher). You can find the Falcon models on the Hugging Face Hub [here](https://huggingface.co/tiiuae). Here, we test the `tiiuae/falcon-7b-instruct` model. You can use the same code previously used for the LLaMA example by changing the `model_id`. ```python model_id = "tiiuae/falcon-7b-instruct" ```


Open-Source LLMs

# Open-Source LLMs
## **Dolly**
[Dolly](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) is an open-source LLM introduced by [Databricks](https://www.databricks.com/). It was first unveiled as Dolly 1.0, a language model that showcased ChatGPT-like human interactivity. The team has now released Dolly 2.0, a better instruction-following LLM. One of the critical features of Dolly 2.0 is that it is built on a new, high-quality human-generated instruction dataset called "databricks-dolly-15k". This dataset consists of 15,000 prompt/response pairs designed explicitly for instruction tuning large language models. Unlike many instruction-following models, Dolly 2.0's dataset is open-source and licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. This means that anyone can use, modify, or extend the dataset for any purpose, including commercial applications. The Dolly 2.0 model is based on the [EleutherAI Pythia-12 b](https://huggingface.co/EleutherAI/pythia-12b) architecture, comprising 12 billion parameters, which makes it capable of high-quality instruction following behavior. Despite being smaller than some other models, such as Alpaca, Dolly 2.0 has demonstrated great performance due to its reliance on real-world, human-generated training records rather than synthesized data. You can find the Databricks models on the Hugging Face Hub [here](https://huggingface.co/databricks). Here, we test the `databricks/dolly-v2-3b` model. You can use the same code previously used for the LLaMA example by changing the `model_id`. ```python model_id = "databricks/dolly-v2-3b" ```


Open-Source LLMs

# Open-Source LLMs
## **Open Assistant**
The [Open Assistant](https://open-assistant.io/) project is an initiative aiming to make high-quality large language models accessible to everyone through an open-source and collaborative approach. Unlike some other ChatGPT open-source alternatives with restricted licenses, Open Assistant seeks to provide a versatile chat-based language model comparable to ChatGPT and GPT-4 that can be used for commercial purposes. The heart of the project lies in its commitment to openness and inclusivity. They have collected a substantial dataset from over 13,000 volunteers, comprising more than 600,000 interactions, 150,000 messages, and 10,000 fully annotated conversation trees on various topics and in multiple languages. This dataset serves as the foundation for training various models hosted on platforms like Hugging Face. Users can explore the potential of Open Assistant by interacting with the model through the Hugging Face demo or the official chat interface, both designed to solicit user feedback to help improve the chatbot's responses. The project encourages community involvement and contributions, allowing users to participate in data collection and ranking tasks to enhance the capabilities of the language model. As with most open-source large language models, Open Assistant does have some limitations, particularly in answering math and coding questions, as they are trained on fewer interactions in these domains. However, the model is generally adept at generating interesting and human-like responses, though occasional inaccuracies may occur.


Open-Source LLMs

# Open-Source LLMs
## Mistral
[Mistral](https://mistral.ai/), in September 2023, has released their language model [Mistral 7B](https://mistral.ai/news/announcing-mistral-7b/) under the Apache 2.0 license. This model, with 7.3 billion parameters, has shown superior performance compared to the Llama 2 13B and Llama 1 34B models on all and many benchmarks respectively. It also approaches the performance of CodeLlama 7B on code while maintaining proficiency in English tasks. Mistral 7B uses Grouped-query attention (GQA) for faster inference and Sliding Window Attention (SWA) to handle longer sequences more cost-effectively. This, along with modifications to FlashAttention and xFormers, has led to a 2x speed improvement for sequence lengths of 16k with a window of 4k. The model can be downloaded and used anywhere, including locally, with the team's reference implementation. It can also be deployed on any cloud (AWS/GCP/Azure) using the [vLLM inference server](https://github.com/vllm-project/vllm), or used on HuggingFace. Mistral 7B is easily fine-tuned for any task. As a demonstration, the team has provided a model fine-tuned for chat, which outperforms the Llama 2 13B chat model. The fine-tuned model, Mistral 7B Instruct, outperforms all 7B models on [MT-Bench](https://arxiv.org/abs/2306.05685) and is comparable to 13B chat models.


Open-Source LLMs

# Open-Source LLMs
## The Hugging Face Open LLM Leaderboard
Hugging Face hosts an **[LLM leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)**. This leaderboard is created by evaluating community-submitted models on text generation benchmarks on Hugging Face’s clusters. It’s an excellent resource for checking the new best-performant open-source LLMs. If you can’t find the language or domain you’re looking for, you can filter them out and find the one that meets your specific requirements.


Open-Source LLMs

# Open-Source LLMs
## **Conclusion**
In this lesson, we explored several open-source LLMs) and their features, capabilities, and licenses. We discussed LLaMA 2, Falcon, Dolly, and Open Assistant as some of the most prominent open-source LLMs available. - LLaMA 2, developed by Meta, is a cutting-edge language model with impressive performance and is available in various parameter variations. It has been trained on a massive scale and demonstrates remarkable performance comparable to GPT3.5. - Falcon models, developed and trained by the Technology Innovation Institute (TII) of Abu Dhabi, have gained attention for their decoder-only approach and have shown great performance, especially the Falcon-40B model. - Dolly, introduced by Databricks, is an open-source LLM with a focus on instruction following. It has a high-quality human-generated instruction dataset and is licensed under Creative Commons, allowing for versatile use, including commercial applications. - Open Assistant is an ambitious project aiming to make high-quality LLMs accessible to everyone through openness and inclusivity. It encourages community involvement and contributions to enhance the capabilities of the language model. It is essential to acknowledge the importance of open-source LLMs in advancing the field of natural language processing and enabling wider access to state-of-the-art language models for research and commercial purposes. In the next lesson, we will explore an equally important aspect of LLMs - hallucinations and bias. Hallucinations refer to the generation of fake or incorrect information by LLMs, while bias entails the perpetuation of prejudiced or discriminatory content. Understanding and addressing these challenges are crucial to ensuring the responsible and ethical use of large language models in various applications.


Focus on the GPT Architecture

# Focus on the GPT Architecture
## Introduction
The Generative Pre-trained Transformer (GPT) is a type of transformer-based language model developed by OpenAI. The 'transformer' part of its name refers to its transformer architecture, which was introduced in the research paper "[Attention is All You Need](https://arxiv.org/abs/1706.03762)" by Vaswani et al. You should have a good understanding of the fundamental elements comprising the transformer architecture. In this session, we will cover the decoder-only networks that play an essential role in developing large language models. We will explore their unique attributes and the reasons behind their effectiveness. In contrast to conventional Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks, the transformer architecture departs from recurrence and adopts self-attention mechanisms, resulting in substantial advancements in speed and scalability. An immensely powerful architecture was unleashed by harnessing the potential for parallelization within the network (simultaneously running multiple head attentions) along with the abundant small cores available in a GPU.


Focus on the GPT Architecture

# Focus on the GPT Architecture
## The GPT Architecture
The GPT family comprises decoder-only models, wherein each block in the stack is comprised of a self-attention mechanism and a position-wise fully connected feed-forward network. The self-attention mechanism, also known as scaled dot-product attention, allows the model to weigh the importance of each word in the input when generating the next word in the sequence. It computes a weighted sum of all the words in the sequence, where the weights are determined by the attention scores. The critical aspect to focus on is the addition of “masking” to the self-attention that prevents the model from attending to certain positions/words. ![Illustrating which tokens are attended to by masked self-attention at a particular timestamp. (Image taken from [NLPiation](https://medium.com/mlearning-ai/what-are-the-differences-in-pre-trained-transformer-base-models-like-bert-distilbert-xlnet-gpt-4b3ea30ef3d7)) As you see in the figure, we pass the whole sequence to the model, but the model at timestep 5 tries to predict the next token by only looking at the previously generated tokens, masking the future tokens. This prevents the model from “cheating” by predicting tokens leveraging future tokens.](Focus%20on%20the%20GPT%20Architecture%20a6de3e541de44464859c4a62f4c132c1/1_mwO-oDzhdMqLhV9XaxKQLQ.webp) Illustrating which tokens are attended to by masked self-attention at a particular timestamp. (Image taken from [NLPiation](https://medium.com/mlearning-ai/what-are-the-differences-in-pre-trained-transformer-base-models-like-bert-distilbert-xlnet-gpt-4b3ea30ef3d7)) As you see in the figure, we pass the whole sequence to the model, but the model at timestep 5 tries to predict the next token by only looking at the previously generated tokens, masking the future tokens. This prevents the model from “cheating” by predicting tokens leveraging future tokens. The following code simply implements the “masked self-attention” mechanism. ```python import numpy as np def self_attention(query, key, value, mask=None): # Compute attention scores scores = np.dot(query, key.T) if mask is not None: # Apply mask by setting masked positions to a large negative value scores = scores + mask * -1e9 # Apply softmax to obtain attention weights attention_weights = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True) # Compute weighted sum of value vectors output = np.dot(attention_weights, value) return output ``` The first step is to compute a Query, Key, and Value vector for each word in the input sequence using separate learned linear transformations of the input vector. It is a simple feedforward linear layer that the model learns during training. Then, we can calculate the attention scores by taking the dot product of its Query vector with the Key vector of every other word. Currently, the application of masking is feasible by setting the scores in specific locations to a large negative number. This effectively informs the model that those words are unimportant and should be disregarded during attention. To get the attention weights, apply the SoftMax function to the attention scores to convert them into probabilities. This gives the weights of the input words and effectively turns the significant negative scores to zero. Lastly, multiply each Value vector by its corresponding weight and sum them up. This produces the output of the masked self-attention mechanism for the word. The provided code snippet illustrates the process of a single self-attention head, but in reality, each layer contains multiple heads, which could range from 16 to 32 heads, depending on the architecture. These heads operate simultaneously to enhance the model's performance.


Focus on the GPT Architecture

# Focus on the GPT Architecture
## ****Causal Language Modeling****
LLMs utilize a **self-supervised learning** process for pre-training. This process eliminates the need to provide explicit labels to the model for learning, making it capable of acquiring knowledge autonomously. For instance, when training a summarization model using supervised learning, it is necessary to provide articles and their corresponding summaries as reference points during the training process. However, LLMs employ the causal language modeling objective to acquire knowledge from any textual data without the explicit need for human-provided labels. Why is it called “causal”? Because the prediction at each step depends only on earlier steps in the sequence and not on future steps. > This process involves feeding a segment of the document to the model and asking it to predict the next word. > Subsequently, the predicted word is concatenated to the original input and fed back to the model to predict a new token. This iterative loop continues, consistently feeding the newly generated token back into the network. During the pre-training process, the network acquires substantial knowledge about language and grammar. We can then fine-tune the pre-trained model using a supervised approach for different tasks or a specific domain. Compared to other well-known objectives, the advantage of this approach is that it models how humans naturally write or speak. In contrast to other objectives like masked language modeling, where masked tokens are introduced in the input, the causal language modeling approach constructs sentences one word at a time. This key difference ensures that our model's performance is not adversely affected when dealing with real-world passages lacking masking tokens. Moreover, we can utilize extensive, high-quality, human-generated content spanning centuries. This content can be derived from books, Wikipedia, news websites, and more. Familiar datasets and repositories, such as [ActiveLoop](https://datasets.activeloop.ai) and [Huggingface](https://huggingface.co/datasets), provide convenient access to some well-known datasets. We will cover this topic in more detail in later lessons.


Focus on the GPT Architecture

# Focus on the GPT Architecture
## MinGPT
Numerous implementations of the GPT architecture exist, each designed for specific purposes. In upcoming lessons, we will thoroughly explore alternative libraries that are better suited for production environments. However, we are introducing a lightweight repository implemented by Andrej Karpathy, named [minGPT](https://github.com/karpathy/minGPT). This represents a minimal implementation of OpenAI's GPT-2 model. In his own words, this serves as an educational implementation that strives to remove all complexities, achieving a length of just 300 lines of code and using the PyTorch library. This valuable resource provides an excellent opportunity to read and enhance your understanding of what's happening under the hood. Abundant comments in the code describe the processes and act as a helpful guide. Three main files can be found within the repository. First, [model.py](https://github.com/karpathy/minGPT/blob/master/mingpt/model.py) handles the definition of architecture details. Second, [bpe.py](https://github.com/karpathy/minGPT/blob/master/mingpt/bpe.py) is responsible for the tokenization process using the BPE algorithm. Lastly, [train.py](https://github.com/karpathy/minGPT/blob/master/mingpt/trainer.py) represents the implementation of a generic training loop for any neural network, not limited to the GPT architecture. Furthermore, the [demo.ipynb](https://github.com/karpathy/minGPT/blob/master/demo.ipynb) file contains a notebook that demonstrates the complete utilization of the code, including the inference process. The code can be executed on a MacBook Air, making it accessible for use on your local PC. Alternatively, you can fork the repository and utilize services like Colab.


Focus on the GPT Architecture

# Focus on the GPT Architecture
## Conclusion
The decoder-only architecture and GPT-family models have driven the recent advancements in large language models. It is essential to possess a strong grasp of the transformer architecture and comprehend the distinctive features that set the decoder-only models apart, making them well-suited for language modeling. We have explored the shared components and delved deeper into what makes their architecture unique. Subsequent lessons will cover various other aspects of language models.


# Fine-Tuning using SFT for Financial Sentiment
## Introduction
In the previous lesson, we experimented with the method of fine-tuning an LLM to follow the instructions like a chatbot. Although this proves beneficial across various applications, we can similarly employ this strategy to train a model tailored for a particular domain. In this lesson, our goal is to create a thoroughly tuned model for conducting **sentiment analysis on financial statements**. Ideally, the LLM would assess financial tweets by categorizing them as Positive, Negative, or Neutral. The dataset utilized in this lesson is the one curated in the [FinGPT project](https://arxiv.org/pdf/2307.10485.pdf). As previously stated, the dataset remains the pivotal and influential factor. Having acknowledged that, and given that we've extensively addressed the process of Supervised Fine-Tuning (SFT) before, this lesson will predominantly touch upon the dataset we utilized and the preprocessing steps involved. Nonetheless, a comprehensive notebook script for running and experimenting is provided at the conclusion of this lesson. The activities showcased in this tutorial involve the utilization of the 4th Generation Intel® Xeon® Scalable Processors (with 64GB RAM), with the use of [Intel® Advanced Matrix Extensions](https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/advanced-matrix-extensions/overview.html) (Intel® AMX). Both finetuning and inference can be accomplished by leveraging its optimization technologies. You can spin up a virtual machine using GCP Compute Engine as explained in the previous lesson. Follow the instructions in the course introduction to spin up a VM with Compute Engine with high-end Intel CPUs. <aside> ⚠ Beware of costs when you spin up virtual machines. The total cost will depend on the machine type and the up time of the machine. Always remember to monitor your costs in the billing section of GCP and to spin off your virtual machines when you don’t use them. </aside> <aside> 💡 If you just want to replicate the code in the lesson spending very few money, you can just run the training in your virtual machine and stop it after a few iterations. </aside>


# Fine-Tuning using SFT for Financial Sentiment
## Load the Dataset
We are set to utilize the FinGPT sentiment dataset, comprising a set of financial tweets along with their corresponding labels. Additionally, this dataset features an `instruction` column containing the initial task directive. Typically, this instruction prompts something akin to "What is the sentiment of the following content? Choose from Positive, Negative, or Neutral.” A smaller subset of the dataset can be accessed from the [300+ free public datasets](https://app.activeloop.ai/public/) curated by team Activeloop accessible in Deep Lake format. We've deliberately chosen a smaller subset to expedite the fine-tuning process. Specifically, the [training set](https://app.activeloop.ai/genai360/FingGPT-sentiment-train-set) comprises 20,000 data points, while we employ 2,000 samples for [validation](https://app.activeloop.ai/genai360/FingGPT-sentiment-valid-set) purposes. The dataset can be explored and queried using the Deep Lake Web UI or filtered using the Python package. The Deep Lake visualization engine enables us to [query the dataset](https://docs.activeloop.ai/performance-features/querying-datasets) and filter relevant rows using its query field. The NLP feature allows you to compose your query in English and receive the corresponding TQL query. ![The Deep Lake Visualization Engine table view with filtering.](Fine-Tuning%20using%20SFT%20for%20Financial%20Sentiment%20f744059ba2f8490b92b5c5a641c1e96a/Screenshot_2023-10-05_at_10.02.05_AM.png) The Deep Lake Visualization Engine table view with filtering. By utilizing the `deeplake.load()` function, we can create the Dataset object and load the samples. ```python import deeplake # Connect to the training and testing datasets ds = deeplake.load('hub://genai360/FingGPT-sentiment-train-set') ds_valid = deeplake.load('hub://genai360/FingGPT-sentiment-valid-set') print(ds) ``` ```python Dataset(path='hub://genai360/FingGPT-sentiment-train-set', read_only=True, tensors=['input', 'instruction', 'output']) ``` At this point, we can proceed to create the function that formats a sample from the dataset into a suitable input for the model. The primary distinction from our previous approach lies in incorporating the instructions at the start of the prompt. The structure is as outlined below: `<instruction>\n\nContent: <tweet>\n\nSentiment: <sentiment>`. The placeholders enclosed in `<>` will be substituted with corresponding values from the dataset. ```python def prepare_sample_text(example): """Prepare the text from a sample of the dataset.""" text = f"{example['instruction'].text()}\n\nContent: {example['input'].text()}\n\nSentiment: {example['output'].text()}" return text ``` Presented here is a formatted input derived from an entry in the dataset. ``` What is the sentiment of this news? Please choose an answer from {negative/neutral/positive} Content: Diageo Shares Surge on Report of Possible Takeover by Lemann Sentiment: positive ``` The subsequent steps should be recognizable from earlier lessons. We initialize the tokenizer for the [OPT-1.3B large language model](https://huggingface.co/facebook/opt-1.3b) and use the `ConstantLengthDataset` to structure the samples. The tokenizer is then employed to convert them into token IDs. Additionally, the class packs multiple samples until the sequence length threshold is reached, thus enhancing the efficiency of the training process. ```python # Load the tokenizer from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b") # Create the ConstantLengthDataset from trl.trainer import ConstantLengthDataset train_dataset = ConstantLengthDataset( tokenizer, ds, formatting_func=prepare_sample_text, infinite=True, seq_length=1024 ) eval_dataset = ConstantLengthDataset( tokenizer, ds_valid, formatting_func=prepare_sample_text, seq_length=1024 ) # Show one sample from train set iterator = iter(train_dataset) sample = next(iterator) print(sample) ``` ```python {'input_ids': tensor([50118, 35212, 8913, ..., 2430, 2, 2]),'labels': tensor([50118, 35212, 8913, ..., 2430, 2, 2])} ``` <aside> 💡 Remember to execute the following code to reset the iterator pointer, if the iterator is used to print a sample from the dataset, `train_dataset.start_iteration = 0` </aside>


# Fine-Tuning using SFT for Financial Sentiment
## Initialize the Model and Trainer
The "Fine-Tuning using SFT" tutorial clarifies the code snippets within this subsection. For additional inquiries, kindly refer to that resource for further understanding. We will quickly walk through the code. Please bear in mind that the fine-tuned checkpoint will be accessible in the Inference section if resources for the fine-tuning process are needed. Additionally, the specifics of the training process are recorded and can be accessed through [Weights and Biases](https://wandb.ai/site). During the training process, system activity can be tracked, including metrics such as memory usage, CPU utilization, duration, loss values, and a range of other parameters. Here’s the Weights and Biases [report](https://wandb.ai/ala_/GenAI360/runs/p08s2n5f?workspace=user-ala_) of the finetuning of this lesson. We start by defining the arguments necessary to configure the training process. We use the `LoraConfig` class from the [PEFT library](https://github.com/huggingface/peft) for that. Subsequently, we employ the `TrainingArguments` class from the transformers library to control the training loop. ```python # Define LoRAConfig from peft import LoraConfig lora_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", ) # Define TrainingArguments from transformers import TrainingArguments training_args = TrainingArguments( output_dir="./OPT-fine_tuned-FinGPT-CPU", dataloader_drop_last=True, evaluation_strategy="epoch", save_strategy="epoch", num_train_epochs=10, logging_steps=5, per_device_train_batch_size=12, per_device_eval_batch_size=12, learning_rate=1e-4, lr_scheduler_type="cosine", warmup_steps=100, gradient_accumulation_steps=1, gradient_checkpointing=False, fp16=False, bf16=True, weight_decay=0.05, ddp_find_unused_parameters=False, run_name="OPT-fine_tuned-FinGPT-CPU", report_to="wandb", ) ``` The subsequent task involves loading the OPT-1.3B model in the `bfloat16` format, which is easily managed by Intel® CPUs and saves memory during fine-tuning. ```python from transformers import AutoModelForCausalLM import torch model = AutoModelForCausalLM.from_pretrained( "facebook/opt-1.3b", torch_dtype=torch.bfloat16 ) ``` The subsequent stage entails casting specific layers within the network to complete 32-bit precision, enhancing the model's stability throughout training. ```python from torch import nn for param in model.parameters(): param.requires_grad = False # freeze the model - train adapters later if param.ndim == 1: # cast the small parameters (e.g. layernorm) to fp32 for stability param.data = param.data.to(torch.float32) model.gradient_checkpointing_enable() # reduce number of stored activations model.enable_input_require_grads() class CastOutputToFloat(nn.Sequential): def forward(self, x): return super().forward(x).to(torch.float32) model.lm_head = CastOutputToFloat(model.lm_head) ``` Now, connect the model, dataset, training arguments, and Lora config together using the `SFTTrainer` class to start the training process by invoking the `.train()` method. ```python from trl import SFTTrainer trainer = SFTTrainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, peft_config=lora_config, packing=True, ) print("Training...") trainer.train() ``` <aside> 💡 Access the best checkpoint that we trained by using the following URL. Additionally, find more information about the fine-tuning process on the Weights and Biases [project page](https://wandb.ai/ala_/GenAI360/runs/p08s2n5f?workspace=user-ala_). </aside> [OPT-fine_tuned-FinGPT-CPU.zip](Fine-Tuning%20using%20SFT%20for%20Financial%20Sentiment%20f744059ba2f8490b92b5c5a641c1e96a/OPT-fine_tuned-FinGPT-CPU.zip)


# Fine-Tuning using SFT for Financial Sentiment
## Merging LoRA and OPT
Before conducting inference and observing the results, the final task is to load the LoRA adaptors from the preceding stage and merge them with the base model. ```python # Load the base model (OPT-1.3B) from transformers import AutoModelForCausalLM import torch model = AutoModelForCausalLM.from_pretrained( "facebook/opt-1.3b", return_dict=True, torch_dtype=torch.bfloat16 ) # Load the LoRA adaptors from peft import PeftModel # Load the Lora model model = PeftModel.from_pretrained(model, "./OPT-fine_tuned-FinGPT-CPU/<desired_checkpoint>/") model.eval() model = model.merge_and_unload() # Save for future use model.save_pretrained("./OPT-fine_tuned-FinGPT-CPU/merged") ```