# Optimizing Your Communication: The Importance of Monitoring Message History
powerful tool that leverages past messages to produce fitting replies, resulting in comprehensive and knowledgeable outputs. This extra memory is invaluable when chatbots have to remember lots of details, especially when users ask for complicated information or engage in complex chats. By implementing the ConversationChain, users can enjoy seamless interactions with chatbots, ultimately enhancing their overall experience. ### **ConversationBufferMemory** The `ConversationChain` uses the `ConversationBufferMemory` class by default to provide a history of messages. This memory can save the previous conversations in form of variables. The class accepts the `return_messages` argument which is helpful for dealing with chat models. This is how the CoversationChain keep context under the hood. ```python from langchain.memory import ConversationBufferMemory memory = ConversationBufferMemory(return_messages=True) memory.save_context({"input": "hi there!"}, {"output": "Hi there! It's nice to meet you. How can I help you today?"}) print( memory.load_memory_variables({}) ) ``` ```python {'history': [HumanMessage(content='hi there!', additional_kwargs={}, example=False), AIMessage(content="Hi there! It's nice to meet you. How can I help you today?", additional_kwargs={}, example=False)]} ``` Alternatively, the code in the previous section is the same as the following. It will automatically call the `.save_context()` object after each interaction. ```python from langchain.chains import ConversationChain conversation = ConversationChain( llm=llm, verbose=True, memory=ConversationBufferMemory() ) ``` The next code snippet shows the full usage of the `ConversationChain` and the `ConversationBufferMemory` class. Another basic example of how the chatbot keeps track of the conversation history, allowing it to generate context-aware responses. ```python from langchain import ConversationChain from langchain.memory import ConversationBufferMemory from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder, SystemMessagePromptTemplate, HumanMessagePromptTemplate prompt = ChatPromptTemplate.from_messages([ SystemMessagePromptTemplate.from_template("The following is a friendly conversation between a human and an AI."), MessagesPlaceholder(variable_name="history"), HumanMessagePromptTemplate.from_template("{input}") ]) memory = ConversationBufferMemory(return_messages=True) conversation = ConversationChain(memory=memory, prompt=prompt, llm=llm) print( conversation.predict(input="Tell me a joke about elephants") ) print( conversation.predict(input="Who is the author of the Harry Potter series?") ) print( conversation.predict(input="What was the joke you told me earlier?") ) ``` ``` **AI: What did the elephant say to the naked man? "How do you breathe through that tiny thing? AI: The author of the Harry Potter series is J.K. Rowling AI: The joke I told you earlier was "What did the elephant say to the naked man? \'How do you breathe through that tiny thing?** ``` Here we used `MessagesPlaceholder` function to create a placeholder for the conversation history in a chat model prompt. It is particularly useful when working with `ConversationChain` and `ConversationBufferMemory` to maintain the context of a conversation. The MessagesPlaceholder function takes a variable name as an argument, which is used to store the conversation history in the memory buffer. We will cover that function later. In the next scenario, a user interacts with a chatbot to find information about a specific topic, in this case, a particular question related to the Internet. ```python from langchain import ConversationChain from langchain.memory import ConversationBufferMemory from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder, SystemMessagePromptTemplate, HumanMessagePromptTemplate prompt = ChatPromptTemplate.from_messages([ SystemMessagePromptTemplate.from_template("The following is a friendly conversation between a human and an AI."), MessagesPlaceholder(variable_name="history"), HumanMessagePromptTemplate.from_template("{input}") ]) memory = ConversationBufferMemory(return_messages=True) conversation = ConversationChain(memory=memory, prompt=prompt, llm=llm, verbose=True) ``` If we start with a general question: ```python user_message = "Tell me about the history of the Internet." response = conversation(user_message) print(response) ``` ``` **> Entering new ConversationChain chain...** Prompt after formatting: ***System: The following is a friendly conversation between a human and an AI. Human: Tell me about the history of the Internet.*** **> Finished chain.** {'input': 'Tell me about the history of the Internet.', 'history': [HumanMessage(content='Tell me about the history of the Internet.', additional_kwargs={}, example=False), AIMessage(content='\n\nAI: The Internet has a long and complex history. It began in the 1960s as a project of the United States Department of Defense, which wanted to create a network of computers


# Optimizing Your Communication: The Importance of Monitoring Message History
that could communicate with each other in the event of a nuclear attack. This network eventually evolved into the modern Internet, which is now used by billions of people around the world.', additional_kwargs={}, example=False)], 'response': '\n\nAI: The Internet has a long and complex history. It began in the 1960s as a project of the United States Department of Defense, which wanted to create a network of computers that could communicate with each other in the event of a nuclear attack. This network eventually evolved into the modern Internet, which is now used by billions of people around the world.'} ``` Here is the second query. ```python # User sends another message user_message = "Who are some important figures in its development?" response = conversation(user_message) print(response) # Chatbot responds with names of important figures, recalling the previous topic ``` ``` **> Entering new ConversationChain chain...** Prompt after formatting: ***System: The following is a friendly conversation between a human and an AI. Human: Tell me about the history of the Internet. AI: AI: The Internet has a long and complex history. It began in the 1960s as a project of the United States Department of Defense, which wanted to create a network of computers that could communicate with each other in the event of a nuclear attack. This network eventually evolved into the modern Internet, which is now used by billions of people around the world. Human: Who are some important figures in its development?*** **> Finished chain.** {'input': 'Who are some important figures in its development?', 'history': [HumanMessage(content='Tell me about the history of the Internet.', additional_kwargs={}, example=False), AIMessage(content='\n\nAI: The Internet has a long and complex history. It began in the 1960s as a project of the United States Department of Defense, which wanted to create a network of computers that could communicate with each other in the event of a nuclear attack. This network eventually evolved into the modern Internet, which is now used by billions of people around the world.', additional_kwargs={}, example=False), HumanMessage(content='Who are some important figures in its development?', additional_kwargs={}, example=False), AIMessage(content='\nAI:\n\nSome of the most important figures in the development of the Internet include Vint Cerf and Bob Kahn, who developed the TCP/IP protocol, Tim Berners-Lee, who developed the World Wide Web, and Marc Andreessen, who developed the first web browser.', additional_kwargs={}, example=False)], 'response': '\nAI:\n\nSome of the most important figures in the development of the Internet include Vint Cerf and Bob Kahn, who developed the TCP/IP protocol, Tim Berners-Lee, who developed the World Wide Web, and Marc Andreessen, who developed the first web browser.'} ``` --- And the last query that showcase how using `ConversationBufferMemory` enables the chatbot to recall previous messages and provide more accurate and context-aware responses to the user's questions. ```python user_message = "What did Tim Berners-Lee contribute?" response = conversation(user_message) print(response) ``` ``` **> Entering new ConversationChain chain...** Prompt after formatting: ***System: The following is a friendly conversation between a human and an AI. Human: Tell me about the history of the Internet. AI: AI: The Internet has a long and complex history. It began in the 1960s as a project of the United States Department of Defense, which wanted to create a network of computers that could communicate with each other in the event of a nuclear attack. This network eventually evolved into the modern Internet, which is now used by billions of people around the world. Human: Who are some important figures in its development? AI: AI: Some of the most important figures in the development of the Internet include Vint Cerf and Bob Kahn, who developed the TCP/IP protocol, Tim Berners-Lee, who


# Optimizing Your Communication: The Importance of Monitoring Message History
developed the World Wide Web, and Marc Andreessen, who developed the first web browser. Human: What did Tim Berners-Lee contribute?*** **> Finished chain.** {'input': 'What did Tim Berners-Lee contribute?', 'history': [HumanMessage(content='Tell me about the history of the Internet.', additional_kwargs={}, example=False), AIMessage(content='\n\nAI: The Internet has a long and complex history. It began in the 1960s as a project of the United States Department of Defense, which wanted to create a network of computers that could communicate with each other in the event of a nuclear attack. This network eventually evolved into the modern Internet, which is now used by billions of people around the world.', additional_kwargs={}, example=False), HumanMessage(content='Who are some important figures in its development?', additional_kwargs={}, example=False), AIMessage(content='\nAI:\n\nSome of the most important figures in the development of the Internet include Vint Cerf and Bob Kahn, who developed the TCP/IP protocol, Tim Berners-Lee, who developed the World Wide Web, and Marc Andreessen, who developed the first web browser.', additional_kwargs={}, example=False), HumanMessage(content='What did Tim Berners-Lee contribute?', additional_kwargs={}, example=False), AIMessage(content='\nAI: \n\nTim Berners-Lee is credited with inventing the World Wide Web, which is the system of interlinked documents and other resources that make up the Internet. He developed the Hypertext Transfer Protocol (HTTP) and the Hypertext Markup Language (HTML), which are the two main technologies used to create and display webpages. He also developed the first web browser, which allowed users to access the web.', additional_kwargs={}, example=False)], 'response': '\nAI: \n\nTim Berners-Lee is credited with inventing the World Wide Web, which is the system of interlinked documents and other resources that make up the Internet. He developed the Hypertext Transfer Protocol (HTTP) and the Hypertext Markup Language (HTML), which are the two main technologies used to create and display webpages. He also developed the first web browser, which allowed users to access the web.'} ``` In the upcoming lessons, we will cover several more types of conversational memory such as → ConversationBufferMemory, which is the most straightforward, then → ConversationBufferWindowMemory, which maintains a memory window that keeps a limited number of past interactions based on the specified window size. → And the most complex variant, ConversationSummaryMemory that holds a summary of previous converations. ### Conclusion Keeping track of message history in chatbot interactions yields several benefits. Firstly, the chatbot gains a stronger sense of context from previous interactions, improving the accuracy and relevance of its responses. Secondly, the recorded history serves as a valuable resource for troubleshooting, tracing the sequence of events to identify potential issues. Thirdly, effective monitoring systems that include log tracking can trigger notifications based on alert conditions, aiding in the early detection of conversation anomalies. Lastly, monitoring message history provides a means to evaluate the chatbot's performance over time, paving the way for necessary adjustments and enhancements. While monitoring message history can offer numerous advantages, there are also some trade-offs to consider. Storing extensive message history can lead to significant memory and storage usage, potentially impacting the overall system performance. Additionally, maintaining conversation history might present privacy issues, particularly when sensitive or personally identifiable information is involved. Therefore, it is crucial to manage such data with utmost responsibility and in compliance with the relevant data protection regulations. To sum up, monitoring message history in LangChain is crucial for providing context-aware, accurate, and engaging AI-driven conversations. It also offers valuable information for troubleshooting, alerting, and performance evaluation. However, it's essential to be mindful of the trade-offs, such as memory and storage consumption and privacy concerns. In the next lesson, we’ll see the different memory classes that LangChain has and when to use them. --- You can find the code of this lesson in


# Optimizing Your Communication: The Importance of Monitoring Message History
this online [Notebook](https://colab.research.google.com/drive/1nCQWCoRG5WgyzHSX5c5FjZLkdRNuk3st?usp=sharing).


Exploring the World of Embeddings

# Exploring the World of Embeddings
### Introduction Vector embeddings are among the most intriguing and beneficial aspects of machine learning, playing a pivotal role in many natural language processing, recommendation, and search algorithms. If you've interacted with recommendation engines, voice assistants, or language translators, you've engaged with systems that utilize embeddings. **Embeddings** are dense vector representations of data that encapsulate semantic information, making them suitable for various machine-learning tasks such as clustering, recommendation, and classification. They transform human-perceived semantic similarity into closeness in vector space and can be generated for different data types, including text, images, and audio. For text data, models like the GPT family of models and Llama are employed to create vector embeddings for words, sentences, or paragraphs. In the case of images, convolutional neural networks (CNNs) such as VGG and Inception can generate embeddings. Audio recordings can be converted into vectors using image embedding techniques applied to visual representations of audio frequencies, like spectrograms. Deep neural networks are commonly employed to train models that convert objects into vectors. The resulting embeddings are typically high-dimensional and dense. Embeddings are extensively used in similarity search applications, such as KNN and ANN, which require calculating distances between vectors to determine similarity. Nearest neighbor search can be employed for tasks like de-duplication, recommendations, anomaly detection, and reverse image search. ### Similarity search and vector embeddings OpenAI offers a powerful language model called GPT-3, which can be used for various tasks, such as generating embeddings and performing similarity searches. In this example, we'll use the OpenAI API to generate embeddings for a set of documents and then perform a similarity search using cosine similarity. First, let's install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken scikit-learn`. Next, create an API key from the OpenAI website and set it as an environment variable: ```python export OPENAI_API_KEY="your-api-key" ``` Let's generate embeddings for our documents and perform a similarity search: ```python import openai import numpy as np from sklearn.metrics.pairwise import cosine_similarity from langchain.embeddings import OpenAIEmbeddings # Define the documents documents = [ "The cat is on the mat.", "There is a cat on the mat.", "The dog is in the yard.", "There is a dog in the yard.", ] # Initialize the OpenAIEmbeddings instance embeddings = OpenAIEmbeddings(model="text-embedding-ada-002") # Generate embeddings for the documents document_embeddings = embeddings.embed_documents(documents) # Perform a similarity search for a given query query = "A cat is sitting on a mat." query_embedding = embeddings.embed_query(query) # Calculate similarity scores similarity_scores = cosine_similarity([query_embedding], document_embeddings)[0] # Find the most similar document most_similar_index = np.argmax(similarity_scores) most_similar_document = documents[most_similar_index] print(f"Most similar document to the query '{query}':") print(most_similar_document) # the output: Most similar document to the query 'A cat is sitting on a mat.': The cat is on the mat. ``` We initialize the OpenAI API client by setting the OpenAI API key. This allows us to use OpenAI's services for generating embeddings. We then define a list of documents as strings. These documents are the text data we want to analyze for semantic similarity. In order to perform this analysis, we need to convert our documents into a format that our similarity computation algorithm can understand. This is where `OpenAIEmbeddings` class comes in. We use it to generate embeddings for each document, transforming them into vectors that represent their semantic content. Similarly, we also transform our query string into an embedding. The query string is the text we want to find the most similar document too. With our documents and query now in the form of embeddings, we compute the cosine similarity between the query embedding and each document embedding. The cosine similarity is a metric


Exploring the World of Embeddings

# Exploring the World of Embeddings
used to determine how similar two vectors are. In our case, it gives us a list of similarity scores for our query against each document. With our similarity scores in hand, we then identify the document most similar to our query. We do this by finding the index of the highest similarity score and retrieving the corresponding document from our list of documents. **Embedding vectors** positioned near each other are regarded as similar. At times, they are directly applied to display related items in online shops. In other instances, they are incorporated into various models to share insights across akin items rather than considering them as entirely distinct entities. This renders embeddings effective in representing aspects like web browsing patterns, textual data, and e-commerce transactions for subsequent model applications. --- ### Embedding Models **Embedding models** are a type of machine learning model that convert discrete data into continuous vectors. In the context of natural language processing, these discrete data points can be words, sentences, or even entire documents. The generated vectors, also known as embeddings, are designed to capture the semantic meaning of the original data. For instance, words that are semantically similar (e.g., 'cat' and 'kitten') would have similar embeddings. These embeddings are dense, which means that they use many dimensions (often hundreds) to capture nuances in meaning. The primary benefit of embeddings is that they allow us to use mathematical operations to reason about semantic meaning. For example, we can calculate the cosine similarity between two embeddings to assess how semantically similar the corresponding words or documents are. We initialize our embedding model. For this task, we've chosen the pre-trained "sentence-transformers/all-mpnet-base-v2" model. This model is designed to transform sentences into embeddings - vectors that encapsulate the semantic meaning of the sentences. The `model_kwargs` parameter is used here to specify that we want our computations to be performed on the CPU. Before executing the subsequent code, make sure to install the Sentence Transformer library by using the command `pip install sentence_transformers===2.2.2`. This library offers powerful pre-trained models designed to generate embedding representations. ```python from langchain.llms import HuggingFacePipeline from langchain.embeddings import HuggingFaceEmbeddings model_name = "sentence-transformers/all-mpnet-base-v2" model_kwargs = {'device': 'cpu'} hf = HuggingFaceEmbeddings(model_name=model_name, model_kwargs=model_kwargs) documents = ["Document 1", "Document 2", "Document 3"] doc_embeddings = hf.embed_documents(documents) ``` Now that we have our model, we define a list of documents - these are the pieces of text that we want to convert into semantic embeddings. With our model and documents ready, we move on to generate the embeddings. We do this by calling the `embed_documents` method on our HuggingFaceEmbeddings instance, passing our list of documents as an argument. This method processes each document and returns a corresponding list of embeddings. These embeddings are now ready for any downstream tasks such as classification, clustering, or similarity analysis. They represent our original documents in a form that machines can understand and process, enabling us to perform complex semantic tasks. --- ### Cohere embeddings Cohere is dedicated to making its innovative multilingual language models accessible to all, thereby democratizing advanced NLP technologies worldwide. Their Multilingual Model, which maps text into a semantic vector space for better text similarity understanding, significantly enhances multilingual applications such as search operations. Unlike their English language model, the multilingual model uses dot product computations resulting in superior performance. These multilingual embeddings are represented in a 768-dimensional vector space. To activate the power of the Cohere API, one needs to acquire an API key. Here's a step-by-step guide to doing so: 1. Visit the **[Cohere Dashboard](https://dashboard.cohere.ai/api-keys)**. 2. If you haven't already, you must either log in or sign up for a Cohere


Exploring the World of Embeddings

# Exploring the World of Embeddings
account. Please note that you agree to adhere to the Terms of Use and Privacy Policy by signing up. 3. When you're logged in, the dashboard provides an intuitive interface to create and manage your API keys. Once we have the API key, we initialize an instance of the CohereEmbeddings class within LangChain, specifying the "embed-multilingual-v2.0" model. We then specify a list of texts in various languages. The `embed_documents()` method is subsequently invoked to generate unique embeddings for each text in the list. To illustrate the results, we print each text alongside its corresponding embedding. For simplicity, we only display the first 5 dimensions of each embedding. You also need to install the cohere package by running the following command `pip install cohere`. ```python import cohere from langchain.embeddings import CohereEmbeddings # Initialize the CohereEmbeddings object cohere = CohereEmbeddings( model="embed-multilingual-v2.0", cohere_api_key="your_cohere_api_key" ) # Define a list of texts texts = [ "Hello from Cohere!", "مرحبًا من كوهير!", "Hallo von Cohere!", "Bonjour de Cohere!", "¡Hola desde Cohere!", "Olá do Cohere!", "Ciao da Cohere!", "您好，来自 Cohere！", "कोहेरे से नमस्ते!" ] # Generate embeddings for the texts document_embeddings = cohere.embed_documents(texts) # Print the embeddings for text, embedding in zip(texts, document_embeddings): print(f"Text: {text}") print(f"Embedding: {embedding[:5]}") # print first 5 dimensions of each embedding ``` Your output should be similar to the following. ```python Text: Hello from Cohere! Embedding: [0.23439695, 0.50120056, -0.048770234, 0.13988855, -0.1800725] Text: مرحبًا من كوهير! Embedding: [0.25350592, 0.29968268, 0.010332941, 0.12572688, -0.18180023] Text: Hallo von Cohere! Embedding: [0.10278442, 0.2838264, -0.05107267, 0.23759139, -0.07176493] Text: Bonjour de Cohere! Embedding: [0.15180704, 0.28215882, -0.056877363, 0.117460854, -0.044658754] Text: ¡Hola desde Cohere! Embedding: [0.2516583, 0.43137372, -0.08623046, 0.24681088, -0.11645193] Text: Olá do Cohere! Embedding: [0.18696906, 0.39113742, -0.046254586, 0.14583701, -0.11280365] Text: Ciao da Cohere! Embedding: [0.1157251, 0.43330532, -0.025885003, 0.14538017, 0.07029742] Text: 您好，来自 Cohere！ Embedding: [0.24605744, 0.3085744, -0.11160592, 0.266223, -0.051633865] Text: कोहेरे से नमस्ते! Embedding: [0.19287698, 0.6350239, 0.032287907, 0.11751755, -0.2598813] ``` LangChain, a comprehensive library designed for language understanding and processing, serves as an ideal conduit for Cohere's advanced language models. It simplifies the integration of Cohere's multilingual embeddings into a developer's workflow, thus enabling a broader range of applications, from semantic search to customer feedback analysis and content moderation, across a multitude of languages. When used in tandem with Cohere, LangChain eliminates the need for complex pipelines, making the process of generating and manipulating high-dimensional embeddings straightforward and efficient. Given a list of multilingual texts, the `embed_documents()` method in LangChain's CohereEmbeddings class, connected to Cohere’s embedding endpoint, can swiftly generate unique semantic embeddings for each text. --- ### Deep Lake Vector Store **Vector stores** are data structures or databases designed to store and manage high-dimensional vectors efficiently. They enable efficient similarity search, nearest neighbor search, and other vector-related operations. Vector stores can be built using various data structures such as approximate nearest neighbor (ANN) techniques, KD trees, or Vantage Point trees. **Deep Lake**, serves as both a data lake for deep learning and a multi-modal vector store. As a **multi-modal vector store**, it allows users to store images, audio, videos, text, and metadata in a format **optimized for deep learning**. It enables hybrid search, allowing users to search both embeddings and their attributes. Users can save data locally, in their cloud, or on Activeloop storage. Deep Lake supports the training of PyTorch and TensorFlow models while streaming data with minimal boilerplate code. It also provides features like version control, dataset queries, and distributed workloads using a simple Python API. Moreover, as the size of datasets increases, it becomes increasingly difficult to store them in local memory. A local vector store could have been utilized in this particular instance since only a few


Exploring the World of Embeddings

# Exploring the World of Embeddings
documents are being uploaded. However, the necessity for a centralized cloud dataset arises in a typical production setting, where thousands or millions of documents may be involved and accessed by various programs. Let’s see how to use Deep Lake for our example. ### Creating Deep Lake Vector Store embeddings example Deep Lake provides well-written documentation, and besides other examples for which they added Jupyter Notebooks, we can follow the one for vector store creation. This task aims to leverage the power of NLP technologies, particularly OpenAI and Deep Lake, to generate and manipulate high-dimensional embeddings. These embeddings can be used for a variety of purposes, such as searching for relevant documents, moderating content, and answering questions. In this case, we will create a Deep Lake database for a retrieval-based question-answering system. First, we need to import the required packages and ensure that the Activeloop and OpenAI keys are stored in the environment variables, ACTIVELOOP_TOKEN and OPENAI_API_KEY. Getting ACTIVELOOP_TOKEN is straightforward, you can easily generate one on the Activeloop page. The installation of the `deeplake` library using `pip`, and the initialization of the OpenAI and Activeloop API keys: ```python pip install deeplake ``` Then make sure to specify the right API keys in the “OPENAI_API_KEY” and “ACTIVELOOP_TOKEN” environmental variables. Next, the necessary modules from the `langchain` package are imported. ```python from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import DeepLake from langchain.text_splitter import RecursiveCharacterTextSplitter from langchain.chat_models import ChatOpenAI from langchain.chains import RetrievalQA ``` We then create some documents using the `RecursiveCharacterTextSplitter` class. ```python # create our documents texts = [ "Napoleon Bonaparte was born in 15 August 1769", "Louis XIV was born in 5 September 1638", "Lady Gaga was born in 28 March 1986", "Michael Jeffrey Jordan was born in 17 February 1963" ] text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0) docs = text_splitter.create_documents(texts) ``` The next step is to create a Deep Lake database and load our documents into it. ```python # initialize embeddings model embeddings = OpenAIEmbeddings(model="text-embedding-ada-002") # create Deep Lake dataset # TODO: use your organization id here. (by default, org id is your username) my_activeloop_org_id = "<YOUR-ACTIVELOOP-ORG-ID>" my_activeloop_dataset_name = "langchain_course_embeddings" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" db = DeepLake(dataset_path=dataset_path, embedding_function=embeddings) # add documents to our Deep Lake dataset db.add_documents(docs) ``` If everything worked correctly, you should see a printed output like this: ```python Your Deep Lake dataset has been successfully created! The dataset is private so make sure you are logged in! ``` We now create a retriever from the database. ```python # create retriever from db retriever = db.as_retriever() ``` Finally, we create a `RetrievalQA` chain in LangChain and run it ```python # istantiate the llm wrapper model = ChatOpenAI(model='gpt-3.5-turbo') # create the question-answering chain qa_chain = RetrievalQA.from_llm(model, retriever=retriever) # ask a question to the chain qa_chain.run("When was Michael Jordan born?") ``` This returns: ```python 'Michael Jordan was born on 17 February 1963.' ``` This pipeline demonstrates how to leverage the power of the LangChain, OpenAI, and Deep Lake libraries and products to create a conversational AI model capable of retrieving and answering questions based on the content of a given repository. Let's break down each step to understand how these technologies work together. 1. **OpenAI and LangChain Integration**: LangChain, a library built for chaining NLP models, is designed to work seamlessly with OpenAI's GPT-3.5-turbo model for language understanding and generation. You've initialized OpenAI embeddings using `OpenAIEmbeddings()`, and these embeddings are later used to transform the text into a high-dimensional vector representation. This vector representation captures the semantic essence of the text and is essential for information retrieval tasks. 2. **Deep Lake**: Deep Lake is a Vector Store for creating, storing, and querying vector


Exploring the World of Embeddings

# Exploring the World of Embeddings
representations (also known as embeddings) of data. 3. **Text Retrieval**: Using the `db.as_retriever()` function, you've transformed the Deep Lake dataset into a retriever object. This object is designed to fetch the most relevant pieces of text from the dataset based on the semantic similarity of their embeddings. 4. **Question Answering**: The final step involves setting up a `RetrievalQA` chain from LangChain. This chain is designed to accept a natural language question, transform it into an embedding, retrieve the most relevant document chunks from the Deep Lake dataset, and generate a natural language answer. The `ChatOpenAI` model, which is the underlying model of this chain, is responsible for both the question embedding and the answer generation. ### Conclusion In conclusion, vector embeddings are a cornerstone in capturing and understanding the rich contextual information in our textual data. This representation becomes increasingly important when dealing with language models like GPT-3.5-turbo, which have a limited token capacity. In this tutorial, we've used embeddings from OpenAI and incorporated embeddings from Hugging Face and Cohere. The former, a well-known AI research organization, provides Transformer-based models that are highly versatile and widely used. Cohere offers innovative multilingual language models that are a significant asset in a globally interconnected world. Building upon these technologies, we've walked through the process of creating a conversational AI application, specifically a Q&A system leveraging Deep Lake. This application demonstrates the potential of these combined technologies - LangChain for chaining together complex NLP tasks, Hugging Face, Cohere, and OpenAI for generating high-quality embeddings, and Deep Lake for managing these embeddings in a vector store. In the next lesson we’ll build a customer support question-answering chatbot leveraging our new knowledge about indexes and retrievers.


Improving Our News Articles Summarizer

# Improving Our News Articles Summarizer
### Introduction The purpose of this lesson is to enhance our previous implementation of a News Article Summarizer. Our objective is to make our tool even more effective at distilling key information from lengthy news articles and presenting that information in an easy-to-digest, bulleted list format. This enhancement will enable users to quickly comprehend the main points of an article in a clear, organized way, thus saving valuable time and enhancing the reading experience. To achieve this, we will modify our existing summarizer to instruct the underlying language model to generate summaries as bulleted lists. This task involves a few changes to the way we present our prompt to the model, which we will guide you through in the workflow below. ### **Workflow for Building a News Articles Summarizer with Bulleted Lists** This is what we are going to doin this project. ![Screenshot 2023-06-09 at 14.00.40.png](Improving%20Our%20News%20Articles%20Summarizer%202923472117e148fb8b0b595fffcd3c87/Screenshot_2023-06-09_at_14.00.40.png) We set up the environment and retrieved the news article. 1. **Install required libraries**: The first step is to ensure that the necessary libraries, namely `requests`, `newspaper3k`, and `LangChain`, are installed. 2. **Scrape articles**: We will use the `requests` library to scrape the content of the target news articles from their respective URLs. 3. **Extract titles and text**: The `newspaper` library will be used to parse the scraped HTML, extracting the titles and text of the articles. 4. **Preprocess the text**: The extracted texts need to be cleaned and preprocessed to make them suitable for input to LLM. The rest of the lesson will explore new possibilities to enhance the application’s performance further. 1. **Use Few-Shot Learning Technique**: We use the few-shot learning technique in this step. This template will provide a few examples of the language model to guide it in generating the summaries in the desired format - a bulleted list. 2. **Generate summaries**: With the modified prompt, we utilize the model to generate concise summaries of the extracted articles' text in the desired format. 3. **Use the Output Parsers**: We employ the Output Parsers to interpret the output from the language model, ensuring it aligns with our desired structure and format. 4. **Output the results**: Finally, we present the bulleted summaries along with the original titles, enabling users to quickly grasp the main points of each article in a structured manner. With these steps, you will be able to construct a powerful tool capable of summarizing news articles into easily digestible, bulleted summaries, employing the `FewShotLearning` technique for added precision and `OutputParsers` for formatting the output using a defined data structure. Let's delve into it! The initial steps of the process are technically the same as part 1 of this lesson. Remember to install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken`. Additionally, install the *newspaper3k* package, which has been tested in this lesson with the version `0.2.8`. ```python !pip install -q newspaper3k python-dotenv ``` Set the API key in your Python script or notebook as an environment variable with the `OPENAI_API_KEY` name. In order to set it from a `.env` file, you can use the `load_dotenv` ****function**.** ```python import os import json from dotenv import load_dotenv load_dotenv() ``` We picked the URL of a news article to generate a summary. The following code fetches articles from a list of URLs using the `requests` ****library with a custom User-Agent header. It then extracts the title and text of each article using the `newspaper` ****library. ```python import requests from newspaper import Article headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36' } article_url = "https://www.artificialintelligence-news.com/2022/01/25/meta-claims-new-ai-supercomputer-will-set-records/" session = requests.Session() try: response


Improving Our News Articles Summarizer

# Improving Our News Articles Summarizer
= session.get(article_url, headers=headers, timeout=10) if response.status_code == 200: article = Article(article_url) article.download() article.parse() print(f"Title: {article.title}") print(f"Text: {article.text}") else: print(f"Failed to fetch article at {article_url}") except Exception as e: print(f"Error occurred while fetching article at {article_url}: {e}") ``` ``` Title: Meta claims its new AI supercomputer will set records Text: Ryan is a senior editor at TechForge Media with over a decade of experience covering the latest technology and interviewing leading industry figures. He can often be sighted at tech conferences with a strong coffee in one hand and a laptop in the other. If it's geeky, he’s probably into it. Find him on Twitter (@Gadget_Ry) or Mastodon (@gadgetry@techhub.social) Meta (formerly Facebook) has unveiled an AI supercomputer that it claims will be the world’s fastest. The supercomputer is called the AI Research SuperCluster (RSC) and is yet to be fully complete. However, Meta’s researchers have already begun using it for training large natural language processing (NLP) and computer vision models. RSC is set to be fully built-in mid-2022. Meta says that it will be the fastest in the world once complete and the aim is for it to be capable of training models with trillions of parameters. “We hope RSC will help us build entirely new AI systems that can, for example, power real-time voice translations to large groups of people, each speaking a different language, so they can seamlessly collaborate on a research project or play an AR game together,” wrote Meta in a blog post. “Ultimately, the work done with RSC will pave the way toward building technologies for the next major computing platform — the metaverse, where AI-driven applications and products will play an important role.” For production, Meta expects RSC will be 20x faster than Meta’s current V100-based clusters. RSC is also estimated to be 9x faster at running the NVIDIA Collective Communication Library (NCCL) and 3x faster at training large-scale NLP workflows. A model with tens of billions of parameters can finish training in three weeks compared with nine weeks prior to RSC. Meta says that its previous AI research infrastructure only leveraged open source and other publicly-available datasets. RSC was designed with the security and privacy controls in mind to allow Meta to use real-world examples from its production systems in production training. What this means in practice is that Meta can use RSC to advance research for vital tasks such as identifying harmful content on its platforms—using real data from them. “We believe this is the first time performance, reliability, security, and privacy have been tackled at such a scale,” says Meta. (Image Credit: Meta) Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo. The next events in the series will be held in Santa Clara on 11-12 May 2022, Amsterdam on 20-21 September 2022, and London on 1-2 December 2022. Explore other upcoming enterprise technology events and webinars powered by TechForge here. ``` ### Few Shot Prompting We saw in the previous lessons how to use `FewShotPromptTemplate`; let's now see another way of adding examples to a prompt that is slightly different but achieves similar results. In this experiment, we include several examples that guide the model's summarization process to generate bullet lists. As a result, the model is expected to generate a bulleted list summarizing the given article. ```python from langchain.schema import ( HumanMessage ) # we get the article data from the scraping part article_title = article.title article_text = article.text # prepare template for prompt template = """ As an advanced AI, you've been tasked to summarize online articles into bulleted


Improving Our News Articles Summarizer

# Improving Our News Articles Summarizer
points. Here are a few examples of how you've done this in the past: Example 1: Original Article: 'The Effects of Climate Change Summary: - Climate change is causing a rise in global temperatures. - This leads to melting ice caps and rising sea levels. - Resulting in more frequent and severe weather conditions. Example 2: Original Article: 'The Evolution of Artificial Intelligence Summary: - Artificial Intelligence (AI) has developed significantly over the past decade. - AI is now used in multiple fields such as healthcare, finance, and transportation. - The future of AI is promising but requires careful regulation. Now, here's the article you need to summarize: ================== Title: {article_title} {article_text} ================== Please provide a summarized version of the article in a bulleted list format. """ # Format the Prompt prompt = template.format(article_title=article.title, article_text=article.text) messages = [HumanMessage(content=prompt)] ``` These examples provide the model with a better understanding of how we want it to respond. Here we have a few important components: **Article data**: The title and text of the article are obtained, which will be used as inputs to the model. **Template preparation**: A template is prepared for the prompt. This template includes a few-shot learning style, where the model is provided with examples of how it has previously converted articles into a bulleted list format. The template also includes placeholders for the actual article title and text that will be summarized. Then, the placeholders in the template (`{article_title}` and `{article_text}`) are replaced with the actual title and text of the article using the `.format()` method. The next step is to use `ChatOpenAI` class to load the GPT-4 model for generating the summary. Then, the formatted prompt is passed to the language model as the input/prompt. The `ChatOpenAI` class's chat instance takes a `HumanMessage` list as an input argument. ```python from langchain.chat_models import ChatOpenAI # load the model chat = ChatOpenAI(model_name="gpt-4", temperature=0.0) # generate summary summary = chat(messages) print(summary.content) ``` ``` - Meta (formerly Facebook) has unveiled an AI supercomputer called the AI Research SuperCluster (RSC). - The RSC is yet to be fully complete but is already being used for training large natural language processing (NLP) and computer vision models. - Meta claims that the RSC will be the fastest in the world once complete and capable of training models with trillions of parameters. - The aim is for the RSC to help build entirely new AI systems that can power real-time voice translations to large groups of people. - Meta expects the RSC to be 20x faster than its current V100-based clusters for production. - The RSC is estimated to be 9x faster at running the NVIDIA Collective Communication Library (NCCL) and 3x faster at training large-scale NLP workflows. - Meta says that its previous AI research infrastructure only leveraged open source and other publicly-available datasets. - RSC was designed with security and privacy controls in mind to allow Meta to use real-world examples from its production systems in production training. - Meta can use RSC to advance research for vital tasks such as identifying harmful content on its platforms using real data from them. ``` The key takeaway here is the use of a few-shot learning style in the prompt. This provides the model with examples of how it should perform the task, which guides it to generate a bulleted list summarizing the article. By modifying the prompt and the examples, you can adjust the model's output to meet various requirements and ensure the model follows a specified format, tone, style, etc. ### Output Parsers Now, let’s improve the previous section by using Output Parsers. The


Improving Our News Articles Summarizer

# Improving Our News Articles Summarizer
Pydantic output parser in LangChain offers a flexible way to shape the outputs from language models according to pre-defined schemas. When used alongside prompt templates, it enables more structured interactions with language models, making it easier to extract and work with the information provided by the model. The prompt template includes the format instructions from our parser, which guide the language model to produce the output in the desired structured format. The idea is to demonstrate how you could use `PydanticOutputParser` class to receive the output as a type `List` that holds each bullet point instead of a string. The advantage of having a list is the possibility to loop through the results or index a specific item. As mentioned before, the `PydanticOutputParser` wrapper is used to create a parser that will parse the output from the string into a data structure. The custom `ArticleSummary` class, which inherits the `Pydantic` package’s `BaseModel` class, will be used to parse the model’s output. We defined the schema to present a `title` along with a `summary` variable that represents a list of strings using the `Field` object. The `description` argument will describe what each variable must represent and help the model to achieve it. Our custom class also includes a validator function to ensure that the generated output contains at least three bullet points. ```python from langchain.output_parsers import PydanticOutputParser from pydantic import validator from pydantic import BaseModel, Field from typing import List # create output parser class class ArticleSummary(BaseModel): title: str = Field(description="Title of the article") summary: List[str] = Field(description="Bulleted list summary of the article") # validating whether the generated summary has at least three lines @validator('summary', allow_reuse=True) def has_three_or_more_lines(cls, list_of_lines): if len(list_of_lines) < 3: raise ValueError("Generated summary has less than three bullet points!") return list_of_lines # set up output parser parser = PydanticOutputParser(pydantic_object=ArticleSummary) ``` The next step involves creating a template for the input prompt that instructs the language model to summarize the news article into bullet points. This template is used to instantiate a `PromptTemplate` object, which is responsible for correctly formatting the prompts that are sent to the language model. The `PromptTemplate` uses our custom parser to format the prompt sent to the language model using the `.get_format_instructions()` method, which will include additional instructions on how the output should be shaped. ```python from langchain.prompts import PromptTemplate # create prompt template # notice that we are specifying the "partial_variables" parameter template = """ You are a very good assistant that summarizes online articles. Here's the article you want to summarize. ================== Title: {article_title} {article_text} ================== {format_instructions} """ prompt = PromptTemplate( template=template, input_variables=["article_title", "article_text"], partial_variables={"format_instructions": parser.get_format_instructions()} ) # Format the prompt using the article title and text obtained from scraping formatted_prompt = prompt.format_prompt(article_title=article_title, article_text=article_text) ``` Lastly, the `GPT-3` model with the temperature set to `0.0` is initialized, which means the output will be deterministic, favoring the most likely outcome over randomness/creativity. The parser object then converts the string output from the model to a defined schema using the `.parse()` method. ```python from langchain.llms import OpenAI # instantiate model class model = OpenAI(model_name="text-davinci-003", temperature=0.0) # Use the model to generate a summary output = model(formatted_prompt.to_string()) # Parse the output into the Pydantic model parsed_output = parser.parse(output) print(parsed_output) ``` ```python ArticleSummary(title='Meta claims its new AI supercomputer will set records', summary=['Meta (formerly Facebook) has unveiled an AI supercomputer that it claims will be the world’s fastest.', 'The supercomputer is called the AI Research SuperCluster (RSC) and is yet to be fully complete.', 'Meta says that it will be the fastest in the world once complete and the aim is for it to be capable of


Improving Our News Articles Summarizer

# Improving Our News Articles Summarizer
training models with trillions of parameters.', 'For production, Meta expects RSC will be 20x faster than Meta’s current V100-based clusters.', 'Meta says that its previous AI research infrastructure only leveraged open source and other publicly-available datasets.', 'What this means in practice is that Meta can use RSC to advance research for vital tasks such as identifying harmful content on its platforms—using real data from them.']) ``` The Pydantic output parser is a powerful method for molding and structuring the output from language models. It uses the Pydantic library, known for its data validation capabilities, to define and enforce data schemas for the model's output. This is a recap of what we did: - We defined a Pydantic data structure named `ArticleSummary`. This model serves as a blueprint for the desired structure of the generated article summary. It comprises fields for the title and the summary, which is expected to be a list of strings representing bullet points. Importantly, we incorporate a validator within this model to ensure the summary comprises at least three points, thereby maintaining a certain level of detail in the summarization. - We then instantiate a parser object using our `ArticleSummary` class. This parser plays a crucial role in ensuring the output generated by the language model aligns with the defined structures of our custom schema. - To direct the language model's output, we create the prompt template. The template instructs the model to act as an assistant that summarizes online articles by incorporating the parser object. - So, output parsers enable us to specify the desired format of the model's output, making extracting meaningful information from the model's responses easier. ### **Conclusion** In today's lesson, we've successfully navigated the path of crafting our News Articles Summarizer leveraging the potential of `PromptTemplates` and `OutputParsers`, showing the capabilities of prompt handling LangChain. The Pydantic output parser is a powerful method for molding and structuring the output from language models. It uses the Pydantic library, known for its data validation capabilities, to define and enforce data schemas for the model's output. Following this, we define a Pydantic model named "ArticleSummary.” This model serves as a blueprint for the desired structure of the generated article summary. It comprises fields for the title and the summary, which is expected to be a list of strings representing bullet points. Importantly, we incorporate a validator within this model to ensure the summary comprises at least three points, thereby maintaining a certain level of detail in the summarization. We then instantiate a PydanticOutputParser, passing it to the "ArticleSummary" model. This parser plays a crucial role in ensuring the output generated by the language model aligns with the structure outlined in the "Article Summary" model. A good understanding of prompt and output design nuances equips you to customize the model to produce results that perfectly match your specific requirements. In the next lesson, we’ll do a project where we create a knowledge graph from textual data, making complex information more accessible and easier to understand. --- You can find the code of this lesson in this online [Notebook](https://colab.research.google.com/drive/1-vwClDntLTumSgrT_-PAk55lvFUYLo7n?usp=sharing).


Introduction to Chains

# Introduction to Chains
Welcome to fifth module! Up to this point, the course covered all the fundamental functionalities of the LangChain libraries in previous modules. During the past lessons, we explored various kinds of language models, learned techniques to maximize their effectiveness through effective prompts, and discovered methods to provide context by leveraging external resources. In this module, we will delve into the concept of Chains, which introduces an abstraction layer that builds upon the previously discussed concepts. Chains offer a seamless interface for accomplishing a multitude of tasks out of the box. Here are the lessons you’ll find in this module and what you’ll learn: - **Chains and Why They Are Used:** In our first lesson of the Chains module, we will explore the effectiveness of prompting techniques, which enable natural language querying of these models. We delve deeper into the concept of chains, which provide an end-to-end pipeline for utilizing language models. These chains seamlessly integrate models, prompts, memory, parsing output, and debugging capabilities, offering a user-friendly interface. By inheriting the Chain class, we learned how to design custom pipelines, exemplified by the LLMChain in LangChain. - **Create a YouTube Video Summarizer Using Whisper and LangChain:** Building upon the capabilities of language models, our next lesson introduces a solution to summarize YouTube videos. We acknowledge the overwhelming abundance of information and the time constraints that often hinder our ability to consume it all. Whisper and LangChain come to the rescue as cutting-edge tools for video summarization. Whisper, a sophisticated automatic speech recognition (ASR) system, transcribes voice inputs into text. Leveraging LangChain's summarization techniques, such as `stuff`, `refine`, and `map_reduce`, we can effectively extract key takeaways from lengthy videos. The customizability of LangChain further enhances the summarization process, allowing personalized prompts, multilingual summaries, and storage of URLs in a Deep Lake vector store. This advanced solution empowers users to save time while improving knowledge retention and understanding across various topics. - **Creating a Voice Assistant for your Knowledge Base:** Expanding the realm of language models, the next lesson ventures into the development of a voice assistant powered by artificial intelligence tools. Whisper plays an important role as an ASR system, transcribing voice inputs into text. The voice assistant employs Eleven Labs to generate engaging and natural voice outputs. The heart of the project is a robust question-answering mechanism that utilizes a vector database housing relevant documents. The voice assistant generates precise and timely responses by feeding these documents and the user's questions to the language model. This comprehensive voice assistant project showcases the synergy between ASR systems, language models, and question-answering mechanisms. - ****LangChain & GPT-4 for Code Understanding: Twitter Algorithm:**** Moving beyond textual data, the next lesson delves into the realm of code comprehension. LangChain, in conjunction with Deep Lake and GPT-4, provides a transformative approach to understanding complex codebases. LangChain, as a wrapper for large language models, makes them more accessible and usable, particularly in the context of codebases. Deep Lake, a serverless and open-source vector store, plays an important role in storing embeddings and original data with version control. The Conversational Retriever Chain interacts with the codebase stored in Deep Lake, retrieving relevant code snippets based on user queries. This lesson demonstrates how LangChain, Deep Lake, and GPT-4 can revolutionize code comprehension and facilitate insightful interactions with codebases. - **3 ways to build a recommendation engine for songs with LangChain:** Our next lesson delves into the realm of recommendation engines, where we leverage LangChain's power to craft a song recommendation engine. Large Language Models (LLMs) and vector databases enrich the user experience, focusing on the case study of 'FairyTaleDJ,'


Introduction to Chains

# Introduction to Chains
a web app suggesting Disney songs based on user emotions. Encoding methods, data management, and matching user input are the core areas of exploration. Employing LLMs to encode data makes the retrieval process faster and more efficient. Through the case study, we learn from successes and failures, gaining insights into constructing emotion-responsive recommendation engines with LangChain. - **Guarding Against Undesirable Outputs with the Self-Critique Chain:** While language models have remarkable capabilities, they can occasionally generate undesirable outputs. Our final lesson addresses this issue by introducing the self-critique chain, which acts as a mechanism to ensure model responses are appropriate in a production environment. By iterating over the model's output and checking against predefined expectations, the self-critique chain prompts the model to correct itself when necessary. This approach ensures ethical and responsible behavior, such as student mentoring. Happy learning!


# DataChad: an AI App with LangChain & Deep Lake to Chat with Any Data
### Introduction In this lesson we'll delve into DataChad, an open-source project that permits users to ask questions about any data source. DataChad is an open-source project that enables querying any data source, from local files to URLs, using LangChain, embeddings, Deep Lake, and large language models (LLMs) like GPT-3.5-turbo or GPT-4. Recently, DataChad's capabilities have been expanded to include local deployment using GPT4all. This allows all data to be processed locally without making any API calls, providing enhanced privacy and data security. This lesson will showcase how DataChad can simplify data querying and highlight its potential for on-premises deployment in enterprise settings. We'll delve into the integration of LLMs, vector similarity, and the recently introduced local deployment feature. So, whether you need a deep dive into complex data or swift insights, DataChad offers a new level of efficiency. Let's get started! ### The Workflow The workflow for building an All-In-One Chat with Anything App consists of three main parts: 1. **The Streamlit App**: Defined in app.py, this serves as the user interface for the application, allowing users to interact with the system. 2. **The Processing Functions**: Located in utils.py, this section contains all the crucial processing functionality and API calls. This part enables the extraction, transformation, and loading (ETL) of data and interacts with external APIs to perform complex tasks such as language model inferencing or database querying. 3. **The Constants**: Defined in constants.py, this part of the workflow includes project-specific paths, names, and descriptions. This section is crucial for the application's configuration, ensuring consistency and ease of modifications or updates across the project. [https://www.activeloop.ai/resources/data-chad-an-ai-app-with-lang-chain-deep-lake-to-chat-with-any-data/](https://www.activeloop.ai/resources/data-chad-an-ai-app-with-lang-chain-deep-lake-to-chat-with-any-data/)


# Creating a Voice Assistant for your Knowledge Base
### Introduction We are going to create a voice assistant for your knowledge base! This lesson will outline how you can develop your very own voice assistant employing state-of-the-art artificial intelligence tools. The voice assistant utilizes OpenAI's Whisper, a sophisticated automatic speech recognition (ASR) system. Whisper effectively transcribes our voice inputs into text. Once our voice inputs have been transcribed into text, we turn our attention towards generating voice outputs. To accomplish this, we employ Eleven Labs, which enables the voice assistant to respond to the users in an engaging and natural manner. The core of the project revolves around a robust question-answering mechanism. This process initiates with loading the vector database, a repository housing several documents relevant to our potential queries. On posing a question, the system retrieves the documents from this database and, along with the question, feeds them to the LLM. The LLM then generates the response based on retrieved documents. We aim to create a voice assistant that can efficiently navigate a knowledge base, providing precise and timely responses to a user's queries. For this experiment we’re using the ‘JarvisBase’ repository on GitHub. [GitHub - peterw/JarvisBase: Question-answering chatbot using OpenAI's GPT-3.5-turbo model, DeepLake for the vector database, and the Whisper API for voice transcription. The chatbot also uses Eleven Labs to generate audio responses.](https://github.com/peterw/JarvisBase)


# Creating a Voice Assistant for your Knowledge Base
## **Setup:**
### Library Installation We’d start by installing the requirements. These are the necessary libraries that we’ll be using. While we strongly recommend installing the latest versions of these packages, please note that the codes have been tested with the versions specified in parentheses. ``` langchain==0.0.208 deeplake==3.6.5 openai==0.27.8 tiktoken==0.4.0 elevenlabs==0.2.18 streamlit==1.23.1 beautifulsoup4==4.11.2 audio-recorder-streamlit==0.0.8 streamlit-chat==0.0.2.2 ``` ### Tokens and APIs For this experiment, you’d need to obtain several API keys and tokens. They need to be set in the environment variable as described below. ```python import os os.environ['OPENAI_API_KEY']='<your-openai-api-key>' os.environ['ELEVEN_API_KEY']='<your-eleven-api-key>' os.environ['ACTIVELOOP_TOKEN']='<your-activeloop-token>' ``` To access OpenAI's services, you must first obtain credentials by signing up on their website, completing the registration process, and creating an `API key` from your dashboard. This enables you to leverage OpenAI's powerful capabilities in your projects. 1. If you don't have an account yet, create one by going to **[https://platform.openai.com/](https://platform.openai.com/)**. If you already have an account, skip to step 5. 2. Fill out the registration form with your name, email address, and desired password. 3. OpenAI will send you a confirmation email with a link. Click on the link to confirm your account. 4. Please note that you'll need to verify your email account and provide a phone number for verification. 5. Log in to **[https://platform.openai.com/](https://platform.openai.com/)**. 6. Navigate to the API key section at **[https://platform.openai.com/account/api-keys](https://platform.openai.com/account/api-keys)**. 7. Click "Create new secret key" and give the key a recognizable name or ID. **To get the ELEVEN_API_KEY, follow these steps**: 1. Go to [https://elevenlabs.io/](https://elevenlabs.io/) and click on "Sign Up" to create an account. 2. Once you have created an account, log in and navigate to the "API" section. 3. Click the "Create API key" button and follow the prompts to generate a new API key. 4. Copy the API key and paste it into your code where it says "your-eleven-api-key" in the ELEVEN_API_KEY variable. **For ACTIVELOOP TOKEN, follow these easy steps:** 1. Go to [https://www.activeloop.ai/](https://www.activeloop.ai/) and click on “Sign Up” to create an account. 2. Once you have an Activeloop account, you can create tokens in the Deep Lake App (Organization Details -> API Tokens) 3. Click the "Create API key" button and generate a new API Token. 1. Copy the API key and paste it as your environment variable: ACTIVELOOP_TOKEN='your-Activeloop-token' ---


# Creating a Voice Assistant for your Knowledge Base
## 1. Sourcing Content from Hugging Face Hub
Now that everything is set up, let’s begin by aggregating all Python library articles from the Hugging Face Hub, an open platform to share, collaborate and advance in machine learning. These articles will serve as the knowledge base for our voice assistant. We'll do some web scraping in order to collect some knowledge documents. Let’s observe and run the `script.py` file (i.e. run `python scrape.py` ). This script contains all the code we use in this lesson under the “Sourcing Content from Hugging Face Hub” and “Embedding and storing in Deep Lake” sections. You can fork or download the mentioned [repository](https://github.com/peterw/JarvisBase) and run the files. We start with importing necessary modules, loading environment variables, and setting up the path for Deep Lake, a vector database. It also sets up an `OpenAIEmbeddings` instance, which will be used later to embed the scraped articles: ```python import os import requests from bs4 import BeautifulSoup from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import DeepLake from langchain.text_splitter import CharacterTextSplitter from langchain.document_loaders import TextLoader import re # TODO: use your organization id here. (by default, org id is your username) my_activeloop_org_id = "<YOUR-ACTIVELOOP-ORG-ID>" my_activeloop_dataset_name = "langchain_course_jarvis_assistant" dataset_path= 'hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}' embeddings = OpenAIEmbeddings(model_name="text-embedding-ada-002") ``` We first create a list of relative URLs leading to knowledge documents hosted on the Hugging Face Hub. To do this, we define a function called `get_documentation_urls()`. Using another function, `construct_full_url()`, we then append these relative URLs to the base URL of the Hugging Face Hub, effectively creating full URLs that we can access directly. ```python def get_documentation_urls(): # List of relative URLs for Hugging Face documentation pages, commented a lot of these because it would take too long to scrape all of them return [ '/docs/huggingface_hub/guides/overview', '/docs/huggingface_hub/guides/download', '/docs/huggingface_hub/guides/upload', '/docs/huggingface_hub/guides/hf_file_system', '/docs/huggingface_hub/guides/repository', '/docs/huggingface_hub/guides/search', # You may add additional URLs here or replace all of them ] def construct_full_url(base_url, relative_url): # Construct the full URL by appending the relative URL to the base URL return base_url + relative_url ``` The script then aggregates all the scraped content from the URLs. This is achieved with the `scrape_all_content()` function, which iteratively calls `scrape_page_content()` for each URL and extracts its text. This collected text is then saved to a file. ```python def scrape_page_content(url): # Send a GET request to the URL and parse the HTML response using BeautifulSoup response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') # Extract the desired content from the page (in this case, the body text) text=soup.body.text.strip() # Remove non-ASCII characters text = re.sub(r'[\x00-\x08\x0b-\x0c\x0e-\x1f\x7f-\xff]', '', text) # Remove extra whitespace and newlines text = re.sub(r'\s+', ' ', text) return text.strip() def scrape_all_content(base_url, relative_urls, filename): # Loop through the list of URLs, scrape content and add it to the content list content = [] for relative_url in relative_urls: full_url = construct_full_url(base_url, relative_url) scraped_content = scrape_page_content(full_url) content.append(scraped_content.rstrip('\n')) # Write the scraped content to a file with open(filename, 'w', encoding='utf-8') as file: for item in content: file.write("%s\n" % item) return content ``` ### Loading and splitting texts To prepare the collected text for embedding into our vector database, we load the content from the file and split it into separate documents using the `load_docs()` function. To further refine the content, we split it into individual chunks through the `split_docs()`. Here we’d see a Text loader and text_splitter in action. The instruction`text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)` creates an instance of a text splitter that splits the text into chunks based on characters. Each document in `docs` is split into chunks of approximately 1000 characters, with no overlap between consecutive chunks. ```python # Define a function to load documents from a file def load_docs(root_dir,filename): # Create an empty list to hold the documents


# Creating a Voice Assistant for your Knowledge Base
## 1. Sourcing Content from Hugging Face Hub
docs = [] try: # Load the file using the TextLoader class and UTF-8 encoding loader = TextLoader(os.path.join( root_dir, filename), encoding='utf-8') # Split the loaded file into separate documents and add them to the list of documents docs.extend(loader.load_and_split()) except Exception as e: # If an error occurs during loading, ignore it and return an empty list of documents pass # Return the list of documents return docs def split_docs(docs): text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) return text_splitter.split_documents(docs) ```


# Creating a Voice Assistant for your Knowledge Base
## **2. Embedding and storing in Deep Lake**
Once we've collected the necessary articles, the next step is to embed them using Deep Lake. Deep Lake is a powerful tool for creating searchable vector databases. In this context, it will allow us to efficiently index and retrieve the information contained in our Python library articles. Finally, we're ready to populate our vector database. The `Deep Lake` integration initializes a database instance with the given dataset path and the predefined OpenAIEmbeddings function. The `OpenAIEmbeddings` is converting the text chunks into their embedding vectors, a format suitable for the vector database. The `.add_documents` method will process and store the texts on the database. ```python # Define the main function def main(): base_url = 'https://huggingface.co' # Set the name of the file to which the scraped content will be saved filename='content.txt' # Set the root directory where the content file will be saved root_dir ='./' relative_urls = get_documentation_urls() # Scrape all the content from the relative URLs and save it to the content file content = scrape_all_content(base_url, relative_urls,filename) # Load the content from the file docs = load_docs(root_dir,filename) # Split the content into individual documents texts = split_docs(docs) # Create a DeepLake database with the given dataset path and embedding function db = DeepLake(dataset_path=dataset_path, embedding_function=embeddings) # Add the individual documents to the database db.add_documents(texts) # Clean up by deleting the content file os.remove(filename) # Call the main function if this script is being run as the main program if __name__ == '__main__': main() ``` All these steps are neatly wrapped into our main function. This sets the necessary parameters, invokes the functions we've defined, and oversees the overall process from scraping the content from the web to loading it into the Deep Lake database. As a final step, it deletes the content file to clean up.


# 3. Voice Assistant
Having successfully stored all the necessary data in the vector database, in this instance using Deep Lake by Activeloop, we're ready to utilize this data in our chatbot. Without further ado, let's transition to the coding part of our chatbot. The following code can be found in the `chat.py` file of the directory. To give it a try, run `streamlit run chat.py`. These libraries will help us in building web applications with Streamlit, handling audio input, generating text responses, and effectively retrieving information stored in the Deep Lake: ```python import os import openai import streamlit as st from audio_recorder_streamlit import audio_recorder from elevenlabs import generate from langchain.chains import RetrievalQA from langchain.chat_models import ChatOpenAI from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import DeepLake from streamlit_chat import message # Constants TEMP_AUDIO_PATH = "temp_audio.wav" AUDIO_FORMAT = "audio/wav" # Load environment variables from .env file and return the keys openai.api_key = os.environ.get('OPENAI_API_KEY') eleven_api_key = os.environ.get('ELEVEN_API_KEY') ``` We then create an instance that points to our Deep Lake vector database. ```python def load_embeddings_and_database(active_loop_data_set_path): embeddings = OpenAIEmbeddings() db = DeepLake( dataset_path=active_loop_data_set_path, read_only=True, embedding_function=embeddings ) return db ``` Next, we prepare the code for transcribing audio. ```python # Transcribe audio using OpenAI Whisper API def transcribe_audio(audio_file_path, openai_key): openai.api_key = openai_key try: with open(audio_file_path, "rb") as audio_file: response = openai.Audio.transcribe("whisper-1", audio_file) return response["text"] except Exception as e: print(f"Error calling Whisper API: {str(e)}") return None ``` This transcribes an audio file into text using the OpenAI Whisper API, requiring the path of the audio file and the OpenAI key as input parameters. ```python # Record audio using audio_recorder and transcribe using transcribe_audio def record_and_transcribe_audio(): audio_bytes = audio_recorder() transcription = None if audio_bytes: st.audio(audio_bytes, format=AUDIO_FORMAT) with open(TEMP_AUDIO_PATH, "wb") as f: f.write(audio_bytes) if st.button("Transcribe"): transcription = transcribe_audio(TEMP_AUDIO_PATH, openai.api_key) os.remove(TEMP_AUDIO_PATH) display_transcription(transcription) return transcription # Display the transcription of the audio on the app def display_transcription(transcription): if transcription: st.write(f"Transcription: {transcription}") with open("audio_transcription.txt", "w+") as f: f.write(transcription) else: st.write("Error transcribing audio.") # Get user input from Streamlit text input field def get_user_input(transcription): return st.text_input("", value=transcription if transcription else "", key="input") ``` This part of the code allows users to record audio directly within the application. The recorded audio is then transcribed into text using the Whisper API, and the transcribed text is displayed on the application. If any issues occur during the transcription process, an error message will be shown to the user. ```python # Search the database for a response based on the user's query def search_db(user_input, db): print(user_input) retriever = db.as_retriever() retriever.search_kwargs['distance_metric'] = 'cos' retriever.search_kwargs['fetch_k'] = 100 retriever.search_kwargs['maximal_marginal_relevance'] = True retriever.search_kwargs['k'] = 4 model = ChatOpenAI(model_name='gpt-3.5-turbo') qa = RetrievalQA.from_llm(model, retriever=retriever, return_source_documents=True) return qa({'query': user_input}) ``` This segment of the code is for searching the vector database for the most relevant responses to the user's query. It first converts the database into a retriever, which is a tool that searches for the nearest embeddings in the vector space. It then sets various parameters for the search, such as the metric to use when measuring distance in the embedding space, the number of documents to fetch initially, whether or not to use maximal marginal relevance to balance the diversity and relevance of the results, and how many results to return. The retrieved results are then passed through the language model, which is GPT-3.5 Turbo in this case, to generate the most appropriate response to the user's query. ### **Streamlit** Streamlit is a Python framework used for building data visualization web applications. It provides an intuitive way to create interactive web apps for machine learning and data science projects. Now we have the part with the conversation history between the user and the chatbot using Streamlit's


# 3. Voice Assistant
messaging functionality. It goes through the previous messages in the conversation and displays each user message followed by the corresponding chatbot response. It employs the Eleven Labs API to convert the chatbot's text response into speech and give the chatbot a voice. This voice output, in MP3 format, is then played on the Streamlit interface, adding an auditory dimension to the conversation: ```python # Display conversation history using Streamlit messages def display_conversation(history): for i in range(len(history["generated"])): message(history["past"][i], is_user=True, key=str(i) + "_user") message(history["generated"][i],key=str(i)) #Voice using Eleven API voice= "Bella" text= history["generated"][i] audio = generate(text=text, voice=voice,api_key=eleven_api_key) st.audio(audio, format='audio/mp3') ``` ### **User Interaction** After the knowledge base is set up, the next stage is user interaction. The voice assistant is designed to accept queries either in the form of voice recordings or typed text. ```python # Main function to run the app def main(): # Initialize Streamlit app with a title st.write("# JarvisBase 🧙") # Load embeddings and the DeepLake database db = load_embeddings_and_database(dataset_path) # Record and transcribe audio transcription = record_and_transcribe_audio() # Get user input from text input or audio transcription user_input = get_user_input(transcription) # Initialize session state for generated responses and past messages if "generated" not in st.session_state: st.session_state["generated"] = ["I am ready to help you"] if "past" not in st.session_state: st.session_state["past"] = ["Hey there!"] # Search the database for a response based on user input and update the session state if user_input: output = search_db(user_input, db) print(output['source_documents']) st.session_state.past.append(user_input) response = str(output["result"]) st.session_state.generated.append(response) #Display conversation history using Streamlit messages if st.session_state["generated"]: display_conversation(st.session_state) # Run the main function when the script is executed if __name__ == "__main__": main() ``` This is the main driver of the entire application. First, it sets up the Streamlit application and loads the Deep Lake vector database along with its embeddings. It then offers two methods for user input: through text or through an audio recording which is then transcribed. The application keeps a record of past user inputs and generated responses in a session state. When new user input is received, the application searches the database for the most suitable response. This response is then added to the session state. Finally, the application displays the entire conversation history, including both user inputs and chatbot responses. If the input was made via voice, the chatbot's responses are also generated in an audio format using the Eleven Labs API. You should now run the following command in your terminal: ```python streamlit run chat.py ``` When you run your application using the Streamlit command, the application will start a local web server and provide you with a URL where your application is running and can be accessed via a web browser. In your case, you have two URLs: a Network URL and an External URL. Your application will be running as long as the command is running in your terminal, and it will stop once you stop the command (ctrl+Z) or close the terminal. ### Trying Out the UI We have now explained the main code parts and are ready to test the Streamlit app! This is how it presents itself. ![Screenshot 2023-05-19 at 11.12.55.png](Creating%20a%20Voice%20Assistant%20for%20your%20Knowledge%20Base%202ca8576e55f74b13a6ad2a23f111614a/Screenshot_2023-05-19_at_11.12.55.png) By clicking on the microphone icon, your microphone will be active for some seconds and you’ll be able to ask a question. Let’s try “How do I search for models in the Hugging Face Hub?”. After a few seconds, the app will show an audio player that can be used to listen to your registered audio. You may then click on the “Transcribe” button. ![Screenshot 2023-05-19 at 11.25.52.png](Creating%20a%20Voice%20Assistant%20for%20your%20Knowledge%20Base%202ca8576e55f74b13a6ad2a23f111614a/Screenshot_2023-05-19_at_11.25.52.png) This button will invoke a call to the Whisper API and transcribe your audio. The produced


# 3. Voice Assistant
text will be soon pasted to the chat text entry underneath. ![Screenshot 2023-05-19 at 11.27.41.png](Creating%20a%20Voice%20Assistant%20for%20your%20Knowledge%20Base%202ca8576e55f74b13a6ad2a23f111614a/Screenshot_2023-05-19_at_11.27.41.png) Here we see that the Whisper API didn’t do a perfect job at transcribing “Hugging Face” correctly and instead wrote “Huggy Face”. This is unwanted, but let’s see if ChatGPT is still able to understand the query and give it an appropriate answer by leveraging the knowledge documents stored in Deep Lake. After a few more seconds, the underlying chat will be populated with your audio transcription, along with the chatbot's textual response and its audio version, generated by calling the ElevenLabs API. As we can see, ChatGPT was smart enough to understand that “Huggy Face” was a misspelling of “Hugging Face” and was still able to give an appropriate answer. ![Screenshot 2023-05-19 at 11.30.00.png](Creating%20a%20Voice%20Assistant%20for%20your%20Knowledge%20Base%202ca8576e55f74b13a6ad2a23f111614a/Screenshot_2023-05-19_at_11.30.00.png) ### Conclusion In this lesson we integrated several popular generative AI tools and models, namely OpenAI Whisper and ElevenLabs text-to-speech. In the next lesson we’ll see how LLMs can be used to aid in understanding new codebases, such as the Twitter Algorithm public repository. ### Github Repo: [GitHub - peterw/JarvisBase: Question-answering chatbot using OpenAI's GPT-3.5-turbo model, DeepLake for the vector database, and the Whisper API for voice transcription. The chatbot also uses Eleven Labs to generate audio responses.](https://github.com/peterw/JarvisBase)


# 3 ways to build a recommendation engine for songs with LangChain
Community member **Francesco Saverio Zuppichini** contributed to this course entry 😇 ### Introduction Welcome to the lesson on crafting a song recommendation engine with LangChain. We'll explore the use of Large Language Models (LLMs) and vector databases to enrich user experience through a case study of 'FairyTaleDJ,' a unique web app that suggests Disney songs based on user emotions. We'll demonstrate how to use LLMs to encode data, enhancing the retrieval process and making it faster and more efficient. By the end of this lesson, you'll have explored three strategies for constructing an emotion-responsive recommendation engine, and learned from their successes and failures. Our focus will be on three core areas: data management, encoding methods, and matching user input to generate fitting song recommendations. Get ready for the journey through the innovative world of recommendation engines with LangChain. ### The Workflow Building a song recommendation engine using LangChain involves data collection, encoding, and matching. We scrape Disney song lyrics and gather their Spotify URLs. Using Activeloop Deep Lake Vector Database in LangChain, we convert the lyrics into embedded data with relevant metadata. For matching songs to user input, we convert both song lyrics and user inputs into a list of emotions with the help of the OpenAI model. These emotions are embedded and stored in Deep Lake. A similarity search is then conducted in the vector database based on these emotions to provide song recommendations. We filter out low-scoring matches and ensure the same song isn't recommended twice to add variation. Finally, we create a user-friendly interface using Streamlit and host it on Hugging Face Spaces. [https://www.activeloop.ai/resources/3-ways-to-build-a-recommendation-engine-for-songs-with-lang-chain/](https://www.activeloop.ai/resources/3-ways-to-build-a-recommendation-engine-for-songs-with-lang-chain/)


# Create a YouTube Video Summarizer Using Whisper and LangChain
We previously explored the powerful feature of LangChain called chains, which allow for the creation of an end-to-end pipeline for using language models. We learned how chains combine multiple components such as models, prompts, memory, parsing output, and debugging to provide a user-friendly interface. We also discussed the process of designing custom pipelines by inheriting the Chain class and explored the LLMChain as a simple example. This lesson served as a foundation for future lessons, where we will apply these concepts to a hands-on project of summarizing a YouTube video. During this lesson, we delved into the challenge of summarizing YouTube videos efficiently in the context of the digital age. It will introduce two cutting-edge tools, Whisper and LangChain, that can help tackle this issue. We will discuss the strategies of "stuff," "map-reduce," and "refine" for handling large amounts of text and extracting valuable information. It is possible to effectively extract key takeaways from videos by leveraging Whisper to transcribe YouTube audio files and utilizing LangChain's summarization techniques, including stuff, refine, and map_reduce. We also highlighted the customizability of LangChain, allowing personalized prompts, multilingual summaries, and storage of URLs in a Deep Lake vector store. By implementing these advanced tools, you can save time, enhance knowledge retention, and improve your understanding of various topics. Enjoy the tailored experience of data storage and summarization with LangChain and Whisper. The following diagram explains what we are going to do in this project. ![Screenshot 2023-06-09 at 13.48.43.png](Create%20a%20YouTube%20Video%20Summarizer%20Using%20Whisper%20an%20c9f694963d044bd8bfe5cdc6a1ccbf71/Screenshot_2023-06-09_at_13.48.43.png) First, we download the youtube video we are interested in and transcribe it using Whisper. Then, we’ll proceed by creating summaries using two different approaches: 1. First we use an existing summarization chain to generate the final summary, which automatically manages embeddings and prompts. 2. Then, we use another approach more step-by-step to generate a final summary formatted in bullet points, consisting in splitting the transcription into chunks, computing their embeddings, and preparing ad-hoc prompts. ### Introduction In the digital era, the abundance of information can be overwhelming, and we often find ourselves scrambling to consume as much content as possible within our limited time. YouTube is a treasure trove of knowledge and entertainment, but it can be challenging to sift through long videos to extract the key takeaways. Worry not, as we've got your back! In this lesson, we will unveil a powerful solution to help you efficiently summarize YouTube videos using two cutting-edge tools: Whisper and LangChain. ![sumlangchain.png](Create%20a%20YouTube%20Video%20Summarizer%20Using%20Whisper%20an%20c9f694963d044bd8bfe5cdc6a1ccbf71/sumlangchain.png) We will guide you through the process of downloading a YouTube audio file, transcribing it using Whisper, and then summarizing the transcribed text with LangChain's innovative stuff, refine, and map_reduce techniques. ### Workflow: 1. Download the YouTube audio file. 2. Transcribe the audio using Whisper. 3. Summarize the transcribed text using LangChain with three different approaches: stuff, refine, and map_reduce. 4. Adding multiple URLs to DeepLake database, and retrieving information. **Installations:** Remember to install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken`. Additionally, install also the *yt_dlp* and *openai-whisper* packages, which have been tested in this lesson with versions `2023.6.21` and `20230314`, respectively. ```python !pip install -q yt_dlp !pip install -q git+https://github.com/openai/whisper.git ``` Then, we must install the ffmpeg application, which is one of the requirements for the yt_dlp package. This application is installed on Google Colab instances by default. The following commands show the installation process on Mac and Ubuntu operating systems. ```bash # MacOS (requires https://brew.sh/) #brew install ffmpeg # Ubuntu #sudo apt install ffmpeg ``` You can read the following article if you're working on an operating system that hasn't been mentioned earlier (like Windows). It contains comprehensive,


# Create a YouTube Video Summarizer Using Whisper and LangChain
step-by-step instructions on "[How to install ffmpeg](https://www.hostinger.com/tutorials/how-to-install-ffmpeg).” Next step is to add the API key for OpenAI and Deep Lake services in the environment variables. You can either use the `load_dotenv` function to read the values from a `.env` file, or by running the following code. Remember that the API keys must remain private since anyone with this information can access these services on your behalf. ```python import os os.environ['OPENAI_API_KEY'] = "<OPENAI_API_KEY>" os.environ['ACTIVELOOP_TOKEN'] = "<ACTIVELOOP_TOKEN>" ``` For this experiment, we have selected a video featuring Yann LeCun, a distinguished computer scientist and AI researcher. In this engaging discussion, LeCun delves into the challenges posed by large language models. The `download_mp4_from_youtube()` function will download the best quality mp4 video file from any YouTube link and save it to the specified path and filename. We just need to copy/paste the selected video’s URL and pass it to mentioned function. ```python import yt_dlp def download_mp4_from_youtube(url): # Set the options for the download filename = 'lecuninterview.mp4' ydl_opts = { 'format': 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]', 'outtmpl': filename, 'quiet': True, } # Download the video file with yt_dlp.YoutubeDL(ydl_opts) as ydl: result = ydl.extract_info(url, download=True) url = "https://www.youtube.com/watch?v=mBjPyte2ZZo" download_mp4_from_youtube(url) ``` **Now it’s time for Whisper!** Whisper is a cutting-edge, automatic speech recognition system developed by OpenAI. Boasting state-of-the-art capabilities, Whisper has been trained on an impressive 680,000 hours of multilingual and multitasking supervised data sourced from the web. This vast and varied dataset enhances the system's robustness, enabling it to handle accents, background noise, and technical language easily. OpenAI has released the models and codes to provide a solid foundation for creating valuable applications harnessing the power of speech recognition. The whisper package that we installed earlier provides the `.load_model()` method to download the model and transcribe a video file. Multiple different models are available: `tiny`, `base`, `small`, `medium`, and `large`. Each one of them has tradeoffs between accuracy and speed. We will use the `'base'` model for this tutorial. ```python import whisper model = whisper.load_model("base") result = model.transcribe("lecuninterview.mp4") print(result['text']) ``` ``` /home/cloudsuperadmin/.local/lib/python3.9/site-packages/whisper/transcribe.py:114: UserWarning: FP16 is not supported on CPU; using FP32 instead warnings.warn("FP16 is not supported on CPU; using FP32 instead") Hi, I'm Craig Smith, and this is I on A On. This week I talked to Jan LeCoon, one of the seminal figures in deep learning development and a long-time proponent of self-supervised learning. Jan spoke about what's missing in large language models and his new joint embedding predictive architecture which may be a step toward filling that gap. He also talked about his theory of consciousness and the potential for AI systems to someday exhibit the features of consciousness. It's a fascinating conversation that I hope you'll enjoy. Okay, so Jan, it's great to see you again. I wanted to talk to you about where you've gone with so supervised learning since last week's spoke. In particular, I'm interested in how it relates to large language models because they have really come on stream since we spoke. In fact, in your talk about JEPA, which is joint embedding predictive architecture. […and so on] ``` We’ve got the result in the form of a raw text and it is possible to save it to a text file. ```python with open ('text.txt', 'w') as file: file.write(result['text']) ``` ### Summarization with LangChain We first import the necessary classes and utilities from the LangChain library. ```python from langchain import OpenAI, LLMChain from langchain.chains.mapreduce import MapReduceChain from langchain.prompts import PromptTemplate from langchain.chains.summarize import load_summarize_chain llm = OpenAI(model_name="text-davinci-003", temperature=0) ``` This imports essential components from the LangChain library for efficient text summarization and initializes an instance of OpenAI's large language model with a


# Create a YouTube Video Summarizer Using Whisper and LangChain
temperature setting of 0. The key elements include classes for handling large texts, optimization, prompt construction, and summarization techniques. This code creates an instance of the `RecursiveCharacterTextSplitter` class, which is responsible for splitting input text into smaller chunks. ```python from langchain.text_splitter import RecursiveCharacterTextSplitter text_splitter = RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=0, separators=[" ", ",", "\n"] ) ``` It is configured with a `chunk_size` ****of 1000 characters, no `chunk_overlap`, and uses spaces, commas, and newline characters as `separators`. This ensures that the input text is broken down into manageable pieces, allowing for efficient processing by the language model. We’ll open the text file we’ve saved previously and split the transcripts using `.split_text()` method. ```python from langchain.docstore.document import Document with open('text.txt') as f: text = f.read() texts = text_splitter.split_text(text) docs = [Document(page_content=t) for t in texts[:4]] ``` Each `Document` object is initialized with the content of a chunk from the `texts` ****list. The `[:4]` slice notation indicates that only the first four chunks will be used to create the `Document` ****objects. ```python from langchain.chains.summarize import load_summarize_chain import textwrap chain = load_summarize_chain(llm, chain_type="map_reduce") output_summary = chain.run(docs) wrapped_text = textwrap.fill(output_summary, width=100) print(wrapped_text) ``` ``` Craig Smith interviews Jan LeCoon, a deep learning developer and proponent of self-supervised learning, about his new joint embedding predictive architecture and his theory of consciousness. Jan's research focuses on self-supervised learning and its use for pre-training transformer architectures, which are used to predict missing words in a piece of text. Additionally, large language models are used to predict the next word in a sentence, but it is difficult to represent uncertain predictions when applying this to video. ``` <aside> 💡 The `textwrap` library in Python provides a convenient way to wrap and format plain text by adjusting line breaks in an input paragraph. It is particularly useful when displaying text within a limited width, such as in console outputs, emails, or other formatted text displays. The library includes convenience functions like `wrap`, `fill`, and `shorten`, as well as the `TextWrapper` class that handles most of the work. If you’re curious, I encourage you to follow this link and find out more, as there are other functions in the `textwrap` library that can be useful depending on your needs. [textwrap — Text wrapping and filling](https://docs.python.org/3/library/textwrap.html) </aside> With the following line of code, we can see the prompt template that is used with the map_reduce technique. Now we’re changing the prompt and using another summarization method: ```python print( chain.llm_chain.prompt.template ) ``` ``` Write a concise summary of the following:\n\n\n"{text}"\n\n\n CONCISE SUMMARY: ``` The `"stuff"` approach is the simplest and most naive one, in which all the text from the transcribed video is used in a single prompt. This method may raise exceptions if all text is longer than the available context size of the LLM and may not be the most efficient way to handle large amounts of text. We’re going to experiment with the prompt below. This prompt will output the summary as bullet points. ```python prompt_template = """Write a concise bullet point summary of the following: {text} CONSCISE SUMMARY IN BULLET POINTS:""" BULLET_POINT_PROMPT = PromptTemplate(template=prompt_template, input_variables=["text"]) ``` Also, we initialized the summarization chain using the stuff as `chain_type` and the prompt above. ```python chain = load_summarize_chain(llm, chain_type="stuff", prompt=BULLET_POINT_PROMPT) output_summary = chain.run(docs) wrapped_text = textwrap.fill(output_summary, width=1000, break_long_words=False, replace_whitespace=False) print(wrapped_text) ``` ``` - Jan LeCoon is a seminal figure in deep learning development and a long time proponent of self-supervised learning - Discussed his new joint embedding predictive architecture which may be a step toward filling the gap in large language models - Theory of consciousness and potential for AI systems to exhibit features