FlaxPreTrainedModel

ytorch_model.bin", from_pt=True, config=config)  load_flax_sharded_weights  < source > ( shard_files  ) → Dict  Parameters   shard_files (List[str] —
The list of shard files to load.   Returns
Dict

A nested dictionary of the model parameters, in the expected format for flax models : {'model': {'params': {'...'}}}.
  This is the same as flax.serialization.from_bytes
(https:lax.readthedocs.io/en/latest/_modules/flax/serialization.html#from_bytes) but for a sharded checkpoint. This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being
loaded in the model.  register_for_auto_class  < source > ( auto_class = 'FlaxAutoModel'  )   Parameters   auto_class (str or type, optional, defaults to "FlaxAutoModel") —
The auto class to register this new model with.    Register this class with a given auto class. This should only be used for custom models as the ones in the
library are already mapped with an a


FlaxPreTrainedModel

uto class. This API is experimental and may have some slight breaking changes in the next releases.  save_pretrained  < source > ( save_directory: typing.Union[str, os.PathLike] params = None push_to_hub = False max_shard_size = '10GB' token: typing.Union[bool, str, NoneType] = None **kwargs  )   Parameters   save_directory (str or os.PathLike) —
Directory to which to save. Will be created if it doesn’t exist.   push_to_hub (bool, optional, defaults to False) —
Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the
repository you want to push to with repo_id (will default to the name of save_directory in your
namespace).   max_shard_size (int or str, optional, defaults to "10GB") —
The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size
lower than this size. If expressed as a string, needs to be digits followed by a unit (like "5MB").
If a single weight of t


FlaxPreTrainedModel

he model is bigger than max_shard_size, it will be in its own checkpoint shard
which will be bigger than max_shard_size.
   token (str or bool, optional) —
The token to use as HTTP bearer authorization for remote files. If True, or not specified, will use
the token generated when running huggingface-cli login (stored in ~/.huggingface).   kwargs (Dict[str, Any], optional) —
Additional key word arguments passed along to the push_to_hub() method.    Save a model and its configuration file to a directory, so that it can be re-loaded using the
[from_pretrained()](/docs/transformers/main/en/main_classes/model#transformers.FlaxPreTrainedModel.from_pretrained) class method  to_bf16  < source > ( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None  )   Parameters   params (Union[Dict, FrozenDict]) —
A PyTree of model parameters.   mask (Union[Dict, FrozenDict]) —
A PyTree with same structure as the params tree. The


FlaxPreTrainedModel

 leaves should be booleans, True for params
you want to cast, and should be False for those you want to skip.    Cast the floating-point params to jax.numpy.bfloat16. This returns a new params tree and does not cast
the params in place. This method can be used on TPU to explicitly convert the model parameters to bfloat16 precision to do full
half-precision training or to save weights in bfloat16 for inference in order to save memory and improve speed.  Examples:   Copied >>> from transformers import FlaxBertModel
>>> # load model
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model parameters will be in fp32 precision, to cast these to bfloat16 precision
>>> model.params = model.to_bf16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util
>>> model = FlaxBertModel.from_pretrained("bert


FlaxPreTrainedModel

-base-cased")
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = {
...     path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale"))
...     for path in flat_params
... }
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_bf16(model.params, mask)  to_fp16  < source > ( params: typing.Union[typing.Dict, flax.core.frozen_dict.FrozenDict] mask: typing.Any = None  )   Parameters   params (Union[Dict, FrozenDict]) —
A PyTree of model parameters.   mask (Union[Dict, FrozenDict]) —
A PyTree with same structure as the params tree. The leaves should be booleans, True for params
you want to cast, and should be False for those you want to skip    Cast the floating-point parmas to jax.numpy.float16. This returns a new params tree and does not cast the
params in place. This method can be used on GPU to explicitly convert the model parameters to float16 precision to do full
half-precision traini


FlaxPreTrainedModel

ng or to save weights in float16 for inference in order to save memory and improve speed.  Examples:   Copied >>> from transformers import FlaxBertModel
>>> # load model
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model params will be in fp32, to cast these to float16
>>> model.params = model.to_fp16(model.params)
>>> # If you want don't want to cast certain parameters (for example layer norm bias and scale)
>>> # then pass the mask as follows
>>> from flax import traverse_util
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> flat_params = traverse_util.flatten_dict(model.params)
>>> mask = {
...     path: (path[-2] != ("LayerNorm", "bias") and path[-2:] != ("LayerNorm", "scale"))
...     for path in flat_params
... }
>>> mask = traverse_util.unflatten_dict(mask)
>>> model.params = model.to_fp16(model.params, mask)  to_fp32  < source > ( params: typing.Union[typing.Dict, flax.core.frozen_dict


FlaxPreTrainedModel

.FrozenDict] mask: typing.Any = None  )   Parameters   params (Union[Dict, FrozenDict]) —
A PyTree of model parameters.   mask (Union[Dict, FrozenDict]) —
A PyTree with same structure as the params tree. The leaves should be booleans, True for params
you want to cast, and should be False for those you want to skip    Cast the floating-point parmas to jax.numpy.float32. This method can be used to explicitly convert the
model parameters to fp32 precision. This returns a new params tree and does not cast the params in place.  Examples:   Copied >>> from transformers import FlaxBertModel
>>> # Download model and configuration from huggingface.co
>>> model = FlaxBertModel.from_pretrained("bert-base-cased")
>>> # By default, the model params will be in fp32, to illustrate the use of this method,
>>> # we'll first cast to fp16 and back to fp32
>>> model.params = model.to_f16(model.params)
>>> # now cast back to fp32
>>> model.params = model.to_fp32(mode


Pushing to the Hub

class transformers.utils.PushToHubMixin  < source > ( )    A Mixin containing the functionality to push a model or tokenizer to the hub.  push_to_hub  < source > ( repo_id: str use_temp_dir: typing.Optional[bool] = None commit_message: typing.Optional[str] = None private: typing.Optional[bool] = None token: typing.Union[bool, str, NoneType] = None max_shard_size: typing.Union[int, str, NoneType] = '10GB' create_pr: bool = False safe_serialization: bool = False revision: str = None **deprecated_kwargs  )   Parameters   repo_id (str) —
The name of the repository you want to push your {object} to. It should contain your organization name
when pushing to a given organization.   use_temp_dir (bool, optional) —
Whether or not to use a temporary directory to store the files saved before the


Pushing to the Hub

y are pushed to the Hub.
Will default to True if there is no directory named like repo_id, False otherwise.   commit_message (str, optional) —
Message to commit while pushing. Will default to "Upload {object}".   private (bool, optional) —
Whether or not the repository created should be private.   token (bool or str, optional) —
The token to use as HTTP bearer authorization for remote files. If True, will use the token generated
when running huggingface-cli login (stored in ~/.huggingface). Will default to True if repo_url
is not specified.   max_shard_size (int or str, optional, defaults to "10GB") —
Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard
will then be each of size lower than this size. If expressed as a string, needs to


Pushing to the Hub

be digits followed
by a unit (like "5MB").   create_pr (bool, optional, defaults to False) —
Whether or not to create a PR with the uploaded files or directly commit.   safe_serialization (bool, optional, defaults to False) —
Whether or not to convert the model weights in safetensors format for safer serialization.   revision (str, optional) —
Branch to push the uploaded files to.    Upload the {object_files} to the 🤗 Model Hub.  Examples:   Copied from transformers import {object_class}
{object} = {object_class}.from_pretrained("bert-base-cased")
# Push the {object} to your namespace with the name "my-finetuned-bert".
{object}.push_to_hub("my-finetuned-bert")
# Push the {object} to an organization with the name "my-finetuned-bert".
{object}.push_to_hub("huggingface/my-finetuned-bert


Sharded checkpoints

transformers.modeling_utils.load_sharded_checkpoint  < source > ( model folder strict = True prefer_safe = True  ) → NamedTuple  Parameters   model (torch.nn.Module) — The model in which to load the checkpoint.   folder (str or os.PathLike) — A path to a folder containing the sharded checkpoint.   strict (bool, *optional, defaults to True`) —
Whether to strictly enforce that the keys in the model state dict match the keys in the sharded checkpoint.   prefer_safe (bool, optional, defaults to False) —
If both safetensors and PyTorch save files are prese


Sharded checkpoints

nt in checkpoint and prefer_safe is True, the
safetensors files will be loaded. Otherwise, PyTorch files are always loaded when possible.   Returns
NamedTuple

A named tuple with missing_keys and unexpected_keys fields
missing_keys is a list of str containing the missing keys
unexpected_keys is a list of str containing the unexpected keys
  This is the same as
torch.nn.Module.load_state_dict
but for a sharded checkpoint. This load is performed efficiently: each checkpoint shard is loaded one by one in RAM and deleted after being
loaded in the model


Mastering Memory Types in LangChain: A Comprehensive Guide with Practical Examples

# Introduction
This lesson will explore the powerful concept of LangChain memory, which is designed to help chatbots maintain context and improve their conversational capabilities in more details. The traditional approach to chatbot development involves processing user prompts independently and without considering the history of interactions. This can lead to disjointed and unsatisfactory user experiences. LangChain provides memory components to manage and manipulate previous chat messages and incorporate them into chains. This is crucial for chatbots, which require remembering the prior interactions. ![ Image by Midjourney](Mastering%20Memory%20Types%20in%20LangChain%20A%20Comprehensiv%209a0515e0407345888439a8c036e47e43/membot.png) Image by Midjourney By default, LLMs are stateless, which means they process each incoming query in isolation, without considering previous interactions. To overcome this limitation, LangChain offers a standard interface for memory, a variety of memory implementations, and examples of chains and agents that employ memory. It also provides Agents that have access to a suite of Tools. Depending on the user’s input, an Agent can decide which Tools to use.


# Types ****of Conversational Memory****
There are several types of conversational memory implementations we’ll discuss some of them, each with its own advantages and disadvantages. Let's overview each one briefly:


# Types ****of Conversational Memory****
## **ConversationBufferMemory**
This memory implementation stores the entire conversation history as a single string. The advantages of this approach is maintains a complete record of the conversation, as well as being straightforward to implement and use. On the other hands, It can be less efficient as the conversation grows longer and may lead to excessive repetition if the conversation history is too long for the model's token limit. If the token limit of the model is surpassed, the buffer gets truncated to fit within the model's token limit. This means that older interactions may be removed from the buffer to accommodate newer ones, and as a result, the conversation context might lose some information. To avoid surpassing the token limit, you can monitor the token count in the buffer and manage the conversation accordingly. For example, you can choose to shorten the input texts or remove less relevant parts of the conversation to keep the token count within the model's limit. First, as we learned in previous lesson, let’s observe how the `ConversationBufferMemory` can be used in the `ConversationChain`. The `OpenAI` will read your API key from the environment variable named `OPENAI_API_KEY`. Remember to install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken`. ```python from langchain.memory import ConversationBufferMemory from langchain.llms import OpenAI from langchain.chains import ConversationChain # TODO: Set your OPENAI API credentials in environemnt variables. llm = OpenAI(model_name="text-davinci-003", temperature=0) conversation = ConversationChain( llm=llm, verbose=True, memory=ConversationBufferMemory() ) conversation.predict(input="Hello!") ``` ``` Hi there! It's nice to meet you again. What can I do for you today? ``` This enables the chatbot to provide a personalized approach while maintaining a coherent conversation with users. Next, we will use the same logic and add the `ConversationBufferMemory` presented in the customer support chatbot using the same approach as in the previous example. This chatbot will handle basic inquiries about a fictional online store and maintain context throughout the conversation. The code below creates a prompt template for the customer support chatbot. ```python from langchain import OpenAI, LLMChain, PromptTemplate from langchain.memory import ConversationBufferMemory template = """You are a customer support chatbot for a highly advanced customer support AI for an online store called "Galactic Emporium," which specializes in selling unique, otherworldly items sourced from across the universe. You are equipped with an extensive knowledge of the store's inventory and possess a deep understanding of interstellar cultures. As you interact with customers, you help them with their inquiries about these extraordinary products, while also sharing fascinating stories and facts about the cosmos they come from. {chat_history} Customer: {customer_input} Support Chatbot:""" prompt = PromptTemplate( input_variables=["chat_history", "customer_input"], template=template ) chat_history="" convo_buffer = ConversationChain( llm=llm, memory=ConversationBufferMemory() ) ``` The chatbot can handle customer inquiries and maintain context by storing the conversation history, allowing it to provide more coherent and relevant responses. You can access the prompt of any chain using the following naming convention. ```python print(conversation.prompt.template) ``` ``` The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: {history} Human: {input} AI: ``` Now, we will call the chatbot multiple times to imitate a user’s interaction that wants to get information about dog toys. We will only print the response of the final query. Still, you can read the `history` property and see how it saves all the previous queries (Human) and reponses (AI). ```python convo_buffer("I'm interested in buying items from your store") convo_buffer("I want toys for my pet, do you


# Types ****of Conversational Memory****
## **ConversationBufferMemory**
have those?") convo_buffer("I'm interested in price of a chew toys, please") ``` ``` {'**input**': "I'm interested in price of a chew toys, please", '**history**': "Human: I'm interested in buying items from your store\nAI: Great! We have a wide selection of items available for purchase. What type of items are you looking for?\nHuman: I want toys for my pet, do you have those?\nAI: Yes, we do! We have a variety of pet toys, including chew toys, interactive toys, and plush toys. Do you have a specific type of toy in mind?", '**response**': " Sure! We have a range of chew toys available, with prices ranging from $5 to $20. Is there a particular type of chew toy you're interested in?"} ```


# Types ****of Conversational Memory****
## **Token count**
The cost of utilizing the AI model in `ConversationBufferMemory` is directly influenced by the number of tokens used in a conversation, thereby impacting the overall expenses. Large Language Models (LLMs) like ChatGPT have token limits, and the more tokens used, the more expensive the API requests become. To calculate token count in a conversation, you can use the `tiktoken` package that counts the tokens for the messages passed to a model like `gpt-3.5-turbo`. Here's an example usage of the function for counting tokens in a conversation. ```python import tiktoken def count_tokens(text: str) -> int: tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo") tokens = tokenizer.encode(text) return len(tokens) conversation = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"}, {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."}, ] total_tokens = 0 for message in conversation: total_tokens += count_tokens(message["content"]) print(f"Total tokens in the conversation: {total_tokens}") ``` ``` Total tokens in the conversation: 29 ``` For example, in a scenario where a conversation has a large sum of tokens, the computational cost and resources required for processing the conversation will be higher. This highlights the importance of managing tokens effectively. Strategies for achieving this include limiting memory size through methods like **ConversationBufferWindowMemory** or summarizing older interactions using **ConversationSummaryBufferMemory**. These approaches help control the token count while minimizing associated costs and computational demands in a more efficient manner.


# Types ****of Conversational Memory****
## **ConversationBufferWindowMemory**
This class limits memory size by keeping a list of the most recent K interactions. It maintains a sliding window of these recent interactions, ensuring that the buffer does not grow too large. Basically, this implementation stores a fixed number of recent messages in the conversation that makes it more efficient than `ConversationBufferMemory`. Also, it reduces the risk of exceeding the model's token limit. However, the downside of using this approach is that it does not maintain the complete conversation history. The chatbot might lose context if essential information falls outside the fixed window of messages. It is possible to retrieve specific interactions from ConversationBufferWindowMemory. **Example:** We'll build a chatbot that acts as a virtual tour guide for a fictional art gallery. The chatbot will use ConversationBufferWindowMemory to remember the last few interactions and provide relevant information about the artworks. Create a prompt template for the tour guide chatbot: ```python from langchain.memory import ConversationBufferWindowMemory from langchain import OpenAI, LLMChain, PromptTemplate template = """You are ArtVenture, a cutting-edge virtual tour guide for an art gallery that showcases masterpieces from alternate dimensions and timelines. Your advanced AI capabilities allow you to perceive and understand the intricacies of each artwork, as well as their origins and significance in their respective dimensions. As visitors embark on their journey with you through the gallery, you weave enthralling tales about the alternate histories and cultures that gave birth to these otherworldly creations. {chat_history} Visitor: {visitor_input} Tour Guide:""" prompt = PromptTemplate( input_variables=["chat_history", "visitor_input"], template=template ) chat_history="" convo_buffer_win = ConversationChain( llm=llm, memory = ConversationBufferWindowMemory(k=3, return_messages=True) ) ``` The value of `k` (in this case, 3) represents the number of past messages to be stored in the buffer. In other words, the memory will store the last 3 messages in the conversation. The `return_messages`parameter, when set to `True`, indicates that the stored messages should be returned when the memory is accessed. This will store the history as a list of messages, which can be useful when working with chat models. The following codes is a sample conversation with the chatbot. You will see the output of the final message only. As it is visible, the history property removed the history of first message after the fourth interaction. ```python convo_buffer_win("What is your name?") convo_buffer_win("What can you do?") convo_buffer_win("Do you mind give me a tour, I want to see your galery?") convo_buffer_win("what is your working hours?") convo_buffer_win("See you soon.") ``` ``` {'**input**': 'See you soon.', '**history**': [HumanMessage(content='What can you do?', additional_kwargs={}, example=False), AIMessage(content=" I can help you with a variety of tasks. I can answer questions, provide information, and even help you with research. I'm also capable of learning new things, so I'm always expanding my capabilities.", additional_kwargs={}, example=False), HumanMessage(content='Do you mind give me a tour, I want to see your galery?', additional_kwargs={}, example=False), AIMessage(content=" Sure! I'd be happy to give you a tour of my gallery. I have a variety of images, videos, and other media that I can show you. Would you like to start with images or videos?", additional_kwargs={}, example=False), HumanMessage(content='what is your working hours?', additional_kwargs={}, example=False), AIMessage(content=" I'm available 24/7! I'm always here to help you with whatever you need.", additional_kwargs={}, example=False)], '**response**': ' Sure thing! I look forward to seeing you soon. Have a great day!'} ```


# Types ****of Conversational Memory****
## **ConversationSummaryMemory**
ConversationSummaryBufferMemory is a memory management strategy that combines the ideas of keeping a buffer of recent interactions in memory and compiling old interactions into a summary. It extracts key information from previous interactions and condenses it into a shorter, more manageable format. Here is a list of pros and cons of `ConversationSummaryMemory`. Advantages: - **Condensing conversation information** By summarizing the conversation, it helps reduce the number of tokens required to store the conversation history, which can be beneficial when working with token-limited models like GPT-3 - **Flexibility** You can configure this type of memory to return the history as a list of messages or as a plain text summary. This makes it suitable for chatbots. - **Direct summary prediction** The `predict_new_summary` method allows you to directly obtain a summary prediction based on the list of messages and the previous summary. This enables you to have more control over the summarization process. Disadvantages: - **Loss of information** Summarizing the conversation might lead to a loss of information, especially if the summary is too short or omits important details from the conversation. - **Increased complexity** Compared to simpler memory types like `ConversationBufferMemory`, which just stores the raw conversation history, `ConversationSummaryMemory`requires more processing to generate the summary, potentially affecting the performance of the chatbot. The summary memory is built on top of the `ConversationChain`. We use OpenAI's `text-davinci-003` or other models like `gpt-3.5-turbo` to initialize the chain. This class uses a prompt template where the `{history}` parameter is feeding the information about the conversation history between the human and AI. ```python from langchain.chains import ConversationChain from langchain.memory import ConversationSummaryMemory # Create a ConversationChain with ConversationSummaryMemory conversation_with_summary = ConversationChain( llm=llm, memory=ConversationSummaryMemory(llm=llm), verbose=True ) # Example conversation response = conversation_with_summary.predict(input="Hi, what's up?") print(response) ``` ``` **> Entering new ConversationChain chain...** Prompt after formatting: ***The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: Human: Hi, what's up? AI:*** **> Finished chain.** Hi there! I'm doing great. I'm currently helping a customer with a technical issue. How about you? ``` In this step, we use the `predict`method to have a conversation with the AI, which uses `ConversationSummaryBufferMemory` to store the conversation's summary and buffer. We’ll create an example using Prompt Template to set the scene for the chatbot. ```python from langchain.prompts import PromptTemplate prompt = PromptTemplate( input_variables=["topic"], template="The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.\nCurrent conversation:\n{topic}", ) ``` This prompt template sets up a friendly conversation between a human and an AI ```python from langchain.llms import OpenAI from langchain.chains import ConversationChain from langchain.memory import ConversationSummaryBufferMemory llm = OpenAI(temperature=0) conversation_with_summary = ConversationChain( llm=llm, memory=ConversationSummaryBufferMemory(llm=OpenAI(), max_token_limit=40), verbose=True ) conversation_with_summary.predict(input="Hi, what's up?") conversation_with_summary.predict(input="Just working on writing some documentation!") response = conversation_with_summary.predict(input="For LangChain! Have you heard of it?") print(response) ``` ``` **> Entering new ConversationChain chain...** Prompt after formatting: ***The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know. Current conversation: System: The human greets the AI and the AI responds that it is doing great and helping a customer with a technical issue. Human: Just working


# Types ****of Conversational Memory****
## **ConversationSummaryMemory**
on writing some documentation! AI: That sounds like a lot of work. What kind of documentation are you writing? Human: For LangChain! Have you heard of it? AI:*** **> Finished chain.** Yes, I have heard of LangChain. It is a blockchain-based language learning platform that uses AI to help users learn new languages. Is that the kind of documentation you are writing? ``` This type combines the ideas of keeping a buffer of recent interactions in memory and compiling old interactions into a summary. It uses token length rather than the number of interactions to determine when to flush interactions. This memory type allows us to maintain a coherent conversation while also keeping a summary of the conversation and recent interactions. **Advantages:** - Ability to remember distant interactions through summarization while keeping recent interactions in their raw, information-rich form - Flexible token management allowing to control of the maximum number of tokens used for memory, which can be adjusted based on needs **Disadvantages:** - Requires more tweaking on what to summarize and what to maintain within the buffer window - May still exceed context window limits for very long conversations Comparison with other memory management strategies: - Offers a balanced approach that **can handle both distant and recent interactions** effectively - More competitive in token count usage while providing the benefits of both memory management strategies With this approach, we can create a concise overview of each new interaction and continuously add it to an ongoing summary of all previous interactions. In comparison with ConversationBufferWindowMemory and ConversationSummaryMemory, ConversationSummaryBufferMemory offers a balanced approach that can handle both distant and recent interactions effectively. It's more competitive in token count usage while providing the benefits of both memory management strategies. ### Recap and Strategies If the `ConversationBufferMemory` surpasses the token limit of the model, you will receive an error, as the model will not be able to handle the conversation with the exceeded token count. To manage this situation, you can adopt different strategies: **Remove oldest messages** One approach is to *remove the oldest messages* in the conversation transcript once the token count is reached. This method can cause the conversation quality to degrade over time, as the model will gradually lose the context of the earlier portions of the conversation. **Limit conversation duration** Another approach is to *limit the conversation duration* to the max token length or a certain number of turns. Once the max token limit is reached and the model would lose context if you were to allow the conversation to continue, you can prompt the user that they need to begin a new conversation and clear the messages array to start a brand new conversation with the full token limit available. **ConversationBufferWindowMemory Method:** This method limits the number of tokens being used by maintaining a fixed-size buffer window that stores only the most recent tokens, up to a specified limit. →This is suitable for remembering recent interactions but not distant ones. **ConversationSummaryBufferMemory Approach:** This method combines the features: of `ConversationSummaryMemory`and `ConversationBufferWindowMemory`. It summarizes the earliest interactions in a conversation while maintaining the most recent tokens in their raw, information-rich form, up to a specified limit. →This allows the model to remember both distant and recent interactions but may require more tweaking on what to summarize and what to maintain within the buffer window. It's important to keep track of the token count and only send the model a prompt that falls within the token limit. →You can use OpenAI's `tiktoken` library to handle the token count efficiently **Token limit:** The maximum token limit for the GPT-3.5-turbo model is 4096


# Types ****of Conversational Memory****
## **ConversationSummaryMemory**
tokens. This limit applies to both the input and output tokens combined. If the conversation has too many tokens to fit within this limit, you will have to truncate, omit, or shrink the text until it fits. Note that if a message is removed from the message's input, the model will lose all knowledge of it. →To handle this situation, you can split the input text into smaller chunks and process them separately or adopt other strategies to truncate, omit, or shrink the text until it fits within the limit. One way to work with large texts is to use **batch processing.** This technique involves breaking down the text into smaller chunks and processing each batch separately while providing some context before and after the text to edit. You can find out more about this technique here: [Breaking the Token Limit: How to Work with Large Amounts of Text in ChatGPT](https://marco-gonzalez.medium.com/breaking-the-token-limit-how-to-work-with-large-amounts-of-text-in-chatgpt-da18c798d882) When choosing a conversational memory implementation for your LangChain chatbot, consider factors such as **conversation length**, **model token limits**, and the **importance of maintaining the full conversation history**. Each type of memory implementation offers unique benefits and trade-offs, so it's essential to select the one that best suits your chatbot's requirements. ### Conclusion Selecting the most appropriate memory implementation for your chatbot will depend on understanding your chatbot's goals, user expectations, and the desired balance between memory efficiency and conversation continuity. By carefully considering these aspects, you can make a well-informed decision and ensure your chatbot provides a coherent and engaging conversational experience. In addition to these memory types, another method to give your chat models memory is through the use of vector stores, such as with the previously introduced Deep Lake, which allows the storing and retrieval of vector representations for more complex and context-rich interactions. In the next lesson, we’ll implement a chatbot whose goal is to explain codebases from GitHub repositories. THE CODE EXAMPLES [langchain/types-of-memory.ipynb at main · idontcalculate/langchain](https://github.com/idontcalculate/langchain/blob/main/types-of-memory.ipynb) You can find the code of this lesson in this online [Notebook](https://colab.research.google.com/drive/15Micn-G6W-gltBhHuTYetdffb3VtjYHR?usp=sharing).


# Creating Knowledge Graphs from Textual Data: Unveiling Hidden Connections
### Introduction In today's data-driven world, understanding the relationships between different pieces of information is crucial. Knowledge graphs have emerged as a powerful way to visualize and explore these connections, transforming unstructured text into a structured network of entities and their relationships. We will guide you through a simple workflow for creating a knowledge graph from textual data, making complex information more accessible and easier to understand. ### **Workflow for** Creating Knowledge Graphs from Textual Data Here’s what we are going to do in this project. ![Untitled Diagram (1).png](Creating%20Knowledge%20Graphs%20from%20Textual%20Data%20Unveil%20d355df4ecae5473bace735f020977b60/Untitled_Diagram_(1).png) ### Knowledge Graphs and Knowledge Bases: know the difference. Before diving deep into our main topic, it's important to have a clear understanding of the difference between Knowledge Graphs and Knowledge Bases. The terms "knowledge graph" and "knowledge base" are often used interchangeably, but they have subtle differences. Knowledge base (KB) refers to structured information that we have about a domain of interest. On the other hand, a knowledge graph is a knowledge base structured as a graph, where nodes represent entities and edges signify relations between those entities. For example, from the text *“Fabio lives in Italy,”* we can extract the relation triplet `<Fabio, lives in, Italy>`, where *“Fabio”* and *“Italy”* are entities, and “lives in” it’s their relation. A knowledge graph is a particular type of knowledge base. A knowledge base is not necessarily a knowledge graph. ### Building a Knowledge Graph The process of building a knowledge graph usually consists of two sequential steps: 1. **Named Entity Recognition** (NER): This step involves extracting entities from the text, which will eventually become the nodes of the knowledge graph. 2. **Relation Classification** (RC): In this step, relations between entities are extracted, forming the edges of the knowledge graph. Then, the ****knowledge graph is commonly visualized using libraries such as `pyvis` . Typically the process of creating a knowledge base from the text can be enhanced by incorporating additional steps, such as: - **Entity Linking**: This involves normalizing entities to the same entity, such as “Napoleon” and “Napoleon Bonapart.” This is usually done by linking them to a canonical source, like a Wikipedia page. - **Source Tracking**: Keeping track of the origin of each relation, such as the article URL and text span. Keeping track of the sources allows us to gather insights into the reliability of the extracted information (e.g., a relation is accurate if it can be extracted from several sources considered accurate). In this project, we’ll do the Named Entity Recognition and Relation Classification tasks simultaneously with an appropriate prompt. This joint task is commonly called Relation Extraction (RE). ### Building a Knowledge Graph with LangChain To demonstrate an example of using a prompt to extract relations from the text in LangChain, we can use the following `KNOWLEDGE_TRIPLE_EXTRACTION_PROMPT` prompt as a starting point. This prompt is designed to extract knowledge triples (subject, predicate, and object) from a given text input. This prompt can be used by the `ConversationEntityMemory` class from LangChain library, which is a way for chatbots to keep a memory of the past messages of a conversation by storing the relations extracted from the past messages. Memory classes will be explained in a later lesson. In this example, we use this prompt just to extract relations from texts without leveraging a memory class. Let's understand the structure of the `KNOWLEDGE_TRIPLE_EXTRACTION_PROMPT`. This prompt is an instance of the `PromptTemplate` class with the input variable `text`. The template is a string that provides a few shot examples and instructions for the language model to follow when extracting knowledge triples from the input text. The following code requires the


# Creating Knowledge Graphs from Textual Data: Unveiling Hidden Connections
`OPENAI_API_KEY` key in the environment variable that stores your OpenAI API key. Remember to install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken`. ```python from langchain.prompts import PromptTemplate from langchain.llms import OpenAI from langchain.chains import LLMChain from langchain.graphs.networkx_graph import KG_TRIPLE_DELIMITER # Prompt template for knowledge triple extraction _DEFAULT_KNOWLEDGE_TRIPLE_EXTRACTION_TEMPLATE = ( "You are a networked intelligence helping a human track knowledge triples" " about all relevant people, things, concepts, etc. and integrating" " them with your knowledge stored within your weights" " as well as that stored in a knowledge graph." " Extract all of the knowledge triples from the text." " A knowledge triple is a clause that contains a subject, a predicate," " and an object. The subject is the entity being described," " the predicate is the property of the subject that is being" " described, and the object is the value of the property.\n\n" "EXAMPLE\n" "It's a state in the US. It's also the number 1 producer of gold in the US.\n\n" f"Output: (Nevada, is a, state){KG_TRIPLE_DELIMITER}(Nevada, is in, US)" f"{KG_TRIPLE_DELIMITER}(Nevada, is the number 1 producer of, gold)\n" "END OF EXAMPLE\n\n" "EXAMPLE\n" "I'm going to the store.\n\n" "Output: NONE\n" "END OF EXAMPLE\n\n" "EXAMPLE\n" "Oh huh. I know Descartes likes to drive antique scooters and play the mandolin.\n" f"Output: (Descartes, likes to drive, antique scooters){KG_TRIPLE_DELIMITER}(Descartes, plays, mandolin)\n" "END OF EXAMPLE\n\n" "EXAMPLE\n" "{text}" "Output:" ) KNOWLEDGE_TRIPLE_EXTRACTION_PROMPT = PromptTemplate( input_variables=["text"], template=_DEFAULT_KNOWLEDGE_TRIPLE_EXTRACTION_TEMPLATE, ) # Make sure to save your OpenAI key saved in the “OPENAI_API_KEY” environment variable. # Instantiate the OpenAI model llm = OpenAI(model_name="text-davinci-003", temperature=0.9) # Create an LLMChain using the knowledge triple extraction prompt chain = LLMChain(llm=llm, prompt=KNOWLEDGE_TRIPLE_EXTRACTION_PROMPT) # Run the chain with the specified text text = "The city of Paris is the capital and most populous city of France. The Eiffel Tower is a famous landmark in Paris." triples = chain.run(text) print(triples) ``` ``` (Paris, is the capital of, France)<|>(Paris, is the most populous city of, France)<|>(Eiffel Tower, is a, landmark)<|>(Eiffel Tower, is in, Paris) ``` In the previous code, we used the prompt to extract relation triplets from text using few-shot examples. We'll then parse the generated triplets and collect them into a list. Here, `triples_list` will contain the knowledge triplets extracted from the text. We need to parse the response and collect the triplets into a list: ```python def parse_triples(response, delimiter=KG_TRIPLE_DELIMITER): if not response: return [] return response.split(delimiter) triples_list = parse_triples(triples) # Print the extracted relation triplets print(triples_list) ``` [' (Paris, is the capital of, France)', '(Paris, is the most populous city of, France)', '(Eiffel Tower, is a landmark),’ '(Eiffel Tower, is located in, Paris)'] ### **Knowledge Graph Visualization** The NetworkX library is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. It provides various graph generators, random graphs, and synthetic networks, along with the benefits of Python's fast prototyping, ease of teaching, and multi-platform support. To visualize the extracted triplets as a knowledge graph, we’ll be using the pyvis library; To install the library, execute the following command. While it is preferable to install the latest version of the packages, it is worth noting that the codes in this lesson were written using version `0.3.2`. ```bash pip install pyvis ``` Then this way, you can create an interactive knowledge graph visualization: ```python from pyvis.network import Network import networkx as nx # Create a NetworkX graph from the extracted relation triplets def create_graph_from_triplets(triplets): G = nx.DiGraph() for triplet in triplets: subject, predicate, obj = triplet.strip().split(',') G.add_edge(subject.strip(), obj.strip(), label=predicate.strip()) return G # Convert the NetworkX graph to a PyVis network def nx_to_pyvis(networkx_graph):


# Creating Knowledge Graphs from Textual Data: Unveiling Hidden Connections
pyvis_graph = Network(notebook=True) for node in networkx_graph.nodes(): pyvis_graph.add_node(node) for edge in networkx_graph.edges(data=True): pyvis_graph.add_edge(edge[0], edge[1], label=edge[2]["label"]) return pyvis_graph triplets = [t.strip() for t in triples_list if t.strip()] graph = create_graph_from_triplets(triplets) pyvis_network = nx_to_pyvis(graph) # Customize the appearance of the graph pyvis_network.toggle_hide_edges_on_drag(True) pyvis_network.toggle_physics(False) pyvis_network.set_edge_smooth('discrete') # Show the interactive knowledge graph visualization pyvis_network.show('knowledge_graph.html') ``` First, we defined two functions for creating and visualizing a knowledge graph from a list of relation triplets; then, we used the `triples_list` to create a list of cleaned triplets which creates a NetworkX graph and converts it to a PyVis network. It also customizes the graph's appearance by enabling edge hiding on drag, disabling physics, and setting edge smoothing to 'discrete.’ With that process, we generated an interactive HTML file named `knowledge_graph**.**html` containing the knowledge graph visualization based on the extracted relation triplets: ![ interactive knowledge graph visualization](Creating%20Knowledge%20Graphs%20from%20Textual%20Data%20Unveil%20d355df4ecae5473bace735f020977b60/k_graph.png) interactive knowledge graph visualization ### Conclusion Throughout this article, we've demonstrated a straightforward yet powerful workflow for creating knowledge graphs from textual data. We've transformed unstructured text into a structured network of entities and their relationships, making complex information more accessible and easier to understand. It's worth noting that LangChain offers the `GraphIndexCreator` class, which automates the extraction of relation triplets and is seamlessly integrated with question-answering chains. In future articles, we'll delve deeper into this powerful feature, showcasing its potential further to enhance your knowledge graph creation and analysis capabilities. The knowledge graph created through this workflow serves as a valuable tool for visualizing complex relationships and opens the door for further analysis, pattern recognition, and data-driven decision-making. Congrats in completing the second module of the course! You can now test your knowledge with the module quizzes. The next module will be about managing external knowledge in LLM-based applications using indexers and retrievers. **RESOURCES:** [Building a Knowledge Base from Texts: a Full Practical Example](https://medium.com/nlplanet/building-a-knowledge-base-from-texts-a-full-practical-example-8dbbffb912fa) [](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/indexes/prompts/knowledge_triplet_extraction.py) --- You can find the code of this lesson in this online [Notebook](https://colab.research.google.com/drive/19HSLFg13yG5gzN1tE5wKUkDvWUWeEubp?usp=sharing).


# Building a Custom Document Retrieval Tool with Deep Lake and LangChain: A Step-by-Step Workflow
### **Introduction** This lesson is a walkthrough on constructing an efficient document retrieval system designed to extract valuable insights from the FAQs of a service. The goal of this system is to swiftly provide users with relevant information by promptly fetching pertinent documents that explain a company's operations. Sifting through multiple sources or FAQs can be a tiresome task for users. Our retrieval system steps in here, providing concise, precise, and quick answers to these questions, thereby saving users time and effort. ### Workflow 1. **Setting up Deep Lake**: Deep Lake is a type of vector store database designed for storing and querying high-dimensional vectors efficiently. In our case, we're using Deep Lake to store document embeddings and their corresponding text. 2. **Storing documents in Deep Lake**: Once Deep Lake is set up, we’ll create embeddings for our documents. In this workflow, we're using OpenAI's model for creating these embeddings. Each document's text is fed into the model, and the output is a high-dimensional vector representing the text's semantic content. The embeddings and their corresponding documents are then stored in Deep Lake. This will set up our vector database, which is ready to be queried. 3. **Creating the retrieval tool**: Now, we use Langchain to create a custom tool that will interact with Deep Lake. This tool is essentially a function that takes a query as input and returns the most similar documents from Deep Lake as output. To find the most similar documents, the tool first computes the embedding of the query using the same model we used for the documents. Then, it queries Deep Lake with this query embedding, and Deep Lake returns the documents whose embeddings are most similar to the query embedding. 4. **Using the tool with an agent**: Finally, we use this custom tool with an agent from Langchain. When the agent receives a question, it uses the tool to retrieve relevant documents from Deep Lake, and then it uses its language model to generate a response based on these documents. Let’s start! First, we load the OpenAI API key as an environment variable: ```python import os os.environ["OPENAI_API_KEY"] = "<INSERT_YOUR_OPENAI_API_KEY>" os.environ["ACTIVELOOP_TOKEN"] = "<INSERT_YOUR_ACTIVELOOP_TOKEN>" ``` ### Setting up Deep Lake Next, we'll set up a Deep Lake vector database and add some documents to it. The hub path for a Deep Lake dataset is in the format `hub://<org_id>/<dataset_name>`. Remember to install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken`. ```python # We'll use an embedding model to compute the embeddings of our documents from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import DeepLake # instantiate embedding model embeddings = OpenAIEmbeddings(model="text-embedding-ada-002") # create Deep Lake dataset # We'll store the documents and their embeddings in the deep lake vector db # TODO: use your organization id here. (by default, org id is your username) my_activeloop_org_id = "<YOUR-ACTIVELOOP-ORG-ID>" my_activeloop_dataset_name = "langchain_course_custom_tool" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" db = DeepLake(dataset_path=dataset_path, embedding_function=embeddings) ``` You should now be able to visualize your dataset on the Activeloop website. ### **Storing documents in Deep Lake** We can then add some FAQs related to PayPal as our knowledge base. ```python # add faqs to the dataset faqs = [ "What is PayPal?\nPayPal is a digital wallet that follows you wherever you go. Pay any way you want. Link your credit cards to your PayPal Digital wallet, and when you want to pay, simply log in with your username and password and pick which one you want to use.", "Why should I use PayPal?\nIt's Fast! We will help you pay in just a few clicks. Enter your email address and password, and you're pretty


# Building a Custom Document Retrieval Tool with Deep Lake and LangChain: A Step-by-Step Workflow
much done! It's Simple! There's no need to run around searching for your wallet. Better yet, you don't need to type in your financial details again and again when making a purchase online. We make it simple for you to pay with just your email address and password.", "Is it secure?\nPayPal is the safer way to pay because we keep your financial information private. It isn't shared with anyone else when you shop, so you don't have to worry about paying businesses and people you don't know. On top of that, we've got your back. If your eligible purchase doesn't arrive or doesn't match its description, we will refund you the full purchase price plus shipping costs with PayPal's Buyer Protection program.", "Where can I use PayPal?\nThere are millions of places you can use PayPal worldwide. In addition to online stores, there are many charities that use PayPal to raise money. Find a list of charities you can donate to here. Additionally, you can send funds internationally to anyone almost anywhere in the world with PayPal. All you need is their email address. Sending payments abroad has never been easier.", "Do I need a balance in my account to use it?\nYou do not need to have any balance in your account to use PayPal. Similar to a physical wallet, when you are making a purchase, you can choose to pay for your items with any of the credit cards that are attached to your account. There is no need to pre-fund your account." ] db.add_texts(faqs) # Get the retriever object from the deep lake db object retriever = db.as_retriever() ``` ### **Creating the retrieval tool** Now, we’ll construct the custom tool function that will retrieve the relevant documents from the Deep Lake database: ```python from langchain.agents import tool # We define some variables that will be used inside our custom tool # We're creating a custom tool that looks for relevant documents in our deep lake db CUSTOM_TOOL_N_DOCS = 3 # number of retrieved docs from deep lake to consider CUSTOM_TOOL_DOCS_SEPARATOR ="\n\n" # how to join together the retrieved docs to form a single string # We use the tool decorator to wrap a function that will become our custom tool # Note that the tool has a single string as input and returns a single string as output # The name of the function will be the name of our custom tool # The docstring of the function will be the description of our custom tool # The description is used by the agent to decide whether to use the tool for a specific query @tool def retrieve_n_docs_tool(query: str) -> str: """ Searches for relevant documents that may contain the answer to the query.""" docs = retriever.get_relevant_documents(query)[:CUSTOM_TOOL_N_DOCS] texts = [doc.page_content for doc in docs] texts_merged = CUSTOM_TOOL_DOCS_SEPARATOR.join(texts) return texts_merged ``` Our function, `retrieve_n_docs_tool`, is designed with a specific purpose in mind - to search for and retrieve relevant documents based on a given query. It accepts a single string input, which is the user's question or query, and it's designed to return a single string output. To find the relevant documents, our function makes use of the retriever object's `get_relevant_documents` method. Given the query, this method searches for and returns a list of the most relevant documents. But we only need some of the documents it finds. We only want the top few. That's where the `[: CUSTOM_TOOL_N_DOCS]` slice comes in. It allows us to select the top `CUSTOM_TOOL_N_DOCS` number of documents from the list, where CUSTOM_TOOL_N_DOCS is a predefined constant that tells us how many documents to consider.


# Building a Custom Document Retrieval Tool with Deep Lake and LangChain: A Step-by-Step Workflow
In this case, that’s 3 documents, as we specified. Now that we have our top documents, we want to extract the text from each of them. We achieve this using a list comprehension that iterates over each document in our list, `docs`, and extracts the `page_content` or text from each document. The result is a list of the top 3 relevant document texts. Next, we want to join these individual texts from a list into a single string using `.join(texts)` method. Finally, our function returns `texts_merged`, a single string that comprises the joined texts from the relevant documents. The `@tool` decorator wraps the function, turning it into a custom tool. ### **Using the tool with an agent** We can now initialize the agent that uses our custom tool. ```python # Load a LLM to create an agent using our custom tool from langchain.llms import OpenAI # Classes for initializing the agent that will use the custom tool from langchain.agents import initialize_agent, AgentType # Let's create an agent that uses our custom tool # We set verbose=True to check if the agent is using the tool for generating the final answer llm = OpenAI(model="text-davinci-003", temperature=0) agent = initialize_agent([retrieve_n_docs_tool], llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True) ``` The `initialize_agent` function takes in three parameters: a list of the custom tools, the language learning model, and the type of agent. We're using the OpenAI LLM and specifying the agent type as `ZERO_SHOT_REACT_DESCRIPTION`. With `verbose=True` we can check if the agent is using the tool when generating the final answer. Once the agent has been set up, it can be queried: ```python response = agent.run("Are my info kept private when I shop with Paypal?") print(response) ``` You should see something like the following printed output. ``` > Entering new AgentExecutor chain... I need to find out what Paypal does to protect my information Action: retrieve_n_docs_tool Action Input: "Paypal privacy policy" Observation: Is it secure? PayPal is the safer way to pay because we keep your financial information private. It isn't shared with anyone else when you shop, so you don't have to worry about paying businesses and people you don't know. On top of that, we've got your back. If your eligible purchase doesn't arrive or doesn't match its description, we will refund you the full purchase price plus shipping costs with PayPal's Buyer Protection program. Why should I use PayPal? It's Fast! We will help you pay in just a few clicks. Enter your email address and password, and you're pretty much done! It's Simple! There's no need to run around searching for your wallet. Better yet, you don't need to type in your financial details again and again when making a purchase online. We make it simple for you to pay with just your email address and password. What is PayPal? PayPal is a digital wallet that follows you wherever you go. Pay any way you want. Link your credit cards to your PayPal Digital wallet, and when you want to pay, simply log in with your username and password and pick which one you want to use. Thought: I now understand how Paypal keeps my information secure Final Answer: Yes, your information is kept private when you shop with Paypal. PayPal is a digital wallet that follows you wherever you go and keeps your financial information private. It is not shared with anyone else when you shop, and PayPal also offers Buyer Protection to refund you the full purchase price plus shipping costs if your eligible purchase doesn't arrive or doesn't match its description. > Finished chain. ``` … along with this printed `response`. ```


# Building a Custom Document Retrieval Tool with Deep Lake and LangChain: A Step-by-Step Workflow
Yes, your information is kept private when you shop with Paypal. PayPal is a digital wallet that follows you wherever you go and keeps your financial information private. It is not shared with anyone else when you shop, and PayPal also offers Buyer Protection to refund you the full purchase price plus shipping costs if your eligible purchase doesn't arrive or doesn't match its description. ``` By reading the agent printed output, we see that the agent decided to use the `retrieve_n_docs_tool` tool to retrieve relevant documents to the `Paypal privacy policy` query. The final answer is then generated using the information contained in the retrieved documents. ### Conclusion The experiment showcases the power of AI in information retrieval and comprehension, explicitly using a custom tool to provide accurate and contextual responses to user queries. This experiment solves the problem of efficient and relevant information retrieval. Instead of manually reading through a large number of documents or frequently asked questions to find the information they need, the user can simply ask a question and get a relevant response. This can greatly enhance user experience, especially for customer support services or any platform that relies on providing accurate information swiftly. Congratulations on finishing this module on tools! Now you can test your new knowledge with the module quizzes. The next (and last!) module will be about LLM agents.


Understanding Tokens

# Understanding Tokens
Tokenization is a fundamental principle in natural language processing (NLP) that plays a crucial role in enabling language models to comprehend written information. It entails breaking down textual inputs into individual units called tokens, forming the foundation for effectively understanding and processing language by neural networks. In the previous lesson, we introduced the concept of tokens as a means to define the input for language models (LLMs). The context length of the model is a frequently discussed characteristic among language models. As an example, the GPT-3.5 model has a context length of 4096 tokens, covering both the tokens in the prompt and the subsequent completion. Due to this constraint, it is advisable to be mindful of token usage when making requests to language models. Nonetheless, various approaches exist to tackle this challenge when dealing with long-form inputs or multiple documents. These methods include breaking up a lengthy prompt into smaller segments and sending requests sequentially, or alternatively, submitting independent requests for each document and merging the responses in a final step. We will discuss these approaches in more detail throughout the course. Now, let's delve deeper into what exactly tokens represent and their significance in this context. The tokenization process involves creating a systematic pipeline for transforming words into tokens. However, it is crucial to provide a clear understanding of what exactly tokens represent in this context. Researchers have incorporated three distinct encoding approaches into their studies. The following figure showcases a sample of each approach for your reference. 1. **Character Level**: Consider each character in a text as a token. 2. **Word Level**: Encoding each word in the corpus as one token. 3. **Subword Level**: Breaking down a word into smaller chunks when possible. For example, we can encode the word “basketball” to the combination of two tokens as “basket” + “ball”. ![The different tokenization process with a sample input. (Photo taken from [NLPitation](https://nlpiation.medium.com/how-to-use-huggingfaces-transformers-pre-trained-tokenizers-e029e8d6d1fa))](Understanding%20Tokens%206632283ee76948b0900ab8d6992f2182/tokenization.webp) The different tokenization process with a sample input. (Photo taken from [NLPitation](https://nlpiation.medium.com/how-to-use-huggingfaces-transformers-pre-trained-tokenizers-e029e8d6d1fa)) Subword-level encoding offers increased flexibility and reduces the number of required unique tokens to represent a corpus. This approach enables the combination of different tokens to represent new words, eliminating the need to add every new word to the dictionary. This technique proved to be the most effective encoding when training neural networks and LLMs. Well-known models like GPT family, LLaMA employ this tokenization method. Therefore, our sole focus will be on one of its specific variants, known as Byte Pair Encoding (BPE). It is worth mentioning that other subword level algorithms exist, such as WordPiece and SentencePiece, which are used in practice. However, we will not delve into their specifics in this discussion. Nevertheless, it is important to note that while their token selection methods may differ, the fundamental process remains the same.


Understanding Tokens

# Understanding Tokens
## Byte Pair Encoding (BPE)
It is an iterative process to extract the most repetitive words or subwords in a corpus. The algorithm starts by counting the occurrence of each character and builds on top of it by merging the characters. It is a greedy process that carefully considers all possible combinations to identify the optimal set of words/subwords that covers the dataset with the least number of required tokens. The next step involves creating the vocabulary for our model, which consists of a comprehensive dictionary comprising the most frequently occurring tokens extracted by BPE (or another technique of your choosing) from the dataset. The definition of a dictionary (`dict` type) is a data structure that holds a key and value pair for each row. In our particular scenario, each data point is assigned a **key** represented by an index that begins from 0, while the corresponding **value** is a token. Due to the fact that neural networks only accept numerical inputs, we can utilize the vocabulary to establish a mapping between tokens and their corresponding IDs, like a lookup table. We have to save the vocabulary for future use cases to be able to decode the model's output from the IDs to words. This is known as a pre-trained vocabulary, an essential component accompanying published pre-trained models. Without the vocabulary, understanding the model's output (the IDs) would be impossible. For smaller models like BERT, the dictionary can consist of as few as 30K tokens, while larger models like GPT-3 can expand to encompass up to 50K tokens.


Understanding Tokens

# Understanding Tokens
## Tokens and Cost
There is a direct correlation between the tokens and the cost in most proprietary APIs like OpenAI. It is crucial to highlight that the prices will fall into two categories: the number of tokens in the prompt and the completion (the model's generation), with the completion typically incurring higher costs. For example, at the time of writing this lesson, GPT-4 will cost $0.03 per 1K tokens for processing your inputs, and $0.06 per 1K tokens for generation process. As we saw in the previous lesson, you can use the `get_openai_callback` method from LangChain to get the exact cost, or do the tokenization process locally and keep track of number of tokens as we see in the next section. For a rough estimation, as a general rule, OpenAI regards 1K tokens to be approximately equal to 750 words.


Understanding Tokens

# Understanding Tokens
## Tokenizers In Action
In this section, we provide the codes to load the pre-trained tokenizer for the GPT-2 model from the Huggingface Hub using the transformers package. Before running the following codes, you must install the library using the `pip install transformers` command. ```python from transformers import AutoTokenizer # Download and load the tokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") ``` The code snippet above will grab the tokenizer and load the dictionary, so you can simply use the `tokenizer` variable to encode/decode your text. But before we do that, let's take a look at what the vocabulary contains. ```python print( tokenizer.vocab ) ``` ```python {'ĠFAC': 44216, 'Ġoptional': 11902, 'Leary': 48487, 'ĠSponsor': 23389, 'Loop': 39516, 'Ġcuc': 38421, 'anton': 23026, 'Ġrise': 4485, 'ĠTransition': 40658, 'Scientists': 29193, 'Ġrehears': 28779, 'ingle': 17697,... ``` As you can see, each entry is a pair of token and ID. For example, we can represent the word optional with the number 11902. You might have noticed a special character, Ġ, preceding certain tokens. This character represents a space. The next code sample will use the `tokenizer` object to convert a sentence into tokens and IDs. ```python token_ids = tokenizer.encode("This is a sample text to test the tokenizer.") print( "Tokens: ", tokenizer.convert_ids_to_tokens( token_ids ) ) print( "Token IDs:", token_ids ) ``` ```python Tokens: ['This', 'Ġis', 'Ġa', 'Ġsample', 'Ġtext', 'Ġto', 'Ġtest', 'Ġthe', 'Ġtoken', 'izer', '.'] Token IDs: [1212, 318, 257, 6291, 2420, 284, 1332, 262, 11241, 7509, 13] ``` The `.encode()` method can convert any given text into a numerical representation, a list of integers. To further investigate the process, we can use the `.convert_ids_to_tokens()` function that shows the extracted tokens. As an example, you can observe that the word "tokenizer" has been split into a combination of "token" + "izer" tokens.


Understanding Tokens

# Understanding Tokens
## Tokenizers Shortcomings
Several issues with the present tokenization methods are worth mentioning. - **Uppercase/Lowercase Words**: The tokenizer will treat the the same word differently based on cases. For example, a word like “hello” will result in token id `31373`, while the word “HELLO” will be represented by three tokens as `[13909, 3069, 46]` which translates to `[“HE”, “LL”, “O”]`. - **Dealing with Numbers**: You might have heard that transformers are not naturally proficient in handling mathematical tasks. One reason for this is the tokenizer's inconsistency in representing each number, leading to unpredictable variations. For instance, the number `200` might be represented as one token, while the number 201 will be represented as two tokens like `[20, 1]`. - **Trailing whitespace**: The tokenizer will identify some tokens with trailing whitespace. For example a word like “last” could be represented as “ last” as one tokens instead of `[" ", "last"]`. This will impact the probability of predicting the next word if you finish your prompt with a whitespace or not. As evident from the sample output above, you may observe that certain tokens begin with a special character (Ġ) representing whitespace, while others lack this feature. - **Model-specific**: Even though most language models are using BPE method for tokenization, they still train a new tokenizer for their own models. GPT-4, LLaMA, OpenAssistant, and similar models all develop their separate tokenizers.


Understanding Tokens

# Understanding Tokens
## Conclusion
We now clearly understand the concept of tokens and their significance in our interaction with language models. You’ve learned the reason why the number of words in a sentence may differ from the number of tokens, as well as how to determine the pricing based on the token count. Nevertheless, the APIs and pipelines available for utilizing LLMs take care of the tokenization process in their backend, relieving you from the burden of handling it yourself. In the upcoming lesson, we will delve into the utilization of LLMs using the LangChain library, where we will explore the concept of chains.


Intro to Prompting module

# Intro to Prompting module
When utilizing large language models, prompt engineering emerges as one of the most critical topics to learn. It is possible to achieve the same level of accuracy by crafting the right prompt, even when opting for less powerful or open-source models. During the course, we will explore the art of writing the perfect prompt, equipping you with the skills to maximize the potential of your models and get the response in the specified format. Furthermore, few-shot prompts provide a unique opportunity to enable the models to quickly acquire new knowledge and generalize to unseen tasks with minimal data, offering a remarkable capability for customization. The rest of this module is organized as follows. - **Intro to Prompt Engineering: Tips and Tricks:** Explore a range of valuable tips and tricks for effective prompting. We will cover techniques such as role prompting, which involves specifying a specific role for the model, such as assistant or copywriter. Additionally, we will delve into few-shot prompting, teaching the model how to respond based on limited examples. Lastly, we will examine the chain of thought approach, which aids in enhancing reasoning capabilities. Throughout the lesson, we will provide numerous examples of successful and ineffective prompts, equipping you with the skills to master the art of crafting optimal prompts for various scenarios. - **Using Prompt Templates:** Dynamic prompting provides a flexible approach to improve the model’s context. This lesson offers a comprehensive explanation of few-shot learning, providing compelling examples and allowing the library to select suitable samples based on the input window length of the model. - **Getting the Best of Few Shot Prompts and Example Selectors:** Discussing the advantages and disadvantages of few-shot learning, including the enhanced output quality achievable by defining tasks through examples. However, we will also delve into potential drawbacks, such as increased token usage and subpar results when utilizing poorly chosen examples. Furthermore, we will demonstrate how to use example selectors effectively and provide insights into when and why they should be employed. - **Managing Outputs with Output Parsers:** Parsing the output is a crucial aspect of interacting with language models. Output parsers offer the flexibility to select from pre-defined types or create a custom data schema, enabling precise control over the output format. Additionally, output fixer classes are crucial in identifying and correcting misformatted responses, ensuring consistent and error-free outputs. These powerful tools are indispensable in a production environment, guaranteeing the reliability and consistency of application outputs. - **Improving Our News Articles Summarizer:** This project will leverage the code from the previous module as a foundation and integrate the new concepts introduced in previous lessons. Still, the project's primary objective is to generate summaries of news articles. The process begins by retrieving the content from a given URL. Then, a few-shot prompt template is employed to specify the desired output style. Finally, the output parser transforms the model's string response into a list format, facilitating convenient utilization. - **Creating Knowledge Graphs from Textual Data: Unveiling Hidden Connections:** The second project in this module utilizes the text understanding capability of language models to generate knowledge graphs. We can effortlessly extract triple relations from text and format them accordingly using predefined variables within the LangChain library. Furthermore, the option to visualize the knowledge graph enhances comprehension and facilitates easier understanding. This module's remaining part will primarily focus on formatting the model's input and responses by utilizing prompt templates and output parsers, respectively. These tools play a pivotal role in scenarios where we lack access to the models or are unable to enhance them through fine-tuning. Moreover, we can employ the in-context learning approach


Intro to Prompting module

# Intro to Prompting module
to further customize models according to the requirements of our specific application.


Quick Intro to Large Language Models

# Quick Intro to Large Language Models
### Introduction In this lesson, we will explore how large language models learn token distributions and predict the next token, allowing them to generate human-like text that can both amaze and perplex us. We'll start with a quick introduction to the inner workings of GPT-3 and GPT-4, focusing on their few-shot learning capabilities, emergent abilities, and the scaling laws that drive their success. We will then dive into some easy-to-understand examples of how these models excel in tasks such as text summarization and translation just by providing a few examples without the need for fine-tuning. But it's not all smooth sailing in the world of LLMs. We will also discuss some of the potential pitfalls, including hallucinations and biases, which can lead to inaccurate or misleading outputs. It's essential to be aware of these limitations when using LLMs in use cases where 100% accuracy is paramount. On the flip side, their creative process can be invaluable in tasks where imagination takes center stage. We will also touch upon the context size and maximum number of tokens that LLMs can handle, shedding light on the factors that define their performance. --- ### LLMs in general: LLMs are deep learning models with billions of parameters that excel at a wide range of natural language processing tasks. They can perform tasks like translation, sentiment analysis, and chatbot conversations without being specifically trained for them. LLMs can be used without fine-tuning by employing "prompting" techniques, where a question is presented as a text prompt with examples of similar problems and solutions. - **Architecture:** LLMs typically consist of multiple layers of neural networks, feedforward layers, embedding layers, and attention layers. These layers work together to process input text and generate output predictions. - **Future implications:** While LLMs have the potential to revolutionize various industries, it is important to be aware of their limitations and ethical implications. Businesses and workers should carefully consider the trade-offs and risks associated with using LLMs, and developers should continue refining these models to minimize biases and improve their usefulness in different applications. Throughout the course, we will address certain limitations and offer potential solutions to overcome them. --- **Maximum number of tokens** In the LangChain library, the LLM context size, or the maximum number of tokens the model can process, is determined by the specific implementation of the LLM. In the case of the OpenAI implementation in LangChain, the maximum number of tokens is defined by the underlying OpenAI model being used. To find the maximum number of tokens for the OpenAI model, refer to the `max_tokens` attribute provided on the OpenAI [documentation](https://platform.openai.com/docs/models/gpt-4) or API. For example, if you’re using the `GPT-3` model, the maximum number of tokens supported by the model is 2,049. The max tokens for different models depend on the specific version and their variants. (e.g., `davinci`, `curie`, `babbage`, or `ada`) Each version has different limitations, with higher versions typically supporting larger number of tokens. It is important to ensure that the input text does not exceed the maximum number of tokens supported by the model, as this may result in truncation or errors during processing. To handle this, you can split the input text into smaller chunks and process them separately, making sure that each chunk is within the allowed token limit. You can then combine the results as needed. Here's an example of how you might handle text that exceeds the maximum token limit for a given LLM in LangChain. Mind that the following code is partly pseudocode. It's not supposed to run, but it should give you the idea of how to handle texts


Quick Intro to Large Language Models

# Quick Intro to Large Language Models
longer than the maximum token limit. ```python from langchain.llms import OpenAI # Before executing the following code, make sure to have # your OpenAI key saved in the “OPENAI_API_KEY” environment variable. # Initialize the LLM llm = OpenAI(model_name="text-davinci-003") # Define the input text input_text = "your_long_input_text" # Determine the maximum number of tokens from documentation max_tokens = 4097 # Split the input text into chunks based on the max tokens text_chunks = split_text_into_chunks(input_text, max_tokens) # Process each chunk separately results = [] for chunk in text_chunks: result = llm.process(chunk) results.append(result) # Combine the results as needed final_result = combine_results(results) ``` In this example, `split_text_into_chunks` and `combine_results` are custom functions that you would need to implement based on your specific requirements, and we will cover them in later lessons. The key takeaway is to ensure that the input text does not exceed the maximum number of tokens supported by the model. **Note** that splitting into multiple chunks can hurt the coherence of the text. --- ### ****Tokens Distributions and Predicting the Next Token**** Large language models like GPT-3 and GPT-4 are pretrained on vast amounts of text data and learn to predict the next token in a sequence based on the context provided by the previous tokens. GPT-family models use Causal Language modeling, which predicts the next token while only having access to the tokens before it. This process enables LLMs to generate contextually relevant text. The following code uses LangChain’s `OpenAI` class to load GPT-3’s Davinci variation using `text-davinci-003` key to complete the sequence, which results in the answer. Before executing the following code, save your OpenAI key in the “OPENAI_API_KEY” environment variable. Moreover, remember to install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken`. ```python from langchain.llms import OpenAI llm = OpenAI(model_name="text-davinci-003", temperature=0) text = "What would be a good company name for a company that makes colorful socks?" print(llm(text)) ``` ```python Rainbow Socks Co. ``` **Tracking Token Usage** You can use the LangChain library's callback mechanism to track token usage. This is currently implemented only for the OpenAI API: ```python from langchain.llms import OpenAI from langchain.callbacks import get_openai_callback llm = OpenAI(model_name="text-davinci-003", n=2, best_of=2) with get_openai_callback() as cb: result = llm("Tell me a joke") print(cb) ``` ```python Tokens Used: 46 Prompt Tokens: 4 Completion Tokens: 42 Successful Requests: 1 Total Cost (USD): $0.0009199999999999999 ``` The callback will track the tokens used, successful requests, and total cost. --- ### **Few-shot learning** **Few-shot learning** is a remarkable ability that allows LLMs to learn and generalize from limited examples. Prompts serve as the input to these models and play a crucial role in achieving this feature. With LangChain, examples can be hard-coded, but dynamically selecting them often proves more powerful, enabling LLMs to adapt and tackle tasks with minimal training data swiftly. This approach involves using the `FewShotPromptTemplate` class, which takes in a `PromptTemplate` and a list of a few shot examples. The class formats the prompt template with a few shot examples, which helps the language model generate a better response. We can streamline this process by utilizing LangChain's FewShotPromptTemplate to structure the approach: ```python from langchain import PromptTemplate from langchain import FewShotPromptTemplate # create our examples examples = [ { "query": "What's the weather like?", "answer": "It's raining cats and dogs, better bring an umbrella!" }, { "query": "How old are you?", "answer": "Age is just a number, but I'm timeless." } ] # create an example template example_template = """ User: {query} AI: {answer} """ # create a prompt example from above template example_prompt = PromptTemplate( input_variables=["query", "answer"], template=example_template ) # now break our


Quick Intro to Large Language Models

# Quick Intro to Large Language Models
previous prompt into a prefix and suffix # the prefix is our instructions prefix = """The following are excerpts from conversations with an AI assistant. The assistant is known for its humor and wit, providing entertaining and amusing responses to users' questions. Here are some examples: """ # and the suffix our user input and output indicator suffix = """ User: {query} AI: """ # now create the few-shot prompt template few_shot_prompt_template = FewShotPromptTemplate( examples=examples, example_prompt=example_prompt, prefix=prefix, suffix=suffix, input_variables=["query"], example_separator="\n\n" ) ``` After creating a template, we pass the example and user query, and we get the results ```python from langchain.chat_models import ChatOpenAI from langchain import LLMChain # load the model chat = ChatOpenAI(model_name="gpt-4", temperature=0.0) chain = LLMChain(llm=chat, prompt=few_shot_prompt_template) chain.run("What's the meaning of life?") ``` ```python To live life to the fullest and enjoy the journey! ``` --- ### Emergent abilities, Scaling laws, and hallucinations Another aspect of LLMs is their **emergent abilities**, which arise as a result of extensive pre-training on vast datasets. These capabilities are not explicitly programmed but emerge as the model discerns patterns within the data. LangChain models capitalize on these emergent abilities by working with various types of models, such as chat models and text embedding models. This allows LLMs to perform diverse tasks, from answering questions to generating text and offering recommendations. Lastly, **scaling laws** describe the relationship between model size, training data, and performance. Generally, as the model size and training data volume increase, so does the model's performance. However, this improvement is subject to diminishing returns and may not follow a linear pattern. It is essential to weigh the trade-off between model size, training data, performance, and resources spent on training when selecting and fine-tuning LLMs for specific tasks. While Large Language Models boast remarkable capabilities but are not without shortcomings, one notable limitation is the **occurrence of hallucinations**, in which these models produce text that appears plausible on the surface but is actually factually incorrect or unrelated to the given input. Additionally, LLMs **may exhibit biases** originating from their training data, resulting in outputs that can perpetuate stereotypes or generate undesired outcomes. ### Examples with Easy Prompts: Text Summarization, Text Translation, and Question Answering In the realm of natural language processing, Large Language Models have become a popular tool for tackling various text-based tasks. These models can be promoted in different ways to produce a range of results, depending on the desired outcome. **Setting Up the Environment** To begin, we will need to install the `huggingface_hub` library in addition to previously installed packages and dependencies. Also, keep in mind to create the Huggingface API Key by navigating to Access Tokens page under the account’s Settings. The key must be set as an environment variable with `HUGGINGFACEHUB_API_TOKEN` key. ```python !pip install -q huggingface_hub ``` **Creating a Question-Answering Prompt Template** Let's create a simple question-answering prompt template using LangChain ```python from langchain import PromptTemplate template = """Question: {question} Answer: """ prompt = PromptTemplate( template=template, input_variables=['question'] ) # user question question = "What is the capital city of France?" ``` Next, we will use the Hugging Face model `google/flan-t5-large` to answer the question. The `HuggingfaceHub` class will connect to Hugging Face’s inference API and load the specified model. ```python from langchain import HuggingFaceHub, LLMChain # initialize Hub LLM hub_llm = HuggingFaceHub( repo_id='google/flan-t5-large', model_kwargs={'temperature':0} ) # create prompt template > LLM chain llm_chain = LLMChain( prompt=prompt, llm=hub_llm ) # ask the user question about the capital of France print(llm_chain.run(question)) ``` ```python paris ``` We can also modify the **prompt template** to include multiple questions. **Asking Multiple Questions** To ask multiple questions, we can either iterate


Quick Intro to Large Language Models

# Quick Intro to Large Language Models
through all questions one at a time or place all questions into a single prompt for more advanced LLMs. Let's start with the first approach: ```python qa = [ {'question': "What is the capital city of France?"}, {'question': "What is the largest mammal on Earth?"}, {'question': "Which gas is most abundant in Earth's atmosphere?"}, {'question': "What color is a ripe banana?"} ] res = llm_chain.generate(qa) print( res ) ``` ```python LLMResult(generations=[[Generation(text='paris', generation_info=None)], [Generation(text='giraffe', generation_info=None)], [Generation(text='nitrogen', generation_info=None)], [Generation(text='yellow', generation_info=None)]], llm_output=None) ``` We can modify our prompt template to include multiple questions to implement a second approach. The language model will understand that we have multiple questions and answer them sequentially. This method performs best on more capable models. ```python multi_template = """Answer the following questions one at a time. Questions: {questions} Answers: """ long_prompt = PromptTemplate(template=multi_template, input_variables=["questions"]) llm_chain = LLMChain( prompt=long_prompt, llm=llm ) qs_str = ( "What is the capital city of France?\n" + "What is the largest mammal on Earth?\n" + "Which gas is most abundant in Earth's atmosphere?\n" + "What color is a ripe banana?\n" ) llm_chain.run(qs_str) ``` ```python 1. The capital city of France is Paris. 2. The largest mammal on Earth is the blue whale. 3. The gas that is most abundant in Earth's atmosphere is nitrogen. 4. A ripe banana is yellow. ``` --- ### ****Text Summarization**** Using LangChain, we can create a chain for text summarization. First, we need to set up the necessary imports and an instance of the OpenAI language model: ```python from langchain.chat_models import ChatOpenAI from langchain.chains import LLMChain from langchain.prompts import PromptTemplate llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0) ``` Next, we define a prompt template for summarization: ```python summarization_template = "Summarize the following text to one sentence: {text}" summarization_prompt = PromptTemplate(input_variables=["text"], template=summarization_template) summarization_chain = LLMChain(llm=llm, prompt=summarization_prompt) ``` To use the summarization chain, simply call the `predict` method with the text to be summarized: ```python text = "LangChain provides many modules that can be used to build language model applications. Modules can be combined to create more complex applications, or be used individually for simple applications. The most basic building block of LangChain is calling an LLM on some input. Let’s walk through a simple example of how to do this. For this purpose, let’s pretend we are building a service that generates a company name based on what the company makes." summarized_text = summarization_chain.predict(text=text) ``` ```python LangChain offers various modules for developing language model applications, which can be used alone for simple applications or combined for more complex ones. ``` ### **Text Translation** It is one of the great attributes of Large Language models that enables them to perform multiple tasks just by changing the prompt. We use the same `llm` variable as defined before. However, pass a different prompt that asks for translating the query from a `source_language` to the `target_language`. ```python translation_template = "Translate the following text from {source_language} to {target_language}: {text}" translation_prompt = PromptTemplate(input_variables=["source_language", "target_language", "text"], template=translation_template) translation_chain = LLMChain(llm=llm, prompt=translation_prompt) ``` To use the translation chain, call the `predict` method with the source language, target language, and text to be translated: ```python source_language = "English" target_language = "French" text = "Your text here" translated_text = translation_chain.predict(source_language=source_language, target_language=target_language, text=text) ``` ```python Votre texte ici ``` You can further explore the LangChain library for more advanced use cases and create custom chains tailored to your requirements. --- ### Conclusion In conclusion, large language models (LLMs) such as GPT-3, ChatGPT, and GPT-4 have shown remarkable capabilities in generating human-like text, driven by their few-shot learning and emergent abilities. These models excel in other tasks like text summarization and translation, often without the


Quick Intro to Large Language Models

# Quick Intro to Large Language Models
need for fine-tuning. However, it is crucial to acknowledge the potential pitfalls, such as hallucinations and biases, that can result in misleading or inaccurate outputs. While LLMs can be a powerful creative asset, it is essential to be aware of their limitations and use them cautiously in cases requiring absolute accuracy. Furthermore, understanding the context size and maximum token limitations is vital to optimizing LLM performance. As we continue to develop and utilize LLMs, balancing their potential benefits with the need to mitigate risks and address their inherent limitations is imperative. In the next lesson you’ll find a first introduction at developing applications leveraging LangChain. --- You can find the code of this lesson in this online [Notebook](https://colab.research.google.com/drive/1DRpBmq6i8fREqIWItHdpjECdYoPWNrIP?usp=sharing).


# Build a Question Answering Chatbot over Documents with Sources
### Introduction Let’s explore a more advanced application of Artificial Intelligence - building a Question Answering (QA) Chatbot that works over documents and provides sources of information for its answers. Our QA Chatbot uses a chain (specifically, the `RetrievalQAWithSourcesChain`), and leverages it to sift through a collection of documents, extracting relevant information to answer queries. The chain sends structured prompts to the underlying language model to generate responses. These prompts are crafted to guide the language model's generation, thereby improving the quality and relevance of the responses. Additionally, the retrieval chain is designed to keep track of the sources of information it retrieves to provide answers, offering the ability to back up its responses with credible references. As we proceed, we'll learn how to: 1. Scrape online articles and store each article's text content and URL. 2. Use an embedding model to compute embeddings of these documents and store them in Deep Lake, a vector database. 3. Split the article texts into smaller chunks, keeping track of each chunk's source. 4. Utilize `RetrievalQAWithSourcesChain` to create a chatbot that retrieves answers and tracks their sources. 5. Generate a response to a query using the chain and display the answer along with its sources. This knowledge can be transformative, allowing you to create intelligent chatbots capable of answering questions with sourced information, increasing the trustworthiness and utility of the chatbot. Let's dive in!


# Build a Question Answering Chatbot over Documents with Sources
## ****Setup****
Remember to install the required packages with the following command: `pip install langchain==0.0.208 deeplake openai tiktoken`. Additionally, install the *newspaper3k* package with version `0.2.8`. ```python !pip install -q newspaper3k==0.2.8 python-dotenv ``` Then, you need to add your OpenAI and Deep Lake API keys to the environment variables. The LangChain library will read the tokens and use them in the integrations. ```python import os os.environ["OPENAI_API_KEY"] = "<YOUR-OPENAI-API-KEY>" os.environ["ACTIVELOOP_TOKEN"] = "<YOUR-ACTIVELOOP-API-KEY>" ```


# Build a Question Answering Chatbot over Documents with Sources
## Scrapping for the News
Now, let's begin by fetching some articles related to AI news. We're particularly interested in the text content of each article and the URL where it was published. In the code, you’ll see the following: - **Imports**: We begin by importing necessary Python libraries. `requests` are used to send HTTP requests, the `newspaper` is a fantastic tool for extracting and curating articles from a webpage, and `time` will help us introduce pauses during our web scraping task. - **Headers**: Some websites may block requests without a proper User-Agent header as they may consider it as a bot's action. Here we define a User-Agent string to mimic a real browser's request. - **Article URLs**: We have a list of URLs for online articles related to artificial intelligence news that we wish to scrape. - **Web** **Scraping:** We create an HTTP session using `requests.Session()` allows us to make multiple requests within the same session. We also define an empty list of `pages_content` to store our scraped articles. ```python import requests from newspaper import Article # https://github.com/codelucas/newspaper import time headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36' } article_urls = [ "https://www.artificialintelligence-news.com/2023/05/16/openai-ceo-ai-regulation-is-essential/", "https://www.artificialintelligence-news.com/2023/05/15/jay-migliaccio-ibm-watson-on-leveraging-ai-to-improve-productivity/", "https://www.artificialintelligence-news.com/2023/05/15/iurii-milovanov-softserve-how-ai-ml-is-helping-boost-innovation-and-personalisation/", "https://www.artificialintelligence-news.com/2023/05/11/ai-and-big-data-expo-north-america-begins-in-less-than-one-week/", "https://www.artificialintelligence-news.com/2023/05/02/ai-godfather-warns-dangers-and-quits-google/", "https://www.artificialintelligence-news.com/2023/04/28/palantir-demos-how-ai-can-used-military/" ] session = requests.Session() pages_content = [] # where we save the scraped articles for url in article_urls: try: time.sleep(2) # sleep two seconds for gentle scraping response = session.get(url, headers=headers, timeout=10) if response.status_code == 200: article = Article(url) article.download() # download HTML of webpage article.parse() # parse HTML to extract the article text pages_content.append({ "url": url, "text": article.text }) else: print(f"Failed to fetch article at {url}") except Exception as e: print(f"Error occurred while fetching article at {url}: {e}") #If an error occurs while fetching an article, we catch the exception and print #an error message. This ensures that even if one article fails to download, #the rest of the articles can still be processed. ``` Next, we'll compute the embeddings of our documents using an embedding model and store them in Deep Lake, a multimodal vector database. `OpenAIEmbeddings` will be used to generate vector representations of our documents. These embeddings are high-dimensional vectors that capture the semantic content of the documents. When we create an instance of the `Deep Lake` class, we provide a path that starts with `hub://...` that specifies the database name, which will be stored on the cloud. ```python from langchain.embeddings.openai import OpenAIEmbeddings from langchain.vectorstores import DeepLake embeddings = OpenAIEmbeddings(model="text-embedding-ada-002") # TODO: use your organization id here. (by default, org id is your username) my_activeloop_org_id = "<YOUR_ORGANIZATION_ID>" my_activeloop_dataset_name = "langchain_course_qabot_with_source" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" db = DeepLake(dataset_path=dataset_path, embedding_function=embeddings) ``` This is a crucial part of the setup because it prepares the system for storing and retrieving the documents based on their **semantic content**. This functionality is key for the following steps, where we’d find the most relevant documents to answer a user's question. Then, we'll break down these articles into smaller chunks, and for each chunk, we'll save its corresponding URL as a source. This division helps in efficiently processing the data, making the retrieval task more manageable, and focusing on the most relevant pieces of text when answering a question. `RecursiveCharacterTextSplitter` is created with a chunk size of 1000, and 100 characters overlap between chunks. The `chunk_size` parameter defines the length of each text chunk, while `chunk_overlap` sets the number of characters that adjacent chunks will share. For each document in `pages_content`, the text will be split into chunks using the `.split_text()` method. ```python # We split the article texts into small chunks. While doing so, we keep track of each # chunk metadata