# How To Set Up an Assistant
## **Example demonstration:**
Imagine you're developing an AI assistant for a tech company. This assistant needs to provide detailed product support using a comprehensive knowledge base. ```bash mkdir openai-assistants && cd openai-assistants python3 -m venv openai-assistants-env source openai-assistants-env/bin/activate pip3 install python-dotenv pip3 install --upgrade openai # fire up VSCode and let's get rolling! code . ``` Replace the text with your OpenAI API key, which you can get from your [OpenAI developer account](https://platform.openai.com/api-keys). ```python OPENAI_API_KEY="sh-xxx" ``` ```bash $ pip install -U -q openai ``` ### **Upload Files to a Knowledge Base**: First, make a folder to store all the files you’ll create. **Upload** a detailed PDF manual of a product line (e.g., "tech_manual.pdf") using the API: ```python from openai import OpenAI client = OpenAI() file = client.beta.files.upload( file=open("tech_manual.pdf", "rb"), filetype="application/pdf", description="Tech product manual" ) ``` Now you can create the assistant with an uploaded file and with the ability to retrieve: `tools=[{"type": "retrieval"}]` ```python assistant = client.beta.assistants.create( instructions="You are a tech support chatbot. Use the product manual to respond accurately to customer inquiries.", model="gpt-4-1106-preview", tools=[{"type": "retrieval"}], file_ids=[file.id] ) ``` **User Interaction**: To interact with the assistant, you need a **`thread` and a `message`**. The message should contain the customer's question. Here's an example: ```python thread = client.beta.threads.create() message = client.beta.threads.messages.create( thread_id=thread.id, role="user", content="How do I reset my Model X device?", ) ``` ### RUN T**hread**: - A customer asks, "*How do I reset my Model X device?*" The assistant accesses the uploaded manual, performs a vector search to find the relevant section, and provides clear, step-by-step reset instructions. ```python run = client.beta.threads.runs.create( thread_id=thread.id, assistant_id=assistant.id, ) # the run will enter the **queued** state before it continues it’s execution. ``` ### **Information retrieval:** After the run is complete, you can retrieve the assistant's response: ```python messages = client.beta.threads.messages.list( thread_id=thread.id ) assistant_response = messages.data[0].content[0].text.value ``` The output result should contain the assistant's response to the customer's question based on knowledge from the uploaded manual. You can see the full code and more examples in this Colab notebook. [Google Colaboratory](https://colab.research.google.com/drive/1GkjoTZUd2rpwIvxmCZreKo5ZUZN6i7vJ)


Crafting AI Assistants via OpenAI and Hugging Face

# OpenAI’s Other Advanced Models
OpenAI also offers different types of models that are not yet integrated into the Assistants API but are accessible. These models offer voice processing, image understanding, and image generation capabilities. ### **[Whisper-v3](https://huggingface.co/openai/whisper-large-v3)** Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. It is a transformer-based encoder-decoder model, which is a type of *sequence-to-sequence* model. The latest `large-v3` model shows improved performance over various languages compared to Whisper `large-v2`. OpenAI released the model’s weights with an ****Apache License 2.0.**** The model is available on [Hugging Face](https://huggingface.co/openai/whisper-large-v3). ### **[Text to Speech](https://platform.openai.com/docs/guides/text-to-speech)** TTS is an AI model that converts text to natural-sounding spoken text. They offer two different model variates: **`tts-1`** is optimized for real-time text-to-speech use cases, and **`tts-1-hd`** is optimized for quality. These models can be used with the [Speech endpoint in the Audio API](https://platform.openai.com/docs/guides/text-to-speech). ### **[Dall-E 3](https://openai.com/dall-e-3)** A newer iteration of the DALL-E model is designed for image generation. It can create images based on user prompts, making it a valuable tool for graphic designers, artists, and anyone to generate images quickly and efficiently. You can access the model through the [image generation endpoint](https://platform.openai.com/docs/guides/images/generations). ### **[GPT-4 Vision](https://openai.com/research/gpt-4v-system-card)** GPT-4 with Vision enables you to ask questions about the contents of images. Visual question answering (VQA) is an important computer vision research field. You can also perform other vision tasks, such as Optical Character Recognition (OCR), where a model reads text in an image. Using GPT-4 with Vision, you can ask questions about what is or is not in an image, how objects relate in an image, the spatial relationships between two objects (is one object to the left or right of another), the color of an object, and more. GPT-4V is available through the [OpenAI web interface for ChatGPT Plus](https://openai.com/?ref=blog.roboflow.com) subscribers and through [their API](https://platform.openai.com/docs/guides/vision/vision). This expands the model's utility beyond the traditional text-only inputs, enabling it to be applied in a wider range of contexts. It handles images through the Chat Completions API, but note that the Assistants API does not support GPT-4V at this time. GPT4-V supports advanced use cases like creating image captions, in-depth analysis of visual content, and interpreting text and graphics in documents. ![https://huggingface.co/front/assets/huggingface_logo-noborder.svg](https://huggingface.co/front/assets/huggingface_logo-noborder.svg)


# **Hugging Face Inference API**
Hugging Face (HF) offers a free service for testing and evaluating over 150,000 publicly available machine learning models hosted on their platform through their [Inference API.](https://huggingface.co/docs/api-inference/index) They provide a wide range of models, including transformer and diffusion-based models, that can help solve various NLP or vision tasks such as text classification, sentiment analysis, named entity recognition, etc. <aside> 💡 Note that these free Inference APIs are **rate-limited** and not for production use. You can check out their [Inference Endpoint service if you want good performance](https://huggingface.co/inference-endpoints). </aside>


# **Hugging Face Inference API**
## Steps to use the Inference API**:**
1. [Login](https://huggingface.co/login) to Hugging Face. 2. Navigate to your profile on the top right navigation bar, then click "Edit profile.” 3. Click on the "Access Tokens" menu item. 4. Set the HF HUB API token: ```bash export HUGGINGFACEHUB_API_TOKEN=your-token ``` 1. Use the `HUGGINGFACEHUB_API_TOKEN` as an environment variable ```python import os from huggingface_hub import HfApi hf_api = HfApi(token=os.getenv("HUGGINGFACEHUB_API_TOKEN")) ``` 1. Run the Inference API Inference is the process of using a trained model to predict new data. The `huggingface_hub` library provides an easy way to call a service that runs inference for hosted models. As described above, you have two types of services available. - **[Inference API](https://huggingface.co/docs/api-inference/index)**: run accelerated inference on Hugging Face’s infrastructure **for free**. - **[Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index)**: easily deploy models to production **(paid)** 6.1 Choose a model from the [Model Hub](https://huggingface.co/models) The model checkpoints are stored in the Model Hub; you can search and share them. Note that not all models are available on the Inference API. Once the endpoint has been created, you should see a URL endpoint of it like the following: ```python ENDPOINT = https://api-inference.huggingface.co/models/<MODEL_ID> ``` 1. Run the inference. ```python import requests API_URL = "https://api-inference.huggingface.co/models/<MODEL_ID>" headers = {"Authorization": f"Bearer {API_TOKEN}"} def query(payload): response = requests.post(API_URL, headers=headers, json=payload) return response.json() data = query("Can you please let us know more") ```


# **Hugging Face Inference API**
## Hugging Face Tasks
The team at [Hugging Face has categorized](https://huggingface.co/tasks) several models into the different tasks they can solve. You can find models for popular NLP tasks: Question Answering, Sentence Similarity, Summarization, Table Question Answering, and more. Here is another example of using the Inference API for a summarization task. ```python import requests API_TOKEN = 'your_api_token_here' model_name = 'facebook/bart-large-cnn' text_to_summarize = "Hugging Face's API simplifies accessing powerful NLP models for tasks like summarization, transforming verbose texts into concise, insightful summaries." endpoint = f'https://api-inference.huggingface.co/models/{model_name}' headers = {'Authorization': f'Bearer {API_TOKEN}'} data = {'inputs': text_to_summarize} response = requests.post(endpoint, headers=headers, json=data) summarized_text = response.json()[0]['summary_text'] print(summarized_text) ``` We used a pre-trained model, **`[facebook/bart-large-cnn](https://huggingface.co/facebook/bart-large-cnn)`,** showcasing its ability to produce clear and concise summaries. [Google Colaboratory](https://colab.research.google.com/drive/1GkjoTZUd2rpwIvxmCZreKo5ZUZN6i7vJ?usp=sharing) **Note:** Not all models are available in this Inference API. Verify if the model is available by reviewing its '*Model card*.’ Sentiment analysis task: ```python import requests headers = {"Authorization": f"Bearer {API_TOKEN}"} API_URL = "https://api-inference.huggingface.co/models/distilbert-base-uncased-finetuned-sst-2-english" def query(payload): response = requests.post(API_URL, headers=headers, json=payload) return response.json() data = query({"inputs": "I love how this app simplifies complex tasks effortlessly . I'm frustrated by the frequent errors in the software's latest update"}) print(data) ``` Text-to-image task: ```bash # run a few installations !pip install diffusers["torch"] transformers !pip install -U sentence-transformers ``` ```python from diffusers import StableDiffusionPipeline import torch model_id = "runwayml/stable-diffusion-v1-5" pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16) pipe = pipe.to("cuda") prompt = "Create an image of a futuristic cityscape on an alien planet, featuring towering skyscrapers with glowing neon lights, a sky filled with multiple moons, and inhabitants of various alien species walking through vibrant market streets" image = pipe(prompt).images[0] image.save("astronaut_rides_horse.png") ``` Resulting image: ![Image generated with stable diffusion](Crafting%20AI%20Assistants%20via%20OpenAI%20and%20Hugging%20Face%20788f0523a80d442dba0530f5d4f76198/Crafting%20AI%20Assistants%20via%20OpenAI%20and%20Hugging%20Face%20bdc8bb2675504d8f983e56b9e63cc93b/f93209da-87e6-43e3-b21d-f052f0b27e51.png) Image generated with stable diffusion You can also encode a sentence and get text embeddings. ```python from sentence_transformers import SentenceTransformer sentences = ["GAIA's questions are rooted in practical use cases, requiring AI systems to interact with a diverse and uncertain world, reflecting real-world applications.", " GAIA questions require accurate execution of complex sequences of actions, akin to the Proof of Work concept, where the solution is simple to verify but challenging to generate."] model = SentenceTransformer('Equall/english-beta-0.3', use_auth_token=API_TOKEN) embeddings = model.encode(sentences) print(embeddings) ``` > [[ 0.76227915 -0.5500489 -1.5719271 ... -0.34034422 -0.27251056 0.12204967] [ 0.29783687 0.6476462 -2.0379746 ... -0.28033397 -1.3997376 0.25214267]] > You can also experiment with image-captioning models: ```python from transformers import pipeline image_to_text = pipeline("image-to-text", model="nlpconnect/vit-gpt2-image-captioning") image_to_text("https://ankur3107.github.io/assets/images/image-captioning-example.png") # [{'generated_text': 'a soccer game with a player jumping to catch the ball '}] ``` You can experiment with classification tasks with image-to-text models pre-trained on ImageNet: ```python from transformers import ViTImageProcessor, ViTForImageClassification from PIL import Image import requests url = 'http://images.cocodataset.org/val2017/000000039769.jpg' image = Image.open(requests.get(url, stream=True).raw) processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224') model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224') inputs = processor(images=image, return_tensors="pt") outputs = model(**inputs) logits = outputs.logits # model predicts one of the 1000 ImageNet classes predicted_class_idx = logits.argmax(-1).item() print("Predicted class:", model.config.id2label[predicted_class_idx]) ``` preprocessor_config.json: 100% 160/160 [00:00<00:00, 10.5kB/s] config.json: 100% 69.7k/69.7k [00:00<00:00, 3.60MB/s] model.safetensors: 100% 346M/346M [00:02<00:00, 162MB/s] ``` Predicted class: Egyptian cat ``` Here, we scrape a web page to get the articles and summarize them with a huggingface model using the inference API. ```python import requests # Function to fetch text from the API def fetch_text_from_api(): url = "https://lexper.p.rapidapi.com/v1.1/extract" querystring = { "url": "https://techcrunch.com/2023/11/25/neuralink-elon-musks-brain-implant-startup-quietly-raises-an-additional-43m/", "js_timeout": "30", "media": "true" } headers = { "X-RapidAPI-Key": "xxx", "X-RapidAPI-Host": "lexper.p.rapidapi.com" } response = requests.get(url, headers=headers, params=querystring) data = response.json() # Extract the relevant text from the API response # Adjust the following line according to the structure of your API response return data.get('article', {}).get('text', '') # Function to summarize the text using Hugging Face API def query_huggingface(payload): API_URL = "https://api-inference.huggingface.co/models/facebook/bart-large-cnn" headers = {"Authorization": f"Bearer {API_TOKEN}"} response = requests.post(API_URL, headers=headers, json=payload) return


# **Hugging Face Inference API**
## Hugging Face Tasks
response.json() # Fetch the text text_to_summarize = fetch_text_from_api() # Summarize the text summarization_payload = { "inputs": text_to_summarize, "parameters": {"do_sample": False}, } summary_response = query_huggingface(summarization_payload) print(summary_response) ``` > [{'summary_text': 'Elon Musk-founded company raises $43 million in new venture capital. The company is developing implantable chips that can read brain waves. Critics say the company has a toxic workplace culture and unethical research practices. In June, Reuters reported that the company was valued at about $5 billion.'}] >


# Conclusion
In this lesson, we learned to use the OpenAI Assistants API, which enables tools like Code Interpreter and Knowledge Retrieval for enhanced functionality. Essential components like `Threads` and `Messages` facilitate user interaction, with the Assistant processing inputs and generating responses. We also demonstrated how an AI assistant can be deployed in a tech support example, utilizing these tools and methodologies for effective customer interaction. We also explored Hugging Face's free Inference API, which offers many models that can solve different tasks. Through practical examples, we demonstrated how to authenticate, access models via the Model Hub, and perform various NLP tasks, highlighting the API's versatility and ease of use in handling complex AI challenges. Through Function Calling, the OpenAI models can access the Hugging Face models via the free Inference API. --- **RESOURCES** [OpenAI Assistants API: Walk-through and Coding a Research Assistant](https://medium.com/@datadrifters/openai-assistants-api-walk-through-and-coding-a-research-assistant-3ddb3d767e99) [GitHub - huggingface/api-inference-community](https://github.com/huggingface/api-inference-community/) - OpenAI API Docs [Product](https://openai.com/product) - Assistants API Colab notebook [Google Colaboratory](https://colab.research.google.com/drive/1GkjoTZUd2rpwIvxmCZreKo5ZUZN6i7vJ) - OpenAI Knowledge Retrieval [OpenAI Platform](https://platform.openai.com/docs/assistants/tools/knowledge-retrieval) - Function Calling OpenAI [Functions with OpenAI Assistant API](https://tmmtt.medium.com/functions-with-openai-assistant-api-47e1481a0b42)


LlamaIndex RAG-AGENT Query and Summarize Over Data

# I**ntroduction**
In this lesson, we explore the concept of agents in the LlamaIndex framework, with an emphasis on utilizing these agents as engines with internal reasoning and decision-making mechanisms. Creating an agent-based pipeline includes integrating our RAG-based application with data sources along with various tools. It is essential to remember that developing these tools for the agents requires a deep understanding of how users are likely to engage with the application and predict potential usage patterns. The goal of an RAG system is always to provide users with insightful content more effectively than extensive manual searches. Adding agents to our system is another step towards improving our product's user experience and decision-making ability. The LlamaIndex framework offers numerous possibilities for combining agents and tools to enhance the abilities of Large Language Models. We will examine the implementation of OpenAI agents with various data sources. Additionally, we'll create custom functions to boost the agent's capabilities in areas where they may lack information, such as mathematical operations. The rest of this lesson will demonstrate how these agents are capable of making decisions and integrating various resources to formulate a response. Before diving into codes, we must prepare our environment by installing the necessary packages and configuring the API keys. Execute the following command in your terminal to install the required packages using the Python Package Manager (PIP). Next, run the subsequent Python script to configure the API keys in your environment. Remember to obtain the keys from the OpenAI and Activeloop platforms and substitute them for the placeholders. ```bash pip install -q llama-index==0.9.14.post3 deeplake==3.8.8 openai==1.3.8 cohere==4.37 ``` ```bash import os os.environ['OPENAI_API_KEY'] = '<YOUR_OPENAI_API_KEY>' os.environ['ACTIVELOOP_TOKEN'] = '<YOUR_ACTIVELOOP_API_KEY>' ``` Now, let’s go through the next steps in detail.


# OpenAI Agent
## Step 1: **Defining Data Sources**
Discussing datasets in RAG mainly refers to data sources. It would be interesting to tag and track right from the start. This means keeping track of the general source of the data, whether it is from a specific book, documentation, or a blog. For example, the Towards AI RAG AI tutor currently has 5 “data sources”: Towards AI blogs, Activeloop documentation, LlamaIndex documentation, LangChain documentation, and HuggingFace documentation. Later, when we increase the dataset size with new data points, we add them to those sources or create new ones. Doing this process from the start will improve the chatbot’s efficiency by introducing the “routers” to focus on the related information source to answer a question. A key step in building a data-driven application with the LlamaIndex RAG system is discussing and selecting the appropriate dataset. The quality and relevance of the data are fundamental, as they directly influence the system's performance capabilities. A well-chosen dataset is essential to showcase and test the effectiveness of our RAG system accurately. It will be the same whether they are local files or hosted online with a vector store database like Deep Lake. However, note that using online tools like Deep Lake has built-in features to easily visualize, query, track, and manage your data. It is a good practice to start your RAG pipeline design with a small dataset, such as web articles. Setting up a foundational data environment that is manageable yet sufficiently rich is critical to ensuring a smooth start. This way, you can quickly test, debug, and, most importantly, understand your RAG system. You can easily query and evaluate responses on a dataset you control and grasp. The dataset for this lesson will comprise Nikola Tesla's life, work, and legacy, with detailed information about his innovations, personal history, and impact. We employ two text documents: the first with bold future predictions that Tesla mentioned during his lifetime and the second file with biographical details about his life. Let's import the files and set up the indexes. We will utilize a mix of data sources from the Deep Lake vector store for the first file and establish indexes from local storage for the second file. The initial step involves downloading the documents using the wget command. Alternatively, you can access and manually save the files from the URLs below. ```bash mkdir -p 'data/1k/' wget 'https://github.com/idontcalculate/data-repo/blob/main/machine_to_end_war.txt' -O './data/1k/tesla.txt' wget 'https://github.com/idontcalculate/data-repo/blob/main/prodigal_chapter10.txt' -O './data/1k/web.txt' ``` ### Store Indexes Deep Lake As previously stated, we'll read the first text file and process it for storage in Deep Lake. The `SimpleDirectoryReader` class in LlamaIndex can browse through a directory and transform text files into a `Document` object, facilitating processing. ```python from llama_index import SimpleDirectoryReader tesla_docs = SimpleDirectoryReader( input_files=["/content/data/1k/tesla.txt"] ).load_data() ``` We are now ready to establish a database on the Activeloop platform by specifying the organization ID (which defaults to your username) and naming the database. The `DeepLakeVectorStore` class is used to create an empty database. ```python from llama_index.vector_stores import DeepLakeVectorStore # By default, the organization id is your username. my_activeloop_org_id = "<YOUR_ORGANIZATION_ID>" my_activeloop_dataset_name = "LlamaIndex_tesla_predictions" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" # Create an index over the documnts vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=False) ``` ```python Your Deep Lake dataset has been successfully created! ``` Then, we can utilize the database object to create a storage context, allowing us to generate indexes (embeddings) and insert them into the database using the `VectorStoreIndex` class. ```python from llama_index.storage.storage_context import StorageContext from llama_index import VectorStoreIndex storage_context = StorageContext.from_defaults(vector_store=vector_store) tesla_index = VectorStoreIndex.from_documents(tesla_docs, storage_context=storage_context) ``` ```python Uploading data to deeplake dataset. 100%|██████████| 5/5 [00:00<00:00, 7.17it/s] /Dataset(path='hub://genai360/LlamaIndex_tesla_predictions', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression -------


# OpenAI Agent
## Step 1: **Defining Data Sources**
------- ------- ------- ------- text text (5, 1) str None metadata json (5, 1) str None embedding embedding (5, 1536) float32 None id text (5, 1) str None ``` The index we have created for the first file is ready to be integrated as a source in the pipeline. However, we must also process the second file before proceeding. ### Store Indexes Locally The method to save the index on your hard drive begins similarly to our earlier demonstration, employing the `SimpleDirectoryReader` class. ```python webtext_docs = SimpleDirectoryReader(input_files=["/content/data/1k/web.txt"]).load_data() ``` Just as we utilized the `StorageContext` class earlier for employing the DeepLake database as storage, we can apply the same configuration but specify a directory to store the indexes. The following script initially attempts to load any pre-existing indexes if they were previously computed. If not, it uses the `.persist()` method to store the indexes. As indicated by the output, the index is generated. If you execute this code block again, it will retrieve the stored checkpoint instead of reprocessing and regenerating indexes. ```python try: # Try to load the index if it is already calculated storage_context = StorageContext.from_defaults( persist_dir="/content/storage/webtext" ) webtext_index = load_index_from_storage(storage_context) print("Loaded the pre-computed index.") except: # Otherwise, generate the indexes webtext_index = VectorStoreIndex.from_documents(webtext_docs) webtext_index.storage_context.persist(persist_dir="/content/storage/webtext") print("Generated the index.") ``` ```python Generated the index. ``` With data acquired from two distinct sources, let's utilize the query engine and its tools to develop an agent capable of integrating this information.


# OpenAI Agent
## Step 2: ****Query Engine****
Once the index is established, the query engine used for searching and retrieving data from the index can be efficiently set up. ```python tesla_engine = tesla_index.as_query_engine(similarity_top_k=3) webtext_engine = webtext_index.as_query_engine(similarity_top_k=3) ``` The similarity parameter `top_k=3` is set to `3` for the search, which means to return the top 3 most similar results for a given query. As previously mentioned, the query engine tool comprises two distinct data sources. 1. The `tesla_engine` variable handles queries about general information. 2. The `webtext_engine` variable processes biographical data, focusing on inputs with factual content. This separation of data types ensures data quality when querying instead of always fetching from both sources with equal weight. With the query engine now constructed, the tools can be configured. We can use a combination of the `QueryEngineTool` class to create a new tool that includes a query engine and the `ToolMetaData` class, which assists in assigning names and descriptions to the tools. These descriptions will help the agent determine the most suitable data source based on the user's query. We will create a list of two tools, each representing one of our data sources. ```python from llama_index.tools import QueryEngineTool, ToolMetadata query_engine_tools = [ QueryEngineTool( query_engine=tesla_engine, metadata=ToolMetadata( name="tesla_1k", description=( "Provides information about Tesla's statements that refers to future times and predictions. " "Use a detailed plain text question as input to the tool." ), ), ), QueryEngineTool( query_engine=webtext_engine, metadata=ToolMetadata( name="webtext_1k", description=( "Provides information about tesla's life and biographical data. " "Use a detailed plain text question as input to the tool." ), ), ), ] ``` Here’s a clear schematic representation of our current system. The query engine is depicted at the top, signifying its role as the primary tool orchestrating everything. It is positioned centrally between the data sources and the process of formulating the final answer. It acts as a bridge between the proposed questions and their respective answers. ![Our baseline RAG pipeline with a query engine, data sources, and question-answer configuration.](LlamaIndex%20RAG-AGENT%20Query%20and%20Summarize%20Over%20Data%2088a5dfe909974fa8aade9542b8f3a1dd/LlamaIndex%20RAG-AGENT%20Query%20and%20Summarize%20Over%20Data%2091c649d25137470eb486285c008ba99c/image_2023-12-10_035919501.png) Our baseline RAG pipeline with a query engine, data sources, and question-answer configuration. After establishing the basic RAG mechanism with LlamaIndex, the next step is integrating an agent. This addition enables easy testing of the retrieval system. We can then add system design improvements and feature enhancements once the core functionality has been tested and verified.


# OpenAI Agent
## ****Step 3: The Agent****
Now, let’s set up our agent. In this case, it will be the OpenAI agent. Integrating the query engine tools into the `OpenAIAgent` module from LlamaIndex enables the agent to execute queries. Setting the `verbose` argument to True is excellent for debugging. It will allow us to investigate which tool the agent is using and the intermediate steps. You could set the argument to False to only receive the final output. ```python from llama_index.agent import OpenAIAgent agent = OpenAIAgent.from_tools(query_engine_tools, verbose=True) ``` And that’s it! Now that we have our agent, we can execute an interactive chat interface (REPL, Read-Eval-Print Loop) where the agent can receive inputs (like questions or prompts), process them, and return responses, making it a conversational agent capable of handling a dialogue or chat session. ```python agent.chat_repl() ``` ``` ===== Entering Chat REPL ===== Type "exit" to exit. Human: What influenced Nikola Tesla to become an inventor? STARTING TURN 1 --------------- === Calling Function === Calling function: webtext_1k with args: { "input": "What influenced Nikola Tesla to become an inventor?" } Got output: Nikola Tesla was influenced to become an inventor by his studies of mechanical vibrations. He observed the selective response of objects to vibrations and realized the potential for producing effects of tremendous magnitude on physical objects. This led him to pursue research in the field of high-frequency and high-potential currents, which eventually resulted in his groundbreaking inventions. ======================== STARTING TURN 2 --------------- Assistant: Nikola Tesla was influenced to become an inventor by his studies of mechanical vibrations. He observed the selective response of objects to vibrations and realized the potential for producing effects of tremendous magnitude on physical objects. This led him to pursue research in the field of high-frequency and high-potential currents, which eventually resulted in his groundbreaking inventions. Human: exit ``` <aside> 💡 To debug tools in development, a practical approach involves querying the agent about its tools. This process includes asking the agent to - detail the tools at its disposal, - the arguments these tools accept, the significance of these arguments, - and the intended use of each tool. We can then analyze the agent's responses to identify prompt deficiencies or understand why the agent might struggle to utilize a tool under development effectively. </aside>


# Agents with Custom Function
We explored the potential of creating Query Engine tools to enhance an OpenAI-based agent with additional data sources. We observed the capability of agents to select the appropriate tool based on the user's prompt. This decision-making ability can be applied across a broad range of applications. For instance, one area where Large Language Models typically fall short is mathematical operations. A basic addition or subtraction equation, which may seem straightforward to many, can be challenging for these models. A practical solution to this issue is to equip the models with tools like a calculator for use as needed. This section will create a custom function that a chatbot can access for essential multiplication or addition calculations whenever required. Initially, we must define a custom function tailored to each task. These custom functions can accept an arbitrary number of inputs and generate an output. Their capabilities can range from a simple addition operation, as in our example, to more complex tasks such as conducting web searches, querying other Large Language Models, or utilizing data from external APIs to answer a question. ```python def multiply(a: int, b: int) -> int: """Multiply two integers and returns the result integer""" return a * b def add(a: int, b: int) -> int: """Add two integers and returns the result integer""" return a + b from llama_index.tools import FunctionTool multiply_tool = FunctionTool.from_defaults(fn=multiply, name="multiply") add_tool = FunctionTool.from_defaults(fn=add, name="add") all_tools = [multiply_tool, add_tool] ``` The above code establishes two functions, titled 'add' and 'multiply'. It is crucial in this setup to specify data types for the input arguments (`a:int, b:int`), the return type of the function (`->int`), and a concise explanation of the function's purpose, provided within the triple quotes beneath the function name. These details will be used by the `FunctionTool` class’s `.from_defaults()` method to form a description of the function, which can then be used by the agent. The final variable holds a list of all the available tools. These tools can be used to construct an `ObjectIndex`, which is a wrapper class linking a `VectorStoreIndex` with multiple possible tools. Initially, it's necessary to utilize the `SimpleToolNodeMapping` tool to transform the tool implementations into nodes and then tie everything together. ```python from llama_index import VectorStoreIndex from llama_index.objects import ObjectIndex, SimpleToolNodeMapping tool_mapping = SimpleToolNodeMapping.from_objects(all_tools) obj_index = ObjectIndex.from_objects( all_tools, tool_mapping, VectorStoreIndex, ) ``` Note that we do not incorporate any data source in this implementation. This approach is intentional, as we aim to enhance the capabilities of Large Language Models with additional tools. In the next code block, you will see that we are utilizing the defined object index as a retriever! This implies that within the LlamaIndex framework, the custom functions are treated as additional data sources. So, we describe the agent object using the `FnRetrieverOpenAIAgent` class. ```python from llama_index.agent import FnRetrieverOpenAIAgent agent = FnRetrieverOpenAIAgent.from_retriever( obj_index.as_retriever(), verbose=True ) ``` Ultimately, we can employ the agent to ask questions, and the agent utilizes the multiply function to provide answers. ```python agent.chat("What's 12 multiplied by 22? Make sure to use Tools") ``` ``` STARTING TURN 1 --------------- === Calling Function === Calling function: multiply with args: { "a": 12, "b": 22 } Got output: 264 ======================== STARTING TURN 2 --------------- AgentChatResponse(response='12 multiplied by 22 is 264.', sources=[ToolOutput(content='264', tool_name='multiply', raw_input={'args': (), 'kwargs': {'a': 12, 'b': 22}}, raw_output=264)], source_nodes=[]) ``` In the previous example, we specified in the prompt that the agent should utilize the tools. Additionally, it's possible to employ the `tool_choice` argument to explicitly direct the agent to use specific tools or to use the `auto` keyword to let the agent decide. ```python response = agent.chat( "What is 5


# Agents with Custom Function
+ 2?", tool_choice="add" ) ``` ``` STARTING TURN 1 --------------- === Calling Function === Calling function: add with args: { "a": 5, "b": 2 } Got output: 7 ======================== STARTING TURN 2 --------------- AgentChatResponse(response='5 + 2 is equal to 7.', sources=[ToolOutput(content='7', tool_name='add', raw_input={'args': (), 'kwargs': {'a': 5, 'b': 2}}, raw_output=7)], source_nodes=[]) ```


# Agents from LlamaHub
Agents can offer a broad range of functionalities, significantly extending the capabilities of Large Language Models into unexplored realms. LlamaHub streamlines the curation, sharing, and usage of more than 30 agents, achievable with just one line of code. We have already explored its application for scrapping data from Wikipedia in the LlamaIndex Unlocked lesson. To see a complete list of implemented agents, [click here](https://llamahub.ai/?tab=tools). <aside> 💡 We have reviewed the fundamentals of agents and tools by examining a selection of widely used agents in LlamaIndex. For more options and details, you can refer to the [documentation](https://docs.llamaindex.ai/en/stable/understanding/putting_it_all_together/agents.html). </aside>


# Conclusion
In this lesson, we discussed how to utilize agents to enhance the capabilities of Large Language Models by integrating new tools that unlock their potential. We experimented with employing these agents as decision-making functions to incorporate various data sources in response to user queries. Additionally, we explored their use as reasoning machines, combined with custom functions, to further amplify their abilities. The ability to make function calls is a potent aspect of designing agents, enabling the easy integration of additional information into the model from virtually any imaginable resource. --- >> [Notebook](https://colab.research.google.com/drive/1dYeR3M14BOwC7hkWS5Yo_zMFD4HMJpQn?usp=sharing). --- ### R**esources:** the RAG-AGENT example notebook: [Google Colaboratory](https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/agent/openai_agent_with_query_engine.ipynb#scrollTo=f9f3158a-7647-4442-8de1-4db80723b4d2) - LlamaHub on GitHub: [](https://github.com/run-llama/llama-hub/tree/main/llama_hub/tools/notebooks) - data agents [Data Agents - LlamaIndex 🦙 0.9.11.post1](https://docs.llamaindex.ai/en/stable/module_guides/deploying/agents/root.html) - [https://docs.llamaindex.ai/en/latest/examples/agent/openai_agent_with_query_engine.html](https://docs.llamaindex.ai/en/latest/examples/agent/openai_agent_with_query_engine.html) - [https://docs.llamaindex.ai/en/latest/examples/agent/multi_document_agents.html](https://docs.llamaindex.ai/en/latest/examples/agent/multi_document_agents.html)


Course Introduction and Logistics

# Course Introduction and Logistics
## Introduction to the “Retrieval Augmented Generation for Production with LlamaIndex and LangChain” Course
Activeloop, Towards AI, and the Intel Disruptor Initiative are excited to collaborate to bring Gen AI 360: Foundational Model Certification Course for aspiring Generative AI professionals, executives, and enthusiasts of tomorrow. Following the success of our "[LangChain & Vector Databases In Production](Course%20Introduction%20and%20Logistics%20d5eb79d71a2f4d1e9aa986a2f0538b3f.md)" and "[Training and Fine-tuning LLMs for Production](https://www.notion.so/Careers-Activeloop-9dad048520b6452e8954830d61250c5a?pvs=21)” courses, we're excited to welcome you to the third part of the series: “Retrieval Augmented Generation for Production with LlamaIndex and LangChain.” In this course, you'll learn how to build on RAG techniques learned in the first Langchain and Vector DBs in Production course. Apart from a primer on novel LangChain RAG frameworks, you will learn advanced RAG techniques with LlamaIndex and how to build RAG agents and RAG evaluation systems. This course will guide you on the optimal methods and practices for getting RAG production-ready with plenty of applied industry project examples. Let's get started! ### **Why This Course?** The “Retrieval Augmented Generation for Production with LlamaIndex and LangChain” course provides the theoretical knowledge and practical skills necessary to build advanced RAG products. Many human tasks across various industries can be assisted with AI by combining LLMs, prompting, RAG, and fine-tuning workflows. We are huge fans of RAG because it helps with 1) reducing hallucinations by limiting the LLM to answer based on existing documentation, 2) helping with explainability, error checking, and copyright issues by clearly referencing its sources for each comment, 3) giving private/specific or more up-to-date data to the LLM, 4) and not relying on black box LLM training/fine tuning for what the models know and has memorized. We touched upon basic RAG in our first Langchain and Vector DBs course, but building more advanced and reliable products requires more complex techniques and iterations of the model. The 'Retrieval Augmented Generation for Production with LlamaIndex and LangChain' course aims to provide you with the theoretical knowledge and practical skills necessary to develop products and applications centered on RAG. A fundamental pillar of our course is the focus on hands-on learning. Real-world application and experimentation are crucial for a deep understanding and effective use of RAG techniques. In this course, you will move beyond basic RAG apps, develop these applications with more advanced techniques, build RAG agents, and evaluate the performance of RAG systems. ### **Who Should Take This Course?** Whether planning to build a chat with data application for your organization or just learning how to leverage Generative AI in various industries, this course is for you. The course addresses critical issues such as reducing hallucinations in AI outputs, enhancing explainability, addressing copyright concerns, and offering more tailored, up-to-date data inputs. We go beyond basic RAG applications, equipping you with the skills to create more complex, reliable products with tools like LangChain, LlamaIndex, and Deep Memory. Emphasizing hands-on learning, this course is a gateway to mastering advanced RAG techniques and applications in real-world scenarios. Please note that prior knowledge of coding and Python is a prerequisite. ### **What You Will Learn** You will start by learning the basic RAG tools, such as loading, indexing, storing, and querying in both Langchain and LlamaIndex. We’ll also demystify the two libraries to help you select the right one when working with RAG or other LLM applications. You will then move towards more advanced RAG techniques aimed at surfacing and using more relevant information from the dataset. We cover techniques such as Query expansion, Transformation reranking, recursive retrieval, optimization, and production tips and techniques with LlamaIndex. We also introduce how better embedding management through Activeloop’s Deep Memory can be used to improve accuracy. We then progress to the exciting stuff: learning how


Course Introduction and Logistics

# Course Introduction and Logistics
## Introduction to the “Retrieval Augmented Generation for Production with LlamaIndex and LangChain” Course
to build RAG agents in Langchain and Llamaindex, an introduction to OpenAI assistants and some other tools & models that can be used in RAG products. We conclude with a summary of RAG evaluation techniques in LlamaIndex together with an introduction to Langsmith in Langchain. ### Is the Course Free? Yes, the course is entirely free for everybody. However, running the project examples yourself will cost you some API and cloud credits. ### **Certification** By participating in this course and completing the quizzes at the end of each chapter, you will have the opportunity to earn a certification in using Deep Lake - a valuable addition to your professional credentials. This certification program, offered at no cost, forms part of the Deep Lake Foundational Model Certification Program in collaboration with Intel Disruptor Initiative and Towards AI.


Course Introduction and Logistics

# Course Introduction and Logistics
## Course Logistics
Here's everything you need to know about the course. ### **Course Hosting and Pace** This course is hosted by **Activeloop**. It is designed as a **self-paced** learning journey, allowing you to proceed at your own comfort. The online format provides flexibility to engage with the lessons whenever it best suits you. At the end of each module, you can test your new knowledge with multiple-choice quizzes, which are mandatory to continue the course. You will receive your course certification after completing all the quizzes. ### **Community Support** Have questions about this course or specific lessons? Want to exchange ideas with fellow learners? We encourage active interaction in the dedicated forum in the *[Towards AI’s Learn AI Together Discord Community](https://discord.com/invite/learnaitogether), a* vibrant community of over 55,000 AI experts and enthusiasts. Our community has a dedicated channel for this course where you can ask questions and share insights. For queries specifically related to Deep Lake, please join the *[Deep Lake Slack community](https://join.slack.com/t/hubdb/shared_invite/zt-ivhsj8sz-GWv9c5FLBDVw8vn~sxRKqQ),* where experts and users will be ready to assist. ### **Required Platforms, Tools, and Cloud Tokens** The course involves practical projects and exercises that require various tools and platforms. These will be thoroughly guided in the individual lessons. However, the main platforms that you will use throughout the course are: - **Activeloop’s Deep Lake** - **Open AI** - **LlamaIndex** - **Langchain** - **Langchain’s Langsmith**


Course Introduction and Logistics

# Course Introduction and Logistics
## **What is Activeloop?**
[Activeloop](https://www.activeloop.ai/) is a tech company dedicated to building data infrastructure optimized for deep-learning applications. It offers a platform that seamlessly connects unstructured data types, like audio, video, and images, to machine learning models. Their main product, Deep Lake, ensures data streaming, scalable machine learning pipelines, and dataset version control. Such infrastructures are particularly beneficial when dealing with the demands of training and fine-tuning models for production. ### **What is Deep Lake?** Deep Lake is an open-source data lake designed for deep learning applications. It retains essential features of traditional data lakes, including SQL queries, ACID transactions, and dataset visualization. It specializes in storing complex data in tensor form, efficiently streaming data to deep learning frameworks. Built to be serverless on a columnar storage format, it also offers native version control and in-browser data visualization, complementing the needs of LLM training and deployment processes. ### Activeloop **Credits - Free Extended Trial for Deep Lake Growth Plan for a Month** As a part of the course, all course takers can redeem a free extended trial of one month for the Activeloop Growth plan by redeeming GENAI360 promo code at checkout. To redeem the plan ### **How to set up a Deep Lake account?** To set up a Deep Lake account, navigate to the [app’s registration page](https://app.activeloop.ai/register/) and sign up. Follow the on-screen instructions and add the required details. Once you've verified your email and established a secure password, your account will be active and ready for use. **How to get the Deep Lake API token?** 1. After logging in, you should see your homepage. You should now see a “Create API token” button at the top of your homepage. Click on it, and you’ll get redirected to the “API tokens” page. This is where you can generate, manage, and revoke your API keys for accessing Deep Lake. 2. Click on the "Create API token" button. You should see a popup asking for a token name and an expiration date. By default, the token expiration date is one year. Once you’ve set the token name and its expiration date, click the “Create API token” button. 3. You should now see a green banner saying that the token has been successfully generated, along with your new API token, on the “API tokens” page. To copy your token to your clipboard, click the square icon on its right.


Course Introduction and Logistics

# Course Introduction and Logistics
## **Coding Environment and Packages**
Before starting this course, you need to ensure that you have the appropriate coding environment ready. Please make sure to use a Python version equal to or later than **3.8.1**. You can set up your environment by choosing one of the following options: 1. Having a code editor installed on your computer. A popular coding environment is [Visual Studio Code](https://code.visualstudio.com/). 2. Using Python virtual environments to manage Python libraries. 3. Alternatively, you could use Google Colab notebooks. You will need the following packages to successfully execute the sample codes provided in each lesson. They can be installed using the `pip` package manager. ``` deeplake==3.6.19 openai==0.27.8 tiktoken==0.4.0 transformers==4.32.0 torch==2.0.1 numpy==1.23.5 deepspeed==0.10.1 trl==0.7.1 peft==0.5.0 wandb==0.15.8 bitsandbytes==0.41.1 accelerate==0.22.0 tqdm==4.66.1 neural_compressor===2.2.1 onnx===1.14.1 pandas==2.0.3 scipy==1.11.2 ``` While we strongly recommend installing the latest versions of these packages, please note that the codes have been tested with the versions specified in parentheses. Moreover, specific lessons may require the installation of additional packages, which will be explicitly mentioned. The following code will demonstrate how to install a package using pip. ```bash pip install deeplake # Or: (to install an specific version) # pip install deeplake==3.6.5 ``` <aside> 💡 If you encounter any issues while installing the packages, don't hesitate to reach out to us via Activeloop's [Slack channel](https://slack.activeloop.ai/) or the [live chat](https://www.activeloop.ai/) on the website. We are here to assist you. </aside> ### **Google Colab** Google Colaboratory, popularly known as Google Colab, is a *free cloud-based Jupyter notebook environment*. Data scientists and engineers widely use it to train machine learning and deep learning models using CPUs, GPUs, and TPUs. Google Colab comes with an array of features, such as: - Free access to GPUs and TPUs for accelerated model training. - A web-based interface for a service running on a virtual machine, eliminating the need for local software installation. - Seamless integration with Google Drive and GitHub. To use Google Colab, all you need is a Google account. You can run terminal commands directly in notebook cells by appending an exclamation mark (!) before the command. Every notebook created in Google Colab gets stored in your Google Drive for easy access. A convenient way of using API keys in Colab involves: 1. Saving them in a file named `.env` on your Google Drive. Here’s how the file should be formatted to save the Activeloop token and the OpenAI API key. ```python ACTIVELOOP_TOKEN=your_activeloop_token OPENAI_API_KEY=your_openai_key ``` 1. Mounting your Google Drive on your Colab instance. 2. Loading them as environment variables using the **`dotenv`** library, like in the following code. ```python from dotenv import load_dotenv load_dotenv('/content/drive/MyDrive/path/to/.env') ``` ### **Creating Python Virtual Environments** Python virtual environments offer an excellent solution for managing Python libraries and avoiding package conflicts. They create isolated environments for installing packages, ensuring that your packages and their dependencies are contained within that environment. This setup provides clean and isolated environments for your Python projects. Begin by executing the `python` command in your terminal to confirm that the Python version is either equal to or greater than 3.8.1. Then follow these steps to create a virtual environment: 1. Create a virtual environment using the command **`python -m venv my_venv_name`**. 2. Activate the virtual environment by executing **`source my_venv_name/bin/activate`**. 3. Install the required libraries and run the code snippets from the lessons within the virtual environment. *Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries.* ### **📝 Legal Note on Availability of Credits & Applicable Terms 📝** **Availability of Credits:** - **Validity:** Credits acquired upon completion of the GenAI360 Foundational Model Certification Course must be redeemed within [12] months of the


Course Introduction and Logistics

# Course Introduction and Logistics
## **Coding Environment and Packages**
course completion date, after which they will become void. - **Usage:** Credits are applicable for use across select programs, courses, workshops, or other specified opportunities offered by Activeloop and/or other partners and cannot be transferred to external platforms or exchanged for cash or other forms of payment. **Terms & Conditions:** 1. **Eligibility:** Credits are available only to qualifying participants who have successfully completed the GenAI360 Foundational Model Certification Course and have adhered to all course and evaluation criteria. 2. **Redemption:** Credits can only be redeemed by the individual participant to whom they were issued and are non-transferable to other individuals or entities. 3. **Modification:** Activeloop reserves the right to modify the availability, applicability, or value of credits without prior notice. Participants will be notified of any such changes via registered email. 4. **Verification:** All participant details and attached documents will undergo a verification process. Any discrepancies or fraudulent information may result in the forfeiture of credits and possible exclusion from future programs. 5. **Data Protection:** Participant data will be handled with utmost confidentiality and will only be used for the purpose of credit redemption in compliance with applicable data protection regulations. 6. **Limitation:** The credits are not stackable and cannot be combined with other ongoing offers or discounts unless explicitly mentioned. 7. **Discrepancies:** In case of any discrepancies or issues related to credit redemption, decisions made by Activeloop will be considered final. 8. **Communication:** Participants agree to receive communications regarding the credit redemption process and other relevant information from GenAI360. 9. **Governing Law:** The credit redemption process and associated terms are governed by the applicable laws in the jurisdiction of Activeloop’s registered location. By redeeming the credits, participants acknowledge and agree to adhere to the aforementioned terms and conditions. Any violation of the same may result in appropriate actions as deemed suitable by Activeloop.


Chat with Your Code LlamaIndex and Activeloop Deep

# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
This guide is your quickstart toolkit for integrating LlamaIndex with Activeloop Deep Lake. You'll learn how to effortlessly index GitHub repositories into Deep Lake and interact with your code through natural language queries.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## What are LlamaIndex and Activeloop Deep Lake?
### LlamaIndex: Your Data Framework for LLMs [LlamaIndex](https://gpt-index.readthedocs.io/en/stable/) is a bridge between your data and Language Learning Models (LLMs). Whether your data resides in APIs, SQL databases, or PDFs, LlamaIndex ingests and structures it into a format easily consumable by LLMs. It offers features like data connectors for various data sources, indexing capabilities for quick retrieval, and natural language query engines that make your data not just accessible but also interactive. It also offers [Llama Hub](https://llamahub.ai/), a platform aggregating custom plugins for all data types. ### Activeloop Deep Lake: Optimized Data Lake for ML [Activeloop](https://www.activeloop.ai/) Deep Lake is a data lake solution specifically designed for machine learning workflows. Unlike traditional data lakes, it's optimized for quick data retrieval and manipulation, making it an ideal choice for machine learning projects that require efficient data access. It supports various data types and formats, from images and videos to more complex data structures, while maintaining high performance. You can create local vector stores or use the managed serverless service. ### The Synergy When LlamaIndex and Activeloop Deep Lake are combined, they offer a robust, efficient, and interactive data management solution. LlamaIndex takes care of ingesting and structuring your data, while Activeloop Deep Lake provides optimized storage and retrieval capabilities. In this guide, we'll see how to store and interact with your code repositories through natural language queries, offering a unique and powerful way to manage and understand your codebase.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Requirements
Before getting started, make sure you have the following: - [Python](https://www.python.org/downloads/) - Version 3.7 or newer is required. - An active account on OpenAI, along with an [OpenAI API key](https://platform.openai.com/account/api-keys). - A Deep Lake account, complete with a [Deep Lake API key](https://app.activeloop.ai/?utm_source=referral&utm_medium=platform&utm_campaign=signup_promo_settings&utm_id=plg). - a 'classic' personal token from [GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token).


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Getting Started
Creating a new Python virtual environment for this project is strongly advised. It helps maintain a tidy workspace by keeping dependencies in one place. - Create a Python virtual environment with: ```python python3 -m venv repo-ai ``` Then activate it with: ```python source repo-ai/bin/activate ```


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Install the required packages
This project will need Python packages, including LLamaIndex and Deep Lake. Run the following pip command: ```jsx pip install llama-index deeplake openai python-dotenv ``` Let's understand what we are installing and why. ### 1. `llama-index` ### What is it? LlamaIndex is a data framework that works with Language Learning Models (LLMs). It helps ingest, structure, and make data accessible through natural language queries. ### Key Features: - **Data Connectors**: Ingest data from various sources like APIs, SQL databases, and PDFs. - **Data Indexing**: Structure the ingested data for quick and efficient retrieval. - **Query Engines**: Enable natural language queries to interact with your data. ### Use-Case in the guide: In the context of this guide, LlamaIndex will be used to index GitHub repositories and make them queryable through natural language interfaces. ### 2. `deeplake` ### What is it? Activeloop Deep Lake is a specialized data lake optimized for machine learning workflows. It allows for efficient storage and retrieval of various data types. ### Key Features: - **Optimized Storage**: Designed for quick data retrieval, ideal for machine learning applications. - **Data Type Support**: Handles multiple data types like images, videos, and complex data structures. ### Use-Case in the guide: Deep Lake is the storage layer where the GitHub repositories indexed by LlamaIndex will be stored. ### 3. `openai` ### What is it? The OpenAI Python package provides an interface to OpenAI's GPT models and other services. It allows you to make API calls to interact with these models. ### Key Features: - **API Integration**: Easy integration with OpenAI's GPT models. - **Text Generation**: Generate text based on the model's training data and capabilities. ### Use-Case in the guide: LLamaIndex uses this package to interact with the OpenAI models. ### 4. `python-dotenv` ### What is it? Python-dotenv is a library that allows you to specify environment variables in a `.env` file, making it easier to manage configurations. ### Key Features: - **Environment Variable Management**: Store configuration variables in a `.env` file. - **Easy Import**: Automatically import variables from `.env` into your Python environment. ### Use-Case in the guide: This package manages the API keys.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## How does LLamaIndex work?
In the context of leveraging LlamaIndex for data-driven applications, the underlying logic and workflow are pretty simple. Here's a breakdown: - **Load Documents**: The first step involves loading your raw data into the system. You can do this manually, directly inputting the data, or through a data loader that automates the process. LlamaIndex offers specialized data loaders that can ingest data from various sources, transforming them into Document objects, and you can find many plugins on Llama Hub. This is a crucial step as it sets the stage for the subsequent data manipulation and querying functionalities. - **Parse the Documents into Nodes**: Once the documents are loaded, they are parsed into Nodes, essentially structured data units. These Nodes contain chunks of the original documents and carry valuable metadata and relationship information. This parsing process is vital as it organizes the raw data into a structured format, making it easier and more efficient for the system to handle. - **Construct an Index from Nodes or Documents**: After the Nodes are prepared, an index is constructed to make the data searchable and queryable. Depending on your needs, this index can be built directly from the original documents or the parsed Nodes. The index is often stored in structures like `VectorStoreIndex`, optimized for quick data retrieval. This step is the system's heart, turning your structured data into a robust, queryable database. - **Query the Index**: With the index in place, the final step is to query it. A query engine is initialized, allowing you to make natural language queries against the indexed data. This is where the magic happens: you can conversationally ask the system questions, and it will sift through the indexed data to provide accurate and relevant answers.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Let's code
Now that the environment is ready and all the explanations are out of the way let's start with some code. In your project's directory, create a new file named `.env`, paste the following, and add your API keys: ```python GITHUB_TOKEN="YOUR_GH_CLASSIC_TOKEN" OPENAI_API_KEY="YOUR_OPENAI_KEY" ACTIVELOOP_TOKEN="YOUR_ACTIVELOOP_TOKEN" DATASET_PATH="hub://YOUR_ORG/repository_vector_store" ``` > Remember to edit this line DATASET_PATH="hub://YOUR_ORG/repository_vector_store", adding your Activeloop organization's ID to the URL. > After this, create a new file named `main.py` and paste the following code: ```python import os import textwrap from dotenv import load_dotenv from llama_index import download_loader from llama_hub.github_repo import GithubRepositoryReader, GithubClient from llama_index import VectorStoreIndex from llama_index.vector_stores import DeepLakeVectorStore from llama_index.storage.storage_context import StorageContext import re # Load environment variables load_dotenv() # Fetch and set API keys openai_api_key = os.getenv("OPENAI_API_KEY") active_loop_token = os.getenv("ACTIVELOOP_TOKEN") dataset_path = os.getenv("DATASET_PATH") def parse_github_url(url): pattern = r"https://github\.com/([^/]+)/([^/]+)" match = re.match(pattern, url) return match.groups() if match else (None, None) def validate_owner_repo(owner, repo): return bool(owner) and bool(repo) def initialize_github_client(): github_token = os.getenv("GITHUB_TOKEN") return GithubClient(github_token) def main(): # Check for OpenAI API key openai_api_key = os.getenv("OPENAI_API_KEY") if not openai_api_key: raise EnvironmentError("OpenAI API key not found in environment variables") # Check for GitHub Token github_token = os.getenv("GITHUB_TOKEN") if not github_token: raise EnvironmentError("GitHub token not found in environment variables") # Check for Activeloop Token active_loop_token = os.getenv("ACTIVELOOP_TOKEN") if not active_loop_token: raise EnvironmentError("Activeloop token not found in environment variables") github_client = initialize_github_client() download_loader("GithubRepositoryReader") github_url = input("Please enter the GitHub repository URL: ") owner, repo = parse_github_url(github_url) while True: owner, repo = parse_github_url(github_url) if validate_owner_repo(owner, repo): loader = GithubRepositoryReader( github_client, owner=owner, repo=repo, filter_file_extensions=( [".py", ".js", ".ts", ".md"], GithubRepositoryReader.FilterType.INCLUDE, ), verbose=False, concurrent_requests=5, ) print(f"Loading {repo} repository by {owner}") docs = loader.load_data(branch="main") print("Documents uploaded:") for doc in docs: print(doc.metadata) break # Exit the loop once the valid URL is processed else: print("Invalid GitHub URL. Please try again.") github_url = input("Please enter the GitHub repository URL: ") print("Uploading to vector store...") # ====== Create vector store and upload data ====== vector_store = DeepLakeVectorStore( dataset_path=dataset_path, overwrite=True, runtime={"tensor_db": True}, ) storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents(docs, storage_context=storage_context) query_engine = index.as_query_engine() # Include a simple question to test. intro_question = "What is the repository about?" print(f"Test question: {intro_question}") print("=" * 50) answer = query_engine.query(intro_question) print(f"Answer: {textwrap.fill(str(answer), 100)} \n") while True: user_question = input("Please enter your question (or type 'exit' to quit): ") if user_question.lower() == "exit": print("Exiting, thanks for chatting!") break print(f"Your question: {user_question}") print("=" * 50) answer = query_engine.query(user_question) print(f"Answer: {textwrap.fill(str(answer), 100)} \n") if __name__ == "__main__": main() ``` ### Main Function (`main()`)


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Understanding the code
At first glance, a lot is happening here; let's review it. Below is a step-by-step breakdown: ### Initialization and Environment Setup 1. **Import Required Libraries**: The script starts by importing all the necessary modules and packages. 2. **Load Environment Variables**: Using `dotenv`, it loads environment variables stored in the `.env` file. This is where API keys and tokens are stored securely. ### Helper Functions 1. **`parse_github_url(url)`:** This function takes a GitHub URL and extracts the repository owner and name using regular expressions. 2. **`validate_owner_repo(owner, repo)`:** Validates that both the repository owner and name are present. 3. **`initialize_github_client()`:** Initializes the GitHub client using the token fetched from the environment variables. 4. **API Key Checks**: Before proceeding, the script checks for the presence of the OpenAI API key, GitHub token, and Activeloop token, raising an error if any are missing. 5. **Initialize GitHub Client**: Calls `initialize_github_client()` to get a GitHub client instance. 6. **User Input for GitHub URL**: Asks the user to input a GitHub repository URL. 7. **URL Parsing and Validation**: Parses the URL to get the repository owner and name and validates them. 8. **Data Loading**: If the URL is valid, it uses `GithubRepositoryReader` from `llama_index` to load the repository data, specifically Python and Markdown files. 9. **Indexing**: The loaded data is then indexed using `VectorStoreIndex` and stored in a DeepLake vector store. This makes the data queryable. 10. **Query Engine Initialization**: Initializes a query engine based on the indexed data. 11. **Test Query**: Performs a test query to demonstrate the system's operation. 12. **User Queries**: Enters a loop where the user can input natural language queries to interact with the indexed GitHub repository. The loop continues until the user types 'exit'. ### Execution Entry Point - The script uses the standard `if __name__ == "__main__":` Python idiom to ensure that `main()` is called when the script is executed directly.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## A Closer Look at the GithubRepositoryReader loader
For this project, we used the `github_repo` data loader from LLamaIndex, and you can find its documentation on the [Llama Hub](https://llamahub.ai/l/github_repo). This is the part of the code taking care of that: ```python loader = GithubRepositoryReader( github_client, owner=owner, repo=repo, filter_file_extensions=([".py", ".js",".ts", ".md"], GithubRepositoryReader.FilterType.INCLUDE), verbose=False, concurrent_requests=10, ) ``` When it comes to loading data from a GitHub repository, the star of the show is the `GithubRepositoryReader` class. This class is a powerhouse designed to fetch, filter, and format repository data for indexing. Let's break down its key components: - **GitHub Client**: You'll first notice that the initialized GitHub client is passed into `GithubRepositoryReader`. This client provides authenticated access to GitHub repositories. - **Repository Details**: Next, the repository owner and name are specified. These are extracted from the URL you input, ensuring the data is fetched from the correct source. This is to give a nice message in the console. - **File Type Filters**: One of the most flexible features here is the ability to specify which file types to load. In this example, we're focusing on Python, JavaScript, TypeScript, and Markdown files. The inclusion of Markdown files is so the reader will also pull in README files, offering valuable context for the language model's understanding. - **Verbose Logging**: If you're the kind of person who likes to see every detail, you can enable verbose logging. This will print out detailed logs of the data loading process. - **Concurrent Requests**: This is where you can speed things up. The number of concurrent requests specifies how many data-fetching operations will happen simultaneously. A word of caution, though: cranking this number up could make you hit GitHub's rate limits, so tread carefully. Once all these parameters are set, the `load_data()` method swings into action. It fetches the repository data and neatly packages it into a list of `Document` objects, ready for the next stage— indexing. Let's get into the intricacies of the indexing part of the code, which is arguably one of the most crucial steps in the entire process.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Indexing: Transforming Data into Queryable Intelligence
```python # ====== Create vector store and upload data ====== vector_store = DeepLakeVectorStore( dataset_path=dataset_path, overwrite=True, runtime={"tensor_db": True}, ) storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents(docs, storage_context=storage_context) query_engine = index.as_query_engine() ``` After the data is loaded, the next task is to make this raw information searchable and retrievable. This is where the `VectorStoreIndex` and `DeepLakeVectorStore` come into play. Let's dissect how they work: - **Vector Store**: The first thing that happens is the creation of a `DeepLakeVectorStore`. Think of this as a specialized database designed to hold vectors. It's a storage unit and an enabler for high-speed queries. The `dataset_path` parameter specifies where this vector store will reside, and the `overwrite` flag allows you to control whether existing data should be replaced. The parameter `runtime={"tensor_db": True}` specifies that the vector store will use the Managed Tensor Database for storage and query execution on the Deep Lake infrastructure. The default here is to create a cloud vector DB. Instead, You can create a local vector DB by changing `dataset_path` to the name of the directory you want to use as DB. For example - dataset_path = "repository_db" The vector store will create a directory with the name specified in `dataset_path`. - **Storage Context**: Next up is the `StorageContext`, which essentially acts as a manager for the vector store. It ensures the vector store is accessible and manageable throughout the indexing process. - **From Documents to Index**: The `VectorStoreIndex.from_documents()` method is the workhorse here. It takes the list of `Document` objects you got from the GitHub repository and transforms them into a searchable index. This index is stored in the previously initialized `DeepLakeVectorStore`.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## The Interactive Finale: Querying and User Engagement
After the meticulous data loading and indexing process, the stage is set for the user to interact with the system. This is where the query engine comes into play. - **Query Engine**: Once the index is built, a query engine is initialized using `index.as_query_engine()`. You'll interact with this engine when you want to search through the GitHub repository. It's optimized for speed and accuracy, ensuring your natural language queries return the most relevant results. - **Introductory Question**: The code starts with an introductory question: "What is the repository about?" This serves multiple purposes. It acts as a litmus test to ensure the system is operational and gives the user an immediate sense of what questions can be asked. - **Formatting and Display**: The answer is then formatted to fit within a 100-character width for better readability, thanks to Python's `textwrap` library. - **User Input**: The code enters an infinite loop, inviting the user to ask questions. The user can type any query related to the GitHub repository, and the system will attempt to provide a relevant answer. - **Exit Strategy**: The loop continues indefinitely until the user types 'exit', providing a simple yet effective way to end the interaction. - **Query Execution**: Each time the user asks a question, the `query_engine.query()` method is called. This method consults the index built earlier and retrieves the most relevant information. - **Answer Presentation**: Like the introductory question, the answer to the user's query is formatted and displayed. This ensures that regardless of the complexity or length of the solution, it's presented in a readable manner.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Running the code
Now, you're fully equipped to index and query GitHub repositories. To tailor the system to your specific needs, pay attention to the `filter_file_extensions` parameter. This is where you specify which types of files you'd like to include in your index. The current setting—`[".py", ".js", ".ts", ".md"]`—focuses on Python, JavaScript, TypeScript, and Markdown files. Consider storing these extensions in your `.env` file for a more dynamic setup. You can easily switch between different configurations without modifying the codebase. It's a best practice that adds a layer of flexibility to your system. Run the command to start: `python3 main.py` The console will ask for a repository URL. To test, I used my repository explaining [how to build a ChatGPT plugin using Express.js](https://github.com/soos3d/chatgpt-plugin-development-quickstart-express), so this only has `.js` extensions. This is how the console will look during the interaction: ```python Please enter the GitHub repository URL: https://github.com/soos3d/chatgpt-plugin-development-quickstart-express Loading chatgpt-plugin-development-quickstart-express repository by soos3d Documents uploaded: {'file_path': 'README.md', 'file_name': 'README.md'} {'file_path': 'index.js', 'file_name': 'index.js'} {'file_path': 'src/app.js', 'file_name': 'app.js'} Uploading to vector store... Your Deep Lake dataset has been successfully created! Dataset(path='hub://YOUR_ORG/repository_db', tensors=['embedding', 'id', 'metadata', 'text']) tensor htype shape dtype compression ------- ------- ------- ------- ------- embedding embedding (5, 1536) float32 None id text (5, 1) str None metadata json (5, 1) str None text text (5, 1) str None Test question: What is the repository about? ================================================== Answer: The repository is a ChatGPT Plugin Quickstart with Express.js. It provides a foundation for developing custom ChatGPT plugins using JavaScript and Express.js. The sample plugin in the repository showcases how ChatGPT can integrate with external APIs, specifically API-Ninja's API, to enhance its capabilities. The plugin fetches airport data based on a city name provided by the user. Please enter your question (or type 'exit' to quit): how does the server work? Your question: how does the server work? ================================================== Answer: The server in this context works by setting up an Express.js server with various endpoints to serve a ChatGPT plugin. It initializes the server, configures it to parse JSON in the body of incoming requests, and sets up routes for serving the plugin manifest, OpenAPI schema, logo image, and handling API requests. It also defines a catch-all route to handle any other requests. Finally, the server starts and listens for requests on the specified port. Please enter your question (or type 'exit' to quit): exit Exiting, thanks for chatting! ```


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Diving Deeper: LlamaIndex's Low-Level API for customizations
While the high-level API of LlamaIndex offers a seamless experience for most use cases, there might be situations where you need more granular control over the query logic. This is where the low-level API shines, offering customizations to fine-tune your interactions with the indexed data. ### Building the Index To start, you'll need to build an index from your documents, the same as we have done so far. ```python # Create an index of the documents try: vector_store = DeepLakeVectorStore( dataset_path=dataset_path, overwrite=True, runtime={"tensor_db": True}, ) except Exception as e: print(f"An unexpected error occurred while creating or fetching the vector store: {str(e)}") storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents(docs, storage_context=storage_context) ``` ### Configuring the Retriever The retriever is responsible for fetching relevant nodes from the index. LlamaIndex supports various retrieval modes, allowing you to choose the one that best fits your needs: ```python from llama_index.retrievers import VectorIndexRetriever retriever = VectorIndexRetriever(index=index, similarity_top_k=4) ``` Let's break this down: In modern retrieval systems, documents and queries are represented as vectors, often generated by machine learning models. When a query is made, its vector is compared to document vectors in the index using metrics like cosine similarity. The documents are then ranked based on their similarity scores to the query. **Top-k Retrieval**: Instead of returning all the documents sorted by their similarity, often only the top `k` most similar documents are of interest. This is where `similarity_top_k` comes into play. If `similarity_top_k=4`, the system will retrieve the top 4 most similar documents to the given query. In the context of the code: ```python retriever = VectorIndexRetriever(index=index, similarity_top_k=4) ``` The `VectorIndexRetriever` is configured to retrieve the top 4 documents most similar to any given query. Benefits of using `similarity_top_k`: - **Efficiency**: Retrieving only the top-k results can be faster and more memory-efficient than retrieving all results, especially when dealing with large datasets. - **Relevance**: Users are primarily interested in the most relevant results in many applications. By focusing on the top-k results, the system can provide the most pertinent information without overwhelming the user with less relevant results. However, choosing an appropriate value for `k` is essential. Some relevant results might be missed if `k` is too small. If `k` is too large, the system might return more results than necessary, which could be less efficient and potentially less helpful to the user. While creating this guide, I noticed that using a value above '4' for the parameter led the LLM to produce off-context responses. ### Customize the query engine In LLamaIndex, passing extra parameters and customizations to the query engine is possible. ```python from llama_index import get_response_synthesizer response_synthesizer = get_response_synthesizer() query_engine = RetrieverQueryEngine.from_args( retriever=retriever, response_mode='default', response_synthesizer=response_synthesizer, node_postprocessors=[ SimilarityPostprocessor(similarity_cutoff=0.7)] ) ``` Let's break down this customization: **Getting the Response Synthesizer**: ```python response_synthesizer = get_response_synthesizer() ``` Here, the `get_response_synthesizer` function is called to get an instance of the response synthesizer. The query engine will use this synthesizer to combine and refine the information retrieved by the retriever. **Configuring the Query Engine**: ```python query_engine = RetrieverQueryEngine.from_args( retriever=retriever, response_mode='default', response_synthesizer=response_synthesizer, node_postprocessors=[ SimilarityPostprocessor(similarity_cutoff=0.7)] ) ``` This section configures and initializes the query engine with the following components and settings: - `retriever`: This component fetches relevant nodes (or documents) based on the query. It's passed as an argument, and we set it up in the previous step. - `response_mode='default'`: This sets the mode in which the response will be synthesized. The 'default' mode means the system will "create and refine" an answer by sequentially going through each retrieved node, making a separate LLM call for each node. This mode is suitable for generating more detailed explanations. - `response_synthesizer=response_synthesizer`: The previously obtained response


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Diving Deeper: LlamaIndex's Low-Level API for customizations
synthesizer is passed to the query engine. This component will generate the final response using the specified `response_mode`. - `node_postprocessors`: This is a list of postprocessors that can be applied to the nodes retrieved by the retriever. A `SimilarityPostprocessor` with a `similarity_cutoff` of 0.7 is used in this case. This postprocessor filters out nodes based on their similarity scores, ensuring only nodes with a similarity score above 0.7 are considered. All of those are optional except for the retriever; I recommend testing the various modes and values to find what is more suitable for your use case. ### Exploring Different Response Modes The `RetrieverQueryEngine` offers various response modes to tailor the synthesis of responses based on the retrieved nodes. Here's a breakdown of some of the available modes: 1. **Default Mode**: ```python query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='default') ``` In the default mode, the system processes each retrieved node sequentially, making a separate LLM call for each one. This mode is suitable for generating detailed answers. 2. **Compact Mode**: ```python query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='compact') ``` The compact mode fits as many node text chunks as possible within the maximum prompt size during each LLM call. If there are too many chunks, it refines the answer by processing multiple prompts. 3. **Tree Summarize Mode**: ```python query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='tree_summarize') ``` This mode constructs a tree from a set of node objects, and the query then returns the root node as the response. It's beneficial for summarization tasks. 4. **No Text Mode**: ```python query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='no_text') ``` In the no-text mode, the retriever fetches the nodes that would have been sent to the LLM but doesn't send them. This mode allows for inspecting the retrieved nodes without generating a synthesized response.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## How does LLamaIndex compare to LangChain?
Now that you understand how LLamaIndex works comparing it with the big-name LangChain is an excellent time. LlamaIndex and Langchain are both frameworks/libraries designed to enhance the capabilities of large language models by allowing them to interact with external data. While they serve similar overarching goals, their approaches and features differ. **Key Features**: *Llama Index*: - **Data Connectors**: Offers a variety of connectors to ingest data, making it versatile for different data sources. LLama Hub is an excellent source of community-made tools. - **Data Structuring**: Allows users to structure data using various index types, such as list index, tree index, and even the ability to compose indices. - **Query Abstraction**: Provides layers of abstraction, enabling both simple and complex data querying. *Langchain*: - **Modular Design**: Comes with modules for tools, agents, and chains, each serving a distinct purpose. - **Chains**: A series of steps or actions the language model takes, ideal for tasks requiring multiple interactions. - **Agents**: Autonomous entities that can decide the next steps in a process, adding a layer of decision-making to the model's interactions. - **Tools**: Utility agents are used to perform specific tasks, such as searching or querying an index. **Use Cases**: - *The Llama Index* is best suited for applications that require complex data structures and querying. Its strength lies in handling and querying structured data. - *Langchain*, on the other hand, excels in scenarios that require multiple interactions with a language model, especially when those interactions involve decision-making or a series of steps. Of course, you can combine the two; the potential of combining LlamaIndex and Langchain is promising. By integrating LlamaIndex's structured data querying capabilities with the multi-step interaction and decision-making features of Langchain, developers can create robust and versatile applications. As mentioned in the conclusion, This combination can offer the best of both worlds. So which one to use? Use the tool that makes it easier to take care of your use cases; this is usually the leading factor in deciding which one I want to use.


# Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories 🦙🌊
## Conclusion
In this comprehensive guide, we've journeyed through the intricacies of integrating LlamaIndex with Activeloop Deep Lake to create a conversational interface for GitHub repositories. We've seen how these powerful tools can transform a static codebase into an interactive, queryable entity. The synergy between LlamaIndex's data structuring and Deep Lake's optimized storage offers a robust solution for managing and understanding your code repositories. The code we've explored indexes GitHub repositories and makes them accessible through natural language queries. This opens up a plethora of possibilities. Imagine a future where you don't just browse through code; you have conversations with it. This gave you the basics of using LLamaIndex and Deep Lake; try to improve and customize this app to practice.


Iterative Optimization of LlamaIndex RAG Pipeline

# Introduction
In previous lessons, we learned about advanced techniques and evaluation metrics for LlamaIndex Retrieval-Augmented Generation (RAG) pipelines. Building on this knowledge, we now focus on optimizing a LlamaIndex RAG pipeline through a series of iterative evaluations. We aim to enhance the system's ability to retrieve and generate accurate and relevant information. Here's our step-by-step plan: 1. **Baseline Evaluation**: Construct a standard LlamaIndex RAG pipeline and establish an initial performance baseline. 1. Adjusting TOP_K Retrieval Values: Experiment with different values of k (1, 3, 5, 7) to understand their effect on the accuracy of retrieved information and the relevance of generated answers. 2. **Testing Different Embedding Models**: Evaluate models such as "text-embedding-ada-002" and "cohere/embed-english-v3.0" to identify the most effective one for our pipeline. 3. **Incorporating a Reranker**: Implement a reranking mechanism to refine the document selection process of the retriever. 4. **Employing a Deep Memory Approach**: Investigate the impact of a deep memory component on the accuracy of information retrieval. Through these steps, we aim to refine our RAG system systematically, enhancing its performance by providing accurate and relevant information. The code for this lesson is also available through a [Colab notebook](https://colab.research.google.com/drive/1BZGMXoQVEt7__6dNfRyBAHXwr4kQw4PZ#scrollTo=I93VcS630cbQ), where you can follow along.


# 1. Baseline evaluation
The first step is installing the required Python packages. ```python !pip3 install deeplake llama_index langchain openai tiktoken cohere pandas torch sentence-transformers ``` Here, you can set our API keys. You can skip this step if you plan to use other services. ```python import os os.environ['OPENAI_API_KEY'] = '<YOUR_OPENAI_API_KEY>' os.environ['ACTIVELOOP_TOKEN'] = '<YOUR_ACTIVELOOP_KEY>' os.environ['COHERE_API_KEY'] = '<COHERE_API_KEY>' ``` We download the data, which is a [single text file](https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt). You can use this or replace it with your own data. ```python !mkdir -p 'data/paul_graham/' !curl 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -o 'data/paul_graham/paul_graham_essay.txt' ``` Let’s load the Data and build LlamaIndex nodes/chunks. ```python from llama_index.node_parser import SimpleNodeParser from llama_index import SimpleDirectoryReader # First we create Document LlamaIndex objects from the text data documents = SimpleDirectoryReader("./data/paul_graham/").load_data() node_parser = SimpleNodeParser.from_defaults(chunk_size=512) nodes = node_parser.get_nodes_from_documents(documents) # By default, the node/chunks ids are set to random uuids. To ensure same id's per run, we manually set them. for idx, node in enumerate(nodes): node.id_ = f"node_{idx}" print(f"Number of Documents: {len(documents)}") print(f"Number of nodes: {len(nodes)} with the current chunk size of {node_parser.chunk_size}") ``` ``` Number of Documents: 1 Number of nodes: 58 with the current chunk size of 512 ``` The next step is to create a LlamaIndex `VectorStoreIndex` object and use a `DeepLakeVectorStore` to store the vector embeddings. We also choose `gpt-3.5-turbo-1106` as our LLM and OpenAI’s embedding model `text-embedding-ada-002` ```python from llama_index import VectorStoreIndex, ServiceContext, StorageContext from llama_index.vector_stores import DeepLakeVectorStore from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.llms import OpenAI # Create a local Deep Lake VectorStore dataset_path = "./data/paul_graham/deep_lake_db" vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=True, exec_option="compute_engine") # LLM that will answer questions with the retrieved context llm = OpenAI(model="gpt-3.5-turbo-1106") # We use OpenAI's embedding model "text-embedding-ada-002" embed_model = OpenAIEmbedding() service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm,) storage_context = StorageContext.from_defaults(vector_store=vector_store) vector_index = VectorStoreIndex(nodes, service_context=service_context, storage_context=storage_context, show_progress=True) ``` ``` Generating embeddings: 100% 58/58 [00:06<00:00, 8.75it/s] Uploading data to deeplake dataset. 100%|██████████| 58/58 [00:00<00:00, 169.79it/s]Dataset(path='./data/paul_graham/deep_lake_db', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (58, 1) str None metadata json (58, 1) str None embedding embedding (58, 1536) float32 None id text (58, 1) str None ``` With the vector index, we can now build a `QueryEngine`, which generates answers with the LLM and the retrieved chunks of text. ```python query_engine = vector_index.as_query_engine(similarity_top_k=10) response_vector = query_engine.query("What are the main things Paul worked on before college?") print(response_vector.response) ``` ``` Before college, Paul worked on writing and programming. ``` Now that we have a simple RAG pipeline, we can evaluate it. For that, we need a dataset. Since we don’t have one, we will generate one. `LlamaIndex` offers a `generate_question_context_pairs` module specifically for generating questions and context pairs. We will use that dataset to assess the RAG chunk retrieval and response capabilities. Let’s also save the generated dataset in JSON format for later use. In this case we only generate **58 question and context pairs**, but you can increase the number of samples in the dataset for a more thorough evaluation. ```python from llama_index.evaluation import generate_question_context_pairs qc_dataset = generate_question_context_pairs( nodes, llm=llm, num_questions_per_chunk=1 ) # We can save the dataset as a json file for later use. qc_dataset.save_json("qc_dataset.json") ``` ``` 100%|██████████| 58/58 [01:30<00:00, 1.56s/it] ``` You can load the dataset from your local disk if you have already generated it. ```python from llama_index.finetuning.embeddings.common import ( EmbeddingQAFinetuneDataset, ) qc_dataset = EmbeddingQAFinetuneDataset.from_json( "qc_dataset.json" ) ``` <aside> 💡 We now have a synthetic dataset, but here, you must take the time to review it or even consider building it manually. Doing so will increase the accuracy of your RAG pipeline evaluations. </aside> <aside> ⚙ If you want more control over the quality of the generated dataset, you can look into modifying the prompt.


# 1. Baseline evaluation
This is the current default for the `generate_question_context_pairs` function. </aside> ```python DEFAULT_QA_GENERATE_PROMPT_TMPL = """\ Context information is below. --------------------- {context_str} --------------------- Given the context information and not prior knowledge. generate only questions based on the below query. You are a Teacher/ Professor. Your task is to setup \ {num_questions_per_chunk} questions for an upcoming \ quiz/examination. The questions should be diverse in nature \ across the document. Restrict the questions to the \ context information provided." """ ``` With the generated dataset, we can first start with the retrieval evaluations. We will use the `RetrieverEvaluator` class available in LlamaIndex to measure the Hit Rate and Mean Reciprocal Rank (MRR). **Hit Rate:** Think of the Hit Rate as playing a game of guessing. You're given a question and need to guess the correct answer from a list of options. The Hit Rate measures how often you guess the correct answer by only looking at your top few guesses. You have a high Hit Rate if you often find the right answer in your first few guesses. So, in a retrieval system, it's about how frequently the system finds the correct document within its top 'k' picks (where 'k' is a number you decide, like top 5 or top 10). **Mean Reciprocal Rank (MRR):** MRR is like measuring how quickly you can find a treasure in a list of boxes. Imagine you have a row of boxes, and only one has a treasure. The MRR calculates how close to the start of the row the treasure box is, on average. If the treasure is always in the first box you open, you're doing great and have an MRR of 1. If it's in the second box, the score is 1/2, since you took two tries to find it. If it's in the third box, your score is 1/3, and so on. MRR averages these scores across all your searches. So, for a retrieval system, MRR looks at where the correct document ranks in the system's guesses. If it's usually near the top, the MRR will be high, indicating good performance. In summary, **Hit Rate tells you how often the system gets it right in its top guesses**, and **MRR tells you how close to the top the right answer usually is**. Both metrics are useful for evaluating the effectiveness of a retrieval system, like how well a search engine or a recommendation system works. First, we define a function to display the Retrieval evaluation results in table format. ```python import pandas as pd def display_results_retriever(name, eval_results): """Display results from evaluate.""" metric_dicts = [] for eval_result in eval_results: metric_dict = eval_result.metric_vals_dict metric_dicts.append(metric_dict) full_df = pd.DataFrame(metric_dicts) hit_rate = full_df["hit_rate"].mean() mrr = full_df["mrr"].mean() metric_df = pd.DataFrame( {"Retriever Name": [name], "Hit Rate": [hit_rate], "MRR": [mrr]} ) return metric_df ``` Then, Run the evaluation procedure. ```python from llama_index.evaluation import RetrieverEvaluator # We can evaluate the retievers with different top_k values. for i in [2, 4, 6, 8, 10]: retriever = vector_index.as_retriever(similarity_top_k=i) retriever_evaluator = RetrieverEvaluator.from_metric_names( ["mrr", "hit_rate"], retriever=retriever ) eval_results = await retriever_evaluator.aevaluate_dataset(qc_dataset) print(display_results_retriever(f"Retriever top_{i}", eval_results)) ``` ``` Retriever Name Hit Rate MRR 0 Retriever top_2 0.827586 0.702586 Retriever Name Hit Rate MRR 0 Retriever top_4 0.913793 0.729167 Retriever Name Hit Rate MRR 0 Retriever top_6 0.922414 0.730891 Retriever Name Hit Rate MRR 0 Retriever top_8 0.956897 0.735509 Retriever Name Hit Rate MRR 0 Retriever top_10 0.982759 0.738407 ``` We notice that the Hit Rate increases as the top_k value increases, which is what we can expect. We're increasing the probability of the correct answer being included in the returned set. But how does that impact the quality of


# 1. Baseline evaluation
the generated answers?