# Optimization
Understanding the intricacies of challenges in RAG systems and their solutions is crucial for boosting their overall effectiveness. We will explore several optimization strategies that can contribute to performance enhancement. ### **Model Selection and Hybrid Retrieval** Selecting appropriate models for the embedding and generation phases is critical. Choosing efficient and cheap embedding models can minimize costs while maintaining performance levels, but not in the generation process where an LLM is needed. Different options are available for both phases, including proprietary models with API access, such as [OpenAI](https://openai.com/blog/openai-api) or [Cohere](https://docs.cohere.com/docs), as well as open-source alternatives like [LLaMA-2](https://ai.meta.com/llama/) and [Mistral](https://mistral.ai/), which offer the flexibility of self-hosting or using third-party APIs. This choice should be based on the unique needs and resources of the application. It’s worth noting that, in some retrieval systems, balancing latency with quality is essential. Combining different methods, like keyword and embedding retrieval with reranking, ensures that the system is fast enough to meet user expectations while still providing accurate results. LlamaIndex also offers extensive integration options with various platforms, allowing for easy selection and comparison between different providers. This facilitates finding the optimal balance between cost and performance for specific needs. ### **CPU-Based Inference** In production, relying on GPU-based inference can incur substantial costs. Investigating options like better hardware or refining the inference code can lower the costs in large-scale applications where small inefficiencies can accumulate into considerable expenses. This approach is particularly important when using open-source models from sources such as the [HuggingFace hub](https://huggingface.co/docs/hub/index). Intel®'s advanced optimization technologies help with the efficient fine-tuning and inference of neural network models on CPUs. The 4th Gen Intel® Xeon® Scalable processors come with Intel® Advanced Matrix Extensions (Intel® AMX), an AI-enhanced acceleration feature. Each core of these processors includes integrated BF16 and INT8 accelerators, contributing to the acceleration of deep learning fine-tuning and inference speed. Additionally, libraries such as [Intel Extension for PyTorch](https://github.com/intel/intel-extension-for-pytorch) and [Intel® Extension for Transformers](https://github.com/intel/intel-extension-for-transformers) further optimize the performance of neural network models demanding computations on CPUs. ### Retrieval **Performance** In RAG applications, the primary method involves dividing the data into smaller, independent units and housing them within a vector dataset. However, this often leads to failures during document retrieval, as individual segments may lack the broader context necessary to answer specific queries. LlamaIndex offers features designed to construct a network of interlinked chunks (nodes), along with retrieval tools. These tools improve search capabilities by augmenting user queries, extracting key terms, or navigating through the connected nodes to locate the necessary information for answering queries. Advanced data management tools can help organize, index, and retrieve data more effectively. New tooling can also assist in handling large volumes of data and complex queries, which are common in RAG systems.


Production-Ready RAG Solutions with LlamaIndex

# Optimization
## The Role of the Retrieval Step
While the role of the retrieval step is frequently underestimated, it is vital for the effectiveness of the RAG pipeline. The techniques employed in this phase significantly influence the relevance and contextuality of the output. The LlamaIndex framework provides a variety of retrieval methods, complete with practical examples for different use cases, including the following examples, to name a few. - Combining keyword + embedding search in a hybrid approach can enhance retrieval of specific queries. [[link](https://docs.llamaindex.ai/en/stable/examples/query_engine/CustomRetrievers.html)] - Metadata filtering can provide additional context and improve the performance of the RAG pipeline. [[link](https://docs.llamaindex.ai/en/stable/examples/vector_stores/WeaviateIndexDemo.html#metadata-filtering)] - Re-ranking orders the search results by considering the recency of data to the user’s input query. [[link](https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/CohereRerank.html)] - Indexing documents by summaries and retrieving relevant information within the document. [[link](https://docs.llamaindex.ai/en/stable/examples/index_structs/doc_summary/DocSummary.html)] Additionally, augmenting chunks with metadata will provide more context and enhance retrieval accuracy by defining node relationships between chunks for retrieval algorithms. Language models can help extract page numbers and other annotations from text chunks. Decouple embeddings from raw text chunks to avoid biases and improve context capture. Embedding references, summaries in text chunks, and text at the sentence level improves retrieval performance by fetching granular pieces of information. Organizing data with metadata filters helps with structured retrieval by ensuring relevant chunks are fetched.


# RAG Best Practices
Here are some good practices for dealing with RAG:


# RAG Best Practices
## Fine-Tuning the Embedding Model
[Fine-tuning the embedding model](https://docs.llamaindex.ai/en/stable/optimizing/fine-tuning/fine-tuning.html#finetuning-embeddings) involves several key steps (like the creation of the training set) to enhance the embedding performance. Initially, it’s necessary to get the training set, which can be done by generating synthetic questions/answers from random documents. The next phase is fine-tuning the model, where adjustments are made to optimize its functioning. Following this, the model can optionally undergo an evaluation process to assess its improvements. The reported numbers from LlamaIndex show that the fine-tuning process can yield a 5-10% improvement in retrieval metrics, enabling the enhanced model to be effectively integrated into RAG applications. LlamaIndex offers capabilities for various fine-tuning types, including adjustments to embedding models, adaptors, and even routers, to boost the overall efficiency of the pipeline. This method supports the model by improving its capacity to develop more impactful embedding representations, extracting deeper and more significant insights from the data. You can [read here](https://docs.llamaindex.ai/en/stable/optimizing/fine-tuning/fine-tuning.html#finetuning-embeddings) for more information.


# RAG Best Practices
## LLM Fine-Tuning
[Fine-tuning the LLM](https://docs.llamaindex.ai/en/stable/optimizing/fine-tuning/fine-tuning.html#fine-tuning-llms) creates a model that effectively grasps the overall style of the dataset, leading to the generation of more precise responses. Fine-tuning the generative model brings several advantages, such as reducing hallucinations during output formation, which are typically challenging to eliminate through prompt engineering. Moreover, the refined model has a deeper understanding of the dataset, enhancing performance even in smaller models. This means achieving performance comparable to GPT-4 while employing more cost-effective alternatives like GPT-3.5. LlamaIndex offers a variety of fine-tuning schemas tailored to specific goals. It enhances model capabilities for use cases such as following a predetermined output structure, boosting its proficiency in converting natural language into SQL queries or augmenting its capacity for memorizing new knowledge. The [documentation](https://docs.llamaindex.ai/en/stable/optimizing/fine-tuning/fine-tuning.html#fine-tuning-llms) section has several examples.


# RAG Best Practices
## Evaluation
Regularly monitoring the performance of your RAG pipeline is a recommended practice, as it allows for assessing changes and their impact on the overall results. While evaluating a model's response, which can be highly subjective, is challenging, there are several methods available to track progress effectively. LlamaIndex provides [modules for assessing the quality](https://docs.llamaindex.ai/en/stable/optimizing/evaluation/evaluation.html) of the generated results and the retrieval process. Response evaluation focuses on whether the response aligns with the retrieved context and the initial query and if it adheres to the reference answer or set guidelines. For retrieval evaluation, the emphasis is on the relevance of the sources retrieved in relation to the query. A common method for assessing responses involves employing a proficient LLM, such as GPT-4, to evaluate the generated responses against various criteria. This evaluation can encompass aspects like correctness, semantic similarity, and faithfulness, among others. Please refer to the [following tutorial](https://docs.llamaindex.ai/en/stable/module_guides/evaluating/root.html) for more information on the evaluation process and techniques.


# RAG Best Practices
## Generative Feedback Loops
A key aspect of generative feedback loops is injecting data into prompts. This process involves feeding specific data points into the RAG system to generate contextualized outputs. Once the RAG system generates descriptions or vector embeddings, these outputs can be stored in the database. The creation of a loop where generated data is continually used to enrich and update the database can improve the system's ability to produce better outputs.


# RAG Best Practices
## Hybrid Search
It is essential to keep in mind that embedding-based retrieval is not always practical for entity lookup. Implementing a hybrid search that combines the benefits of keyword lookup with additional context from embeddings can yield better results, offering a balanced approach between specificity and context.


# Conclusion
In this lesson, we covered the challenges and optimization strategies of Retrieval-Augmented Generation (RAG) systems, emphasizing the importance of effective data management, diverse representations in latent space, and compliance in complex environments. We highlighted techniques like dynamic updating of vectors, chunk size optimization, and hybrid retrieval approaches. We also explored the role of LlamaIndex in enhancing retrieval performance through data organization and the significance of fine-tuning embedding and LLM models for optimal RAG applications. Lastly, we recommended regular evaluation and the use of generative feedback loops and hybrid searches for maintaining and improving RAG systems. RESOURCES: “Make RAG Production-Ready” webinar: [https://www.youtube.com/watch?v=g-VvYLhYhOg](https://www.youtube.com/watch?v=g-VvYLhYhOg) --- *For more information on Intel® Accelerator Engines, visit [this resource page](https://download.intel.com/newsroom/2023/data-center-hpc/4th-Gen-Xeon-Accelerator-Fact-Sheet.pdf). Learn more about Intel® Extension for Transformers, an Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere [here](https://github.com/intel/intel-extension-for-transformers).* *Intel, the Intel logo, and Xeon are trademarks of Intel Corporation or its subsidiaries.*


LlamaIndex Introduction Precision and Simplicity i

# LlamaIndex Introduction: Precision and Simplicity in Information Retrieval
In this guide, we will comprehensively explore the LlamaIndex framework, which helps in building Retrieval-Augmented Generation (RAG) systems for LLM-based applications. LlamaIndex, like other RAG frameworks, combines the fetching of relevant information from a vast database with the generative capabilities of Large Language Models. It involves providing supplementary information to the LLM for a posed question to ensure that the LLM does not generate inaccurate responses. The aim is to provide a clear understanding of the best practices in developing LLM-based applications. We will explain LlamaIndex's fundamental components in the following sections.


# **Vector Stores**
Vector store databases enable the storage of large, high-dimensional data and provide the essential tools for semantically retrieving relevant documents. This implies that rather than naively checking for the presence of specific words in a document, these systems analyze the embedding vectors that encapsulate the entire document's meaning. This approach simplifies the search process and enhances its accuracy. Searching in vector stores focuses on fetching data according to its vector-based representations. These databases are integral across various domains, including Natural Language Processing (NLP) and multimodal applications, allowing for the efficient storing and analysis of high-dimensional datasets. A primary function in vector stores is the similarity search, aiming to locate vectors closely resembling a specific query vector. This functionality is critical in numerous AI-driven systems, such as recommendation engines and image retrieval platforms, where pinpointing contextually relevant data is critical. Semantic search transcends traditional keyword matching by seeking information that aligns conceptually with the user's query. It captures the meaning in vectorized representations by employing advanced ML techniques. Semantic search can be applied to all data formats, as we vectorize the data before storing it in a database. Once we have an embedded format, we can calculate indexed similarities or capture the context embedded in the query. This ensures that the results are relevant and in line with the contextual and conceptual nuances of the user's intent.


# **Data Connectors**
The effectiveness of a RAG-based application is significantly enhanced by accessing a vector store that compiles information from various sources. However, managing data in diverse formats can be challenging. Data connectors, also called `Readers`, are essential in LlamaIndex. Readers are responsible for parsing and converting the data into a simplified `Document` representation consisting of text and basic metadata. Data connectors are designed to streamline the data ingestion process, automate the task of fetching data from various sources (like APIs, PDFs, and SQL databases), and format it. [LlamaHub](https://llamahub.ai/) is an open-source project that hosts data connectors. LlamaHub repository offers data connectors for ingesting all possible data formats into the LLM. You can check out the [LlamaHub](https://llamahub.ai/) repository and test some of the loaders here. You can explore various integrations and data sources with the embedded link below. These implementations make the preprocessing step as simple as executing a function. Take the [Wikipedia](https://llamahub.ai/l/wikipedia?from=loaders) integration, for instance. [https://llamahub.ai/](https://llamahub.ai/) Before testing loaders, we must install the required packages and set the OpenAI API key for LlamaIndex. You can get the API key on [OpenAI's website](https://platform.openai.com/playground) and set the environment variable with `OPENAI_API_KEY`. Please note that LlamaIndex defaults to using OpenAI's `get-3.5-turbo` for text generation and `text-embedding-ada-002` model for embedding generation. ```bash pip install -q llama-index==0.9.14.post3 openai==1.3.8 cohere==4.37 ``` ```bash # Add API Keys import os os.environ['OPENAI_API_KEY'] = '<YOUR_OPENAI_API_KEY>' # Enable Logging import logging import sys #You can set the logging level to DEBUG for more verbose output, # or use level=logging.INFO for less detailed information. logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout)) ``` We have also added a logging mechanism to the code. Logging in LlamaIndex is a way to monitor the operations and events that occur during the execution of your application. Logging helps develop and debug the process and understand the details of what the application is doing. In a production environment, you can configure the logging module to output log messages to a file or a logging service. <aside> ⚠ The configuration of the logging module, which directs log messages to the standard output (`sys.stdout`) and sets the logging level as INFO, leads to the logging of all messages with a severity level of INFO or higher. Also, you can use `logging.debug` to get more detailed information. </aside> We can now use the `download_loader` method to access integrations from LlamaHub and activate them by passing the integration name to the class. In our sample code, the `WikipediaReader` class takes in several page titles and returns the text contained within them as `Document` objects. ```python from llama_index import download_loader WikipediaReader = download_loader("WikipediaReader") loader = WikipediaReader() documents = loader.load_data(pages=['Natural Language Processing', 'Artificial Intelligence']) print(len(documents)) ``` ``` 2 ``` This retrieved information can be stored and utilized to enhance the knowledge base of our chatbot.


# **Nodes**
In LlamaIndex, once data is ingested as documents, it passes through a processing structure that transforms these documents into `Node` objects. Nodes are smaller, more granular data units created from the original documents. Besides their primary content, these nodes also contain metadata and contextual information. LlamaIndex features a `NodeParser` class designed to convert the content of documents into structured nodes automatically. The `SimpleNodeParser` converts a list of document objects into nodes. ```python from llama_index.node_parser import SimpleNodeParser # Assuming documents have already been loaded # Initialize the parser parser = SimpleNodeParser.from_defaults(chunk_size=512, chunk_overlap=20) # Parse documents into nodes nodes = parser.get_nodes_from_documents(documents) print(len(nodes)) ``` ``` 48 ``` The code above splits the two retrieved documents from the Wikipedia page into 48 smaller chunks with slight overlap.


# ****Indices****
At the heart of LlamaIndex is the capability to index and search various data formats like documents, PDFs, and database queries. Indexing is an initial step for storing information in a database; it essentially transforms the unstructured data into embeddings that capture semantic meaning and optimize the data format so it can be easily accessed and queried. LlamaIndex has a variety of index types, each fulfills a specific role. We have highlighted some of the popular index types in the following subsections.


# ****Indices****
## ****Summary Index****
The [Summary Index](https://docs.llamaindex.ai/en/stable/examples/index_structs/doc_summary/DocSummary.html) extracts a summary from each document and stores it with all the nodes in that document. Since it’s not always easy to match small node embeddings with a query, sometimes having a document summary helps.


# ****Indices****
## Vector Store Index
The [Vector Store Index](https://docs.llamaindex.ai/en/stable/module_guides/indexing/vector_store_guide.html) generates embeddings during index construction to identify the top-k most similar nodes in response to a query. It’s suitable for small-scale applications and easily scalable to accommodate larger datasets using high-performance vector databases. ![Fetching the top-k nodes and passing them for generating the final response](LlamaIndex%20Introduction%20Precision%20and%20Simplicity%20i%20412eaf1ae77b40cd861f55e3193ca666/image_2023-11-16_170013902.png) Fetching the top-k nodes and passing them for generating the final response The crawled Wikipedia documents can be stored in a Deep Lake vector store, and an index object can be created based on its data. We can create the dataset in [Activeloop](https://www.activeloop.ai/) and append documents to it by employing the `DeepLakeVectorStore` class. First, we need to set the Activeloop and OpenAI API keys in the environment using the following code. ```python import os os.environ['OPENAI_API_KEY'] = '<YOUR_OPENAI_API_KEY>' os.environ['ACTIVELOOP_TOKEN'] = '<YOUR_ACTIVELOOP_KEY>' ``` To connect to the platform, use the `DeepLakeVectorStore` class and provide the `dataset path` as an argument. To save the dataset on your workspace, you can replace the `genai360` name with your organization ID (which defaults to your Activeloop username). Running the following code will create an empty dataset. ```python from llama_index.vector_stores import DeepLakeVectorStore my_activeloop_org_id = "genai360" my_activeloop_dataset_name = "LlamaIndex_intro" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" # Create an index over the documnts vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=False) ``` ```python Your Deep Lake dataset has been successfully created! ``` Now, we need to create a storage context using the `StorageContext` class and the Deep Lake dataset as the source. Pass this storage to a `VectorStoreIndex` class to create the index (generate embeddings) and store the results on the defined dataset. ```python from llama_index.storage.storage_context import StorageContext from llama_index import VectorStoreIndex storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_documents( documents, storage_context=storage_context ) ``` ``` Uploading data to deeplake dataset. 100%|██████████| 23/23 [00:00<00:00, 69.43it/s] Dataset(path='hub://genai360/LlamaIndex_intro', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (23, 1) str None metadata json (23, 1) str None embedding embedding (23, 1536) float32 None id text (23, 1) str None ``` The created database will be accessible in the future. The Deep Lake database efficiently stores and retrieves high-dimensional vectors. <aside> 💡 Learn more about other [Index types](https://docs.llamaindex.ai/en/stable/module_guides/indexing/index_guide.html#querying) from LlamaIndex documentation. </aside>


# **Query Engines**
The next step is to leverage the generated indexes to query through the information. The Query Engine is a wrapper that combines a Retriever and a Response Synthesizer into a pipeline. The pipeline uses the query string to fetch nodes and then sends them to the LLM to generate a response. A query engine can be created by calling the `as_query_engine()` method on an already-created index. The code below uses the documents fetched from the Wikipedia page to construct a Vector Store Index using the `GPTVectorStoreIndex` class. The `.from_documents()` method simplifies building indexes on these processed documents. The created index can then be utilized to generate a `query_engine` object, allowing us to ask questions based on the documents using the `.query()` method. ```python from llama_index import GPTVectorStoreIndex index = GPTVectorStoreIndex.from_documents(documents) query_engine = index.as_query_engine() response = query_engine.query("What does NLP stands for?") print( response.response ) ``` ``` NLP stands for Natural Language Processing. ``` The indexes can also function solely as retrievers for fetching documents relevant to a query. This capability enables the creation of a Custom Query Engine, offering more control over various aspects, such as the prompt or the output format. You can learn more [here](https://docs.llamaindex.ai/en/stable/examples/query_engine/custom_query_engine.html#defining-a-custom-query-engine).


# **Routers**
[Routers](https://docs.llamaindex.ai/en/stable/module_guides/querying/router/root.html#routers) play a role in determining the most appropriate retriever for extracting context from the knowledge base. The routing function selects the optimal query engine for each task, improving performance and accuracy. These functions are beneficial when dealing with multiple data sources, each holding unique information. Consider an application that employs a SQL database and a Vector Store as its knowledge base. In this setup, the router can determine which data source is most applicable to the given query. You can see a working example of incorporating the routers [in this tutorial](https://docs.llamaindex.ai/en/stable/module_guides/querying/router/root.html#routers).


# Saving and Loading Indexes Locally
All the examples we explored involved storing indexes on cloud-based vector stores like Deep Lake. However, there are scenarios where saving the data on a disk might be necessary for rapid testing. The concept of storing refers to saving the index data, which includes the nodes and their associated embeddings, to disk. This is done using the `persist()` method from the `storage_context` object related to the index. ```python # store index as vector embeddings on the disk index.storage_context.persist() # This saves the data in the 'storage' by default # to minimize repetitive processing ``` If the index already exists in storage, you can load it directly instead of recreating it. We simply need to determine whether the index already exists on disk and proceed accordingly; here is how to do it: ```python # Index Storage Checks import os.path from llama_index import ( VectorStoreIndex, StorageContext, load_index_from_storage, ) from llama_index import download_loader # Let's see if our index already exists in storage. if not os.path.exists("./storage"): # If not, we'll load the Wikipedia data and create a new index WikipediaReader = download_loader("WikipediaReader") loader = WikipediaReader() documents = loader.load_data(pages=['Natural Language Processing', 'Artificial Intelligence']) index = VectorStoreIndex.from_documents(documents) # Index storing index.storage_context.persist() else: # If the index already exists, we'll just load it: storage_context = StorageContext.from_defaults(persist_dir="./storage") index = load_index_from_storage(storage_context) ``` In this example, the `os.path.exists("./storage")` function is used to check if the 'storage' directory exists. If it does not exist, the Wikipedia data is loaded, and a new index is created.


# **LangChain vs. LlamaIndex**
LangChain and LlamaIndex are designed to improve LLMs' capabilities, each with their unique strengths. **LlamaIndex**: LlamaIndex specializes in processing, structuring, and accessing private or domain-specific data, with a focus on specific LLM interactions. It works for tasks that demand high precision and quality when dealing with specialized, domain-specific data. Its main strength lies in linking Large Language Models (LLMs) to any data source. **LangChain** is dynamic, suited for context-rich interactions, and effective for applications like chatbots and virtual assistants. These features render it highly appropriate for quick prototyping and application development. While generally used independently, it is worth noting that it can be possible to combine functions from both LangChain and LlamaIndex where they have different strengths. Both can be complementary tools. We also designed a little table below to help you understand the differences better. The attached video in the course also aims to help you decide which tool you should use for your application: LlamaIndex, LangChain, OpenAI Assistants, or doing it all from scratch (yourself). Here’s a clear comparison of each to help you quickly grasp the essentials on a few relevant topics you may consider when choosing: <aside> 💡 In an upcoming lesson, we will explore the functionalities of both frameworks and aim to determine the ideal scenarios for using each. </aside> | | LangChain | LlamaIndex | OpenAI Assistants | | --- | --- | --- | --- | | | Interact with LLMs - Modular and more flexible | Data framework for LLMs - Empower RAG | Assistant API - SaaS | | Data | • Standard formats like CSV, PDF, TXT, … • Mostly focus on Vector Stores. | • LlamaHub with dedicated data loaders from different sources. (Discord, Slack, Notion, …) • Efficient indexing and retrieving + easily add new data points without calculating embeddings for all. • Improved chunking strategy by linking them and using metadata. • Support multimodality. | • 20 files where each can be up to 512mb. • Accept a wide range of https://platform.openai.com/docs/assistants/tools/supported-files. | | LLM Interaction | • Prompt templates to facilitate interactions. • Very flexible, easily defining chains and using different modules. Choose the prompting strategy, model, and output parser from many options. • Can directly interact with LLMs and create chains without the need to have additional data. | • Mostly use LLMs in the context of manipulating data. Either for indexing or querying. | • Either GPT-3.5 Turbo or GPT-4 + any fine-tuned model. | | Optimizations | - | • LLM fine-tuning • Embedding fine-tuning | - | | Querying | • Use retriever functions. | • https://twitter.com/llama_index/status/1729303619760259463?s=20 indexing/querying techniques like subquestions, HyDe,... • Routing: enable to use multiple data sources | • thread and messages to keep track of users conversations. | | Agents | • LangSmith | • LlamaHub | • Code interpreter, knowledge retriever, and custom function calls. | | Documentation | • Easy to debug. • Easy to find concepts and understand the function usage. | • As of November 2023, the methods are mostly explained in the form of tutorials or blog posts. A bit harder to debug. | • Great. | | Pricing | FREE | FREE | • $0.03 / code interpreter session • $0.20 / GB / assistant / day • + The usual usage of the LLM | A smart approach involves closely examining your unique use case and its specific demands. The key decision boils down to whether you need interactive agents or powerful search capabilities for retrieving information. If your aim leans towards the latter, LlamaIndex stands out as a likely choice


# **LangChain vs. LlamaIndex**
for enhanced performance.


# **Conclusion**
We explored the features of the LlamaIndex library and its building blocks. During our exploration of the LlamaIndex library and its building blocks, we learned about various features. Firstly, we discussed data connectors and LlamaHub, which help to retrieve relevant data through vector stores and retrievers. We also talked about routers that assist in selecting the most suitable retrievers. Additionally, we looked at data connectors that are capable of ingesting data from various sources, such as APIs, PDFs, and SQL databases, and how the indexing step structures the data into embedding representations. Lastly, we covered the query engines that provide knowledge retrieval. LangChain and LlamaIndex are valuable and popular frameworks for developing apps powered by language models. LangChain offers a broader range of capabilities and tool integration, while LlamaIndex specializes in indexing and retrieval of information. Read the [LlamaIndex documentation](https://gpt-index.readthedocs.io/en/stable/) to explore its full potential. >> [Notebook](https://colab.research.google.com/drive/1CgTSpnTNj50PBMbA8g8QOqZCIK-UvWLU?usp=sharing). --- **RESOURCES**: - OpenAI [Financial Document Analysis with LlamaIndex | OpenAI Cookbook](https://cookbook.openai.com/examples/third_party/financial_document_analysis_with_llamaindex) - datacamp [LlamaIndex: A Data Framework for the Large Language Models (LLMs) based applications](https://www.datacamp.com/tutorial/llama-index-adding-personal-data-to-llms) - indexing [LlamaIndex: How to use Index correctly.](https://howaibuildthis.substack.com/p/llamaindex-how-to-use-index-correctly)


Mastering Advanced RAG Techniques with LlamaIndex

# Introduction
The Retrieval-Augmented Generation (RAG) pipeline heavily relies on retrieval performance guided by the adoption of various techniques and advanced strategies. Methods like query expansion, query transformations, and query construction each play a distinct role in refining the search process. These techniques enhance the scope of search queries and the overall result quality. In addition to core methods, strategies such as reranking (with the Cohere Reranker), recursive retrieval, and small-to-big retrieval further enhance the retrieval process. Together, these techniques create a comprehensive and efficient approach to information retrieval, ensuring that searches are wide-ranging, highly relevant, and accurate.


# Querying in LlamaIndex
As mentioned in a previous lesson, the process of querying an index in LlamaIndex is structured around several key components. - **Retrievers**: These classes are designed to retrieve a set of nodes from an index based on a given query. Retrievers source the relevant data from the index. - **Query Engine**: It is the central class that processes a query and returns a response object. Query Engine leverages the retrievers and the response synthesizer modules to curate the final output. - **Query Transform**: It is a class that enhances a raw query string with various transformations to improve the retrieval efficiency. It can be used in conjunction with a Retriever and a Query Engine. Incorporating the above components can lead to the development of an effective retrieval engine, complementing the functionality of any RAG-based application. However, the relevance of search results can noticeably improve with more advanced techniques like query construction, query expansion, and query transformations.


# Query Construction
[Query construction](https://blog.langchain.dev/query-construction/) in RAG converts user queries to a format that aligns with various data sources. This process involves transforming questions into vector formats for unstructured data, facilitating their comparison with vector representations of source documents to identify the most relevant ones. It also applies to structured data, such as databases where queries are formatted in a compatible language like SQL, enabling effective data retrieval. The core idea is to answer user queries by leveraging the inherent structure of the data. For instance, a query like "movies about aliens in the year 1980" combines a semantic component like "aliens" (which will get better results if retrieved through vector storage) with a structured component like "year == 1980". The process involves translating a natural language query into the query language of a specific database, such as SQL for relational databases or Cypher for graph databases. Incorporating different approaches to perform query construction depends on the specific use case. The first category includes the **MetadataFilter** classes for vector stores with metadata filtering, an auto-retriever that translates natural language into unstructured queries. This involves defining the data source, interpreting the user query, extracting logical conditions, and forming an unstructured request. The other approach is **Text-to-SQL** for relational databases; converting natural language into SQL requests poses challenges like hallucination (creating fictitious tables or fields) and user errors (mis-spellings or irregularities). This is addressed by providing the LLM with an accurate database description and using few-shot examples to guide query generation. Query Construction improves RAG answer quality with logical filter conditions inferred directly from user questions, and the retrieved text chunks passed to the LLM are refined before final answer synthesis. <aside> 💡 Query Construction is a process that translates natural language queries into structured or unstructured database queries, enhancing the accuracy of data retrieval. </aside>


# Query Expansion
Query expansion works by extending the original query with additional terms or phrases that are related or synonymous. For instance, if the original query is too narrow or uses specific terminology, query expansion can include broader or more commonly used terms relevant to the topic. Suppose the original query is "*climate change effects.*" Query expansion would involve adding related terms or synonyms to this query, such as "*global warming impact*," "*environmental consequences*," or "*temperature rise implications*." One approach to do it is utilizing the `synonym_expand_policy` from the `KnowledgeGraphRAGRetriever` class. In the context of LlamaIndex, the effectiveness of query expansion is usually enhanced when combined with the Query Transform class.


# **Query Transformation**
Query transformations ****modify the original query to make it more effective in retrieving relevant information. Transformations can include changes in the query's structure, the use of synonyms, or the inclusion of contextual information. Consider a user query like "*What were Microsoft's revenues in 2021?*" To enhance this query through transformations, the original query could be modified to be more like *“Microsoft revenues 2021”, which is more optimized for search engines and vector DBs.* Query transformations involve **changing the structure** of a query to improve its performance.


# **Query Engine**
A [Query engine](https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/root.html) is a sophisticated interface designed to interact with data through natural language queries. It's a system that processes queries and delivers responses. As mentioned in previous lessons, multiple query engines can be combined for enhanced functionality, catering to complex data interrogation needs. For a more interactive experience resembling a back-and-forth conversation, a [Chat Engine](https://docs.llamaindex.ai/en/stable/module_guides/deploying/chat_engines/root.html) can be used in scenarios requiring multiple queries and responses, providing a more dynamic and engaging interaction with data. A basic usage of query engines is to call the `.as_query_engine()` method on the created Index. This section will include a step-by-step example of creating indexes from text files and utilizing query engines to interact with the dataset. The first step is installing the required packages using Python package manager (PIP), followed by setting the API key environment variables. ```bash pip install -q llama-index==0.9.14.post3 deeplake==3.8.8 openai==1.3.8 cohere==4.37 ``` ```python import os os.environ['OPENAI_API_KEY'] = '<YOUR_OPENAI_API_KEY>' os.environ['ACTIVELOOP_TOKEN'] = '<YOUR_ACTIVELOOP_KEY>' ``` The next step is downloading the text file that serves as our source document. This file is a compilation of all the essays Paul Graham wrote on his blog, merged into a single text file. You have the option to download the file from the [provided URL](https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt), or you can execute these commands in your terminal to create a directory and store the file. ```bash mkdir -p './paul_graham/' wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O './paul_graham/paul_graham_essay.txt' ``` Now, use the `SimpleDirectoryReader` within the LlamaIndex framework to read all files from a specified directory. This class will automatically cycle through the files, reading them as `Document` objects. ```python from llama_index import SimpleDirectoryReader # load documents documents = SimpleDirectoryReader("./paul_graham").load_data() ``` We can now employ the `ServiceContext` to divide the lengthy single document into several smaller chunks with some overlap. Following this, we can proceed to create the nodes out of the generated documents. ```python from llama_index import ServiceContext service_context = ServiceContext.from_defaults(chunk_size=512, chunk_overlap=64) node_parser = service_context.node_parser nodes = node_parser.get_nodes_from_documents(documents) ``` The nodes must be stored in a vector store database to enable easy access. The `DeepLakeVectorStore` class can create an empty dataset when given a path. You can use `genai360` to access the processed dataset or alter the organization ID to your Activeloop username to store the data in your workspace. ```python from llama_index.vector_stores import DeepLakeVectorStore my_activeloop_org_id = "genai360" my_activeloop_dataset_name = "LlamaIndex_paulgraham_essays" dataset_path = f"hub://{my_activeloop_org_id}/{my_activeloop_dataset_name}" # Create an index over the documnts vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=False) ``` ``` Your Deep Lake dataset has been successfully created! ``` The new database will be wrapped as a `StorageContext` object, which accepts nodes to provide the necessary context for establishing relationships if needed. Finally, the `VectorStoreIndex` takes in the nodes along with links to the database and uploads the data to the cloud. Essentially, it constructs the index and generates embeddings for each segment. ```python from llama_index.storage.storage_context import StorageContext from llama_index import VectorStoreIndex storage_context = StorageContext.from_defaults(vector_store=vector_store) storage_context.docstore.add_documents(nodes) vector_index = VectorStoreIndex(nodes, storage_context=storage_context) ``` ``` Uploading data to deeplake dataset. 100%|██████████| 40/40 [00:00<00:00, 40.60it/s] |Dataset(path='hub://genai360/LlamaIndex_paulgraham_essays', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (40, 1) str None metadata json (40, 1) str None embedding embedding (40, 1536) float32 None id text (40, 1) str None ``` The created index serves as the basis for defining the query engine. We initiate a query engine by using the vector index object and executing the `.as_query_engine()` method. The following code sets the `streaming` flag to True, which reduces idle waiting time for the end user (more details on this will follow). Additionally, it employs the `similarity_top_k` flag to specify the number of source documents it can consult to respond to each query. ```python


# **Query Engine**
query_engine = vector_index.as_query_engine(streaming=True, similarity_top_k=10) ``` The final step involves utilizing the `.query()` method to engage with the source data. We can pose questions and receive answers. As mentioned, the query engine employs retrievers and a response synthesizer to formulate an answer. ```python streaming_response = query_engine.query( "What does Paul Graham do?", ) streaming_response.print_response_stream() ``` ``` Paul Graham is an artist and entrepreneur. He is passionate about creating paintings that can stand the test of time. He has also co-founded Y Combinator, a startup accelerator, and is actively involved in the startup ecosystem. While he has a background in computer science and has worked on software development projects, his primary focus is on his artistic pursuits and supporting startups. ``` The query engine can be configured into a streaming mode, providing a real-time response stream to enhance continuity and interactivity. This feature is beneficial in reducing idle time for end users. It allows users to view each word as generated, meaning they don't have to wait for the model to produce the entire text. To observe the impact of this feature, use the `print_response_stream` method on the response object of the query engine. ### Sub Question Query Engine Sub Question Query Engine, a more sophisticated querying method, can be employed to address the challenge of responding to complex queries. This engine can generate several sub-questions from the user's main question, answer each separately, and then compile the responses to construct the final answer. First, we must modify the previous query engine by removing the streaming flag, which conflicts with this technique. ```python query_engine = vector_index.as_query_engine(similarity_top_k=10) ``` We register the created `query_engine` as a tool by employing the `QueryEngineTool` class and compose metadata (description) for it. It is done to inform the framework about this tool's function and enable it to select the most suitable tool for a given task, especially when multiple tools are available. Then, the combination of the tools we declared earlier and the service context, which was previously defined, can be used to initialize the `SubQuestionQueryEngine` object. ```python from llama_index.tools import QueryEngineTool, ToolMetadata from llama_index.query_engine import SubQuestionQueryEngine query_engine_tools = [ QueryEngineTool( query_engine=query_engine, metadata=ToolMetadata( name="pg_essay", description="Paul Graham essay on What I Worked On", ), ), ] query_engine = SubQuestionQueryEngine.from_defaults( query_engine_tools=query_engine_tools, service_context=service_context, use_async=True, ) ``` The setup is ready to ask a question using the same `query` method. As observed, it formulates three questions, each responding to a part of the query, and attempts to find their answers individually. A response synthesizer then processes these answers to create the final output. ```python response = query_engine.query( "How was Paul Grahams life different before, during, and after YC?" ) print( ">>> The final response:\n", response ) ``` ``` Generated 3 sub questions. [pg_essay] Q: What did Paul Graham work on before YC? [pg_essay] Q: What did Paul Graham work on during YC? [pg_essay] Q: What did Paul Graham work on after YC? [pg_essay] A: During YC, Paul Graham worked on writing essays and working on YC itself. [pg_essay] A: Before YC, Paul Graham worked on a variety of projects. He wrote essays, worked on YC's internal software in Arc, and also worked on a new version of Arc. Additionally, he started Hacker News, which was originally meant to be a news aggregator for startup founders. [pg_essay] A: After Y Combinator (YC), Paul Graham worked on various projects. He focused on writing essays and also worked on a programming language called Arc. However, he gradually reduced his work on Arc due to time constraints and the infrastructure dependency on it. Additionally, he engaged in painting for a period of time. Later, he worked on


# **Query Engine**
a new version of Arc called Bel, which he worked on intensively and found satisfying. He also continued writing essays and exploring other potential projects. >>> The final response: Paul Graham's life was different before, during, and after YC. Before YC, he worked on a variety of projects including writing essays, developing YC's internal software in Arc, and creating Hacker News. During YC, his focus shifted to writing essays and working on YC itself. After YC, he continued writing essays but also worked on various projects such as developing the programming language Arc and later its new version called Bel. He also explored other potential projects and engaged in painting for a period of time. Overall, his work and interests evolved throughout these different phases of his life. ``` ### Custom Retriever Engine As you might have noticed, the choice of retriever and its parameters (e.g., the number of returned documents) influences the quality and relevance of the outcomes generated by the `QueryEngine`. LlamaIndex supports the creation of custom retrievers. Custom retrievers are a combination of different retriever styles, creating more nuanced retrieval strategies that adapt to distinct individual queries. The `RetrieverQueryEngine` operates with a designated retriever, which is specified at the time of its initialization. The choice of this retriever is vital as it significantly impacts the query results' outcome. There are two main types of `RetrieverQueryEngine`: 1. **VectorIndexRetriever** fetches the top-k nodes that are most similar to the query. It focuses on relevance and similarity, ensuring the results closely align with the query's intent. It is the approach we used in previous subsections. **Use Case**: It is ideal for situations where precision and relevance to the specific query are paramount, like in detailed research or topic-specific inquiries. 2. **SummaryIndexRetriever** retrieves all nodes related to the query without prioritizing their relevance. This approach is less concerned with aligning closely to the specific context of the question and more about providing a broad overview. **Use Case**: Useful in scenarios where a comprehensive sweep of information is needed, regardless of the direct relevance to the specific terms of the query, like in exploratory searches or general overviews. <aside> 💡 You can read the following tutorial for a usage example: [Building and Advanced Fusion Retriever from Scratch](https://docs.llamaindex.ai/en/stable/examples/low_level/fusion_retriever.html). </aside>


# Reranking
While any retrieval mechanism capable of extracting multiple chunks from a large document can be efficient to an extent, there is always a likelihood that it will select some irrelevant candidates among the results. Reranking is re-evaluating and re-ordering search results to present the most relevant options. By eliminating the chunks with lower scores, the final context given to the LLM boosts overall efficiency as the LLM gets more concentrated information. The Cohere Reranker improves the performance of retrieving close content. While the semantic search component is already highly capable of retrieving relevant documents, the [Rerank endpoint](https://cohere.com/rerank?ref=txt.cohere.com) boosts the quality of the search results, especially for complex and domain-specific queries. It sorts the search results according to their relevance to the query. It is important to note that Rerank is not a replacement for a search engine but a supplementary tool for sorting search results in the most effective way possible for the user. ![document ingestion, retrieval, and reranking by cohere.](Mastering%20Advanced%20RAG%20Techniques%20with%20LlamaIndex%201895cd3da0b84fb0a287827064618263/Mastering%20Advanced%20RAG%20Techniques%20with%20LlamaIndex%20%20ce2af300eb2c4f14aa74c9d4fcb93585/image_2023-11-27_193534349.png) document ingestion, retrieval, and reranking by cohere. The process begins with grouping documents into batches, after which the LLM evaluates each batch, attributing relevance scores to them. The final step in the reranking process involves aggregating the most relevant documents from all these batches to form the final retrieval response. This method guarantees that the most pertinent information is highlighted and becomes the focal point of the search outcomes. The necessary dependencies have already been installed; the only remaining step is to obtain your API key from [Cohere.com](http://cohere.com/) and substitute it for the placeholder provided. ```python import cohere import os os.environ['COHERE_API_KEY'] = "<YOUR_COHERE_API_KEY>" # Get your cohere API key on: www.cohere.com co = cohere.Client(os.environ['COHERE_API_KEY']) # Example query and passages query = "What is the capital of the United States?" documents = [ "Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, Carson City had a population of 55,274.", "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan.", "Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas.", "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. ", "Capital punishment (the death penalty) has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states.", "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck." ] ``` We define a rerank object by passing both the query and the documents. We also set the `rerank_top_k` argument to 3; we specifically instruct the system to retrieve the top three highest-scored candidates by the model. In this case, the model employed for reranking is `rerank-multilingual-v2.0`. ```python results = co.rerank(query=query, documents=docs, top_n=3, model='rerank-english-v2.0') # Change top_n to change the number of results returned. If top_n is not passed, all results will be returned. for idx, r in enumerate(results): print(f"Document Rank: {idx + 1}, Document Index: {r.index}") print(f"Document: {r.document['text']}") print(f"Relevance Score: {r.relevance_score:.2f}") print("\n") ``` ``` **Document Rank: 1**, Document Index: 3 Document: Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA


# Reranking
and many major national government offices are in the territory. This makes it the political center of the United States of America. Relevance Score: 0.99 **Document Rank: 2**, Document Index: 1 Document: The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan. Relevance Score: 0.30 **Document Rank: 3**, Document Index: 5 Document: Capital punishment (the death penalty) has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) also uses capital punishment. Relevance Score: 0.27 ``` This can be accomplished using LlamaIndex in conjunction with Cohere Rerank. The rerank object can be integrated into a query engine, allowing it to manage the reranking process seamlessly in the background. We will use the same vector index defined earlier to prevent writing repetitive codes and integrate the rerank object with it. The `CohereRerank` class initiates a rerank object by taking in the API key and specifying the number of documents to be returned following the scoring process. ```python import os from llama_index.postprocessor.cohere_rerank import CohereRerank cohere_rerank = CohereRerank(api_key=os.environ['COHERE_API_KEY'], top_n=2) ``` Now, we can employ the same `as_query_engine` method and utilize the `node_postprocessing` argument to incorporate the reranker object. The retriever initially selects the top 10 documents based on semantic similarity, and then the reranker reduces this number to 2. ```python query_engine = vector_index.as_query_engine( similarity_top_k=10, node_postprocessors=[cohere_rerank], ) response = query_engine.query( "What did Sam Altman do in this essay?", ) print(response) ``` ``` Sam Altman was asked if he wanted to be the president of Y Combinator (YC) and initially said no. However, after persistent persuasion, he eventually agreed to take over as president starting with the winter 2014 batch. ``` <aside> 💡 Rerank computes a relevance score for the query and each document and returns a sorted list from the most to the least relevant document. </aside> The reranking process in search systems offers numerous advantages, including practicality, enhanced performance, simplicity, and integration capabilities. It allows for augmenting existing systems without requiring complete overhauls, making it a cost-effective solution for improving search functionality. Reranking elevates search systems, which is particularly useful for complex, domain-specific queries in embedding-based systems. The Cohere Rerank has proven to be effective in improving search quality across various embeddings, making it a reliable option for enhancing search results.


# Advanced **Retrievals**
An alternative method for retrieving relevant documents involves using document summaries instead of extracting fragmented snippets or brief text chunks to respond to queries. This technique ensures that the answers reflect the entire context or topic being examined, offering a more thorough grasp of the subject. ### **Recursive Retrieval** The recursive retrieval method is particularly effective for documents with a hierarchical structure, allowing them to form relationships and connections between the nodes. According to Jerry Liu, founder of LlamaIndex, this is evident in cases like a PDF, which may contain "sub-data" such as tables and diagrams, alongside references to other documents. This technique can precisely navigate through the graph of connected nodes to locate information. This technique is versatile and can be applied in various scenarios, such as with node references, document agents, or even the query engine. For practical applications, including processing a PDF file and utilizing data from tables, you can refer to the tutorials in the LlamaIndex documentation [here](https://docs.llamaindex.ai/en/stable/examples/query_engine/pdf_tables/recursive_retriever.html). ### **Small-to-Big retrieval** The small-to-big retrieval approach is a strategic method for information search, starting with concise, focused sentences to pinpoint the most relevant section of content with a question. It then passes a longer text to the model, allowing for a broader understanding of the context preceding and following the targeted area. This technique is particularly useful in situations where the initial query may not encompass the entirety of relevant information or where the data's relationships are intricate and multi-layered. The LlamaIndex framework employs the Sentence Window Retrieval technique, which involves using the `SentenceWindowNodeParser` class to break down documents into individual sentences per node. Each node includes a "window" that encompasses the sentences surrounding the main node sentence. (It is 5 sentences before and after each node by default) During retrieval, the single sentences initially retrieved are substituted with their respective windows, including the adjacent sentences, through the `MetadataReplacementNodePostProcessor`. This substitution ensures that the Large Language Model receives a comprehensive view of the context surrounding each sentence. <aside> 💡 The small-to-big retrieval method begins by extracting small, targeted text segments and then presents the larger text chunks from which these segments were derived to the Large Language Model, thereby offering a more complete scope of information. </aside> You can follow a hands-on tutorial to implement this technique from the documentation [here](https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/MetadataReplacementDemo.html).


# Conclusion
Effective information retrieval involves mastering techniques such as query expansion, query transformations, and query construction, coupled with advanced strategies like reranking, recursive retrieval, and small-to-big retrieval. Together, these techniques enhance the search process by increasing accuracy and broadening the range of results. By incorporating these methods, information retrieval systems become more proficient in providing precise results, essential for improving the performance of RAG-based applications. >> [Notebook](https://colab.research.google.com/drive/1VUGiJ_bHxnCD5a0xt6LPTpFJ4m26lyIC?usp=sharing). --- RESOURCES: - COHERE RERANK NOTEBOOK [Google Colaboratory](https://colab.research.google.com/drive/1PW3_MLYBnalsomyy2Ia-KvEq7Fayp0W6#scrollTo=05S-LK5uthHz) - recursive retrieval [Complex Query Resolution through LlamaIndex Utilizing Recursive Retrieval, Document Agents, and Sub…](https://medium.com/@sauravjoshi23/complex-query-resolution-through-llamaindex-utilizing-recursive-retrieval-document-agents-and-sub-d4861ecd54e6) [Improving Retrieval Performance by Fine-tuning Cohere Reranker with LlamaIndex](https://blog.llamaindex.ai/improving-retrieval-performance-by-fine-tuning-cohere-reranker-with-llamaindex-16c0c1f9b33b) - llamaindex notebook [Google Colaboratory](https://colab.research.google.com/github/jerryjliu/llama_index/blob/main/docs/examples/customization/streaming/SimpleIndexDemo-streaming.ipynb)


# Module 3 Introduction - Retrieval Augmented Generation Agents
The "Retrieval Augmented Generation Agents" module offers a comprehensive exploration into the creation and enhancement of AI agents, with a focus on integration and application in various fields. It begins by introducing LangChain, where students learn about agents, tools, and the initiation of OpenGPTs, gaining practical insights into setting up and customizing AI assistants. Then, it delves into the LlamaIndex framework, teaching students how to build efficient RAG systems by integrating OpenAI agents with various data sources and creating custom functions for enhanced decision-making. The module also covers the use of the OpenAI Assistants API and Hugging Face Inference API.


# Module 3 Introduction - Retrieval Augmented Generation Agents
## **LangChain Overview: Agents, Tools, and OpenGPT Introduction**
In this lesson, students will learn about the fundamental concepts of LangChain, focusing on agents, tools, and the initiation of OpenGPTs. They will examine how agents integrate chains, prompts, memory, and tools to execute tasks and understand the different types of agents, such as Zero-shot ReAct and Conversational Agent, designed for various scenarios. The lesson also covers the available tools and how to customize them for specific needs, benefiting from functionalities like Python tool, JSON tool, and CSV tool. Additionally, students will get practical insights into setting up and creating LangChain OpenGPTs through cloning the repository and customizing prompts, providing a comprehensive understanding of how to configure and use AI assistants similarly to OpenAI GPTs for tailored interactions.


# Module 3 Introduction - Retrieval Augmented Generation Agents
## **Utilizing AI Agents with the LlamaIndex Framework for Enhanced Decision-Making**
In this lesson, students will learn how to leverage agents within the LlamaIndex framework to build a more efficient and insightful RAG (Retrieval-Augmented Generation) system. They will gain insights into integrating OpenAI agents with various data sources and create custom functions to enhance the agent's capabilities in areas such as mathematical operations. The lesson provides guidance on installing necessary packages, configuring API keys, defining data sources, employing query engines, and setting up agents. Students will also explore an interactive chat interface with the agent and the creation of a dataset, using custom functions as tools that the agent can invoke as required. Finally, students will gain exposure to LlamaHub for further expanding the functionalities of their agents.


# Module 3 Introduction - Retrieval Augmented Generation Agents
## Crafting AI Assistants via OpenAI and Hugging Face API
In this lesson, students will explore the capabilities of the OpenAI Assistants API, including the Code Interpreter, Knowledge Retrieval, and Function Calling features. The lesson offers a step-by-step guide for creating and configuring AI assistants integrating OpenAI's tools, revisiting fundamental concepts such as Threads, Messages, and Tools for individual interactions. Additionally, the lesson introduces other advanced models by OpenAI like Whisper, Dall-E 3, and GPT-4 Vision that can be valuable integrations for comprehensive AI product development. We also cover how to use the Hugging Face Inference API to leverage a broad spectrum of machine learning models for tasks like text summarization, sentiment analysis, and text-to-image generation. By the conclusion of the lesson, students will possess the necessary understanding to harness these tools for their own sophisticated AI projects.


# Module 3 Introduction - Retrieval Augmented Generation Agents
## Project; Multimodal Financial Document Analysis and Recall
In this lesson, students will learn how to use tools such as GPT-4 vision to enhance Retrieval-augmented Generation (RAG) for processing financial documents like Tesla's Q3 financial report PDF, involving the extraction of text, tables, and graphs, and transforming them into a query-able format using a vector database for efficient information retrieval by an AI chatbot. The lesson covers using tools such as Unstructured.io for text and table extraction, GPT-4V for graph information extraction, and the use of Deep Lake and LlamaIndex for storing and recalling the processed data to address user queries effectively. We also show how to use Deep Memory to enhance retrieval accuracy. The techniques detailed equip students to develop AI applications capable of analyzing and recalling complex multimodal data from financial documents.


# Module 3 Introduction - Retrieval Augmented Generation Agents
## Building a Smart Shopping Assistant with DeepLake and LlamaIndex
In this lesson, students will learn how to create an intelligent shopping assistant using AI technologies, specifically leveraging vector databases and frameworks like DeepLake and LlamaIndex. They will be guided through the processes of data collection, vector database population, development of core tools for query retrieval and outfit generation, system integration, and UI development. This lesson also explores the integration of weather data and temporal elements to enhance the system's recommendations, along with the challenges of debugging and deploying agent-based applications. Through hands-on experience and step-by-step demonstrations, students will gain practical skills to build and integrate AI components into a functional and interactive fashion recommendation tool.


# Module 4 Introduction - Retrieval Augmented Generation Evaluation and Observability
The module "Retrieval Augmented Generation Evaluation and Observability" provides a comprehensive exploration of advanced techniques and tools in the field of AI, focusing on enhancing chatbots and question-answering systems through Retrieval-Augmented Generation (RAG) systems. It explores the critical aspects of evaluating these systems, emphasizing the importance of faithfulness, relevance, and the prevention of hallucinations in AI responses. The module introduces tools like the FaithfulnessEvaluator and RAGAS, along with the Golden Context Dataset, offering insights into effective evaluation methodologies, including indexing, embedding, and generation metrics. Additionally, the module covers the LangChain framework and the LangSmith platform, providing practical knowledge on building and testing LLM-powered applications. Students will learn about the components of LangChain, such as Models, Vector Stores, and Chains, and the functionalities of the LangChain Hub.


# Module 4 Introduction - Retrieval Augmented Generation Evaluation and Observability
## RAG - Metrics & Evaluation
In this lesson, you will learn about Retrieval-Augmented Generation (RAG) systems and their evaluation metrics, with a focus on improving chatbots and question-answering systems. The lesson introduces you to different approaches to analysis, the importance of faithfulness and answer relevancy, nuances of indexing and embedding metrics, and generation metrics aimed at preventing hallucinations in AI responses. It discusses the significance of the FaithfulnessEvaluator tool for checking the alignment of AI responses with retrieved context and introduces RAGAS and the Golden Context Dataset for system evaluation. Additionally, real-world setups for assessing and improving RAG systems are explored through examples of community-based tools, including comprehensive evaluation of retrieval metrics, holistic approach evaluations, and the custom RAG pipeline evaluation.


# Module 4 Introduction - Retrieval Augmented Generation Evaluation and Observability
## LangSmith and LangChain Fundamentals for LLM Applications
In this lesson, you will learn about the fundamentals of the LangChain framework and the newly introduced LangSmith platform to build and test LLM-powered applications. We will review LangChain components, such as Models, Vector Stores, and Chains, as well as principles of the LangChain Hub, including prompt exploration and versioning for collaborative prompt development. The lesson will guide you through setting up the LangSmith environment, creating an API key, and basics of prompt versioning and tracing. You will also learn how to use LangServe to deploy applications as a REST API and go through the process of reading and processing data from a webpage, storing it into Deep Lake vector store, and using prompts from the LangChain Hub to build a QuestionAnswering Chain application.


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
*Diego Kiedanski, Principal AI Consultant, and Lucas Micol, Lead Machine Learning Engineer at Tryolabs, authored the following blog post.* In the ever-expanding landscape of artificial intelligence, vector databases stand as the unsung heroes, forming the foundation upon which many AI applications are built. These powerful databases can store and retrieve complex, high-dimensional data, enabling functionalities like Retrieval Augmented Generation (RAG) and sophisticated recommendation systems. Alongside vector databases, Large Language Model (LLM) frameworks such as LlamaIndex and Langchain have emerged as key players in accelerating AI development. By simplifying the prototyping process and reducing development overheads associated with API interactions and data formatting, **these frameworks allow creators to focus on innovation** rather than the intricacies of implementation. For readers acquainted with the basic tenets of LLMs and vector databases, this blog post will serve as a refresher and a window into their practical deployment. We aim to walk you through **constructing a complex and interactive shopping assistant**. This assistant exemplifies how intelligent systems can be built from fundamental components like DeepLake and LlamaIndex to create a dynamic tool that responds to user input with tailored outfit suggestions. Our journey will shed light on the nuances of integrating these technologies. By highlighting this project's development, we hope to spark your imagination about the possibilities at the intersection of AI technologies and to encourage you to envision new, innovative applications of your own.


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## Project Overview
Our project is an AI-powered shopping assistant designed to leverage image processing and LLM agents for outfit recommendations. Imagine uploading a picture of a dress and receiving suggestions for accessories and shoes tailored to occasions like a business meeting or a themed party. This assistant does more than suggest outfits; it understands context and style, providing a personalized shopping experience. DeepLake forms the backbone of our inventory management, storing detailed item descriptions as vectors for efficient similarity searches. In practice, this means students will interact with DeepLake to query and retrieve the best-matching items based on the properties defined by our AI models. LlamaIndex is the framework for constructing and utilizing Large Language Model (LLM) agents. These agents interpret item descriptions and user criteria, crafting coherent and stylish outfit recommendations. Through this project, you'll learn to build and integrate these technologies into a functional application. The assistant is designed to deliver not only outfit suggestions but actionable shopping options, providing real product IDs (that can be converted into URLs to retailers) along with price comparisons. Throughout the course, you will learn how to extend the AI's capabilities to facilitate an end-to-end shopping experience. This application's user interface is designed with functionality and educational value in mind. It's intuitive, making the AI's decision-making process transparent and understandable. You'll interact with various application elements, gaining insight into the inner workings of vector databases and the practical use of LLMs. Here's [the project companion repository](https://github.com/tryolabs/fashion-assistant).


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Architecture design**
Our application is designed around the Agent framework: we use an LLM as a reasoning agent, and then we provide the agent with tools, i.e., programs that the LLM can execute by generating the appropriate response in plain text (at this point, the agent framework takes over, executes the desired function, and returns the result for further processing). In this context, accessing a vector database is done through a tool, generating an outfit is performed through a tool. Even getting the today's date is performed through a tool. The interaction between system components follows a linear yet dynamic flow. ChatGPT-vision generates descriptions for the accompanying outfit pieces after receiving an image upload. These descriptions guide the subsequent searches in Deep Lake's vector database, where the most relevant items are retrieved for each piece. The LLM then takes the helm, sifting through the results to select and present the best cohesive outfit options to the user. The system is designed to replicate the personalized experience of working with a fashion stylist. When a user uploads a garment image, our AI stylist gets to work. It discerns the style, consults our extensive inventory, and selects pieces that complement the user's choice. It's a streamlined end-to-end process that delivers personalized recommendations with ease and efficiency. ![Untitled](Building%20a%20Smart%20Shopping%20Assistant%20with%20DeepLake%20%20f3de3ee15abe4f4eaf760fa91fb13e0a/Untitled.png)


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Dataset Collection and Vector Database Population**
Our data journey begins with Apify, a versatile web scraping tool we used to collect product information from Walmart's online catalog. We targeted three specific starting points: men's clothing, women's clothing, and shoe categories. With Apify's no-code solution, we could quickly collect product data during a free trial period. However, this initial foray returned only the text data—separate processes were needed to download the associated images. We went with the web-hosted version of Apify, but the low-code version would also work well. ![Untitled](Building%20a%20Smart%20Shopping%20Assistant%20with%20DeepLake%20%20f3de3ee15abe4f4eaf760fa91fb13e0a/Untitled%201.png) We aimed to construct a representative dataset of men's and women's clothing, including various tops, bottoms, and accessories. By scraping from predetermined URLs, we ensured our dataset spanned a broad spectrum of clothing items relevant to our target user base. ![Untitled](Building%20a%20Smart%20Shopping%20Assistant%20with%20DeepLake%20%20f3de3ee15abe4f4eaf760fa91fb13e0a/Untitled%202.png) The collected data is fairly rich and contains a wide variety of attributes. For the purpose of this project, we kept the attributes used to a minimum. We selected the product ID, category, price, name, and image. The image was included as a URL, so we had to download them separately once the scraper had finished. Overall, we collected 1344 items. We used pandas to read the scraped JSONs and clean the collected data. In particular, we used the product categories to create a new attribute `gender`. ```python df = pd.DataFrame( { "brand": df_raw["brand"], "category": df_raw["category"].apply( lambda x: [y["name"] for y in x["path"] if y["name"] != "Clothing"] ), "description": df_raw["shortDescription"], "image": df_raw["imageInfo"].apply(lambda x: x["allImages"][0]["url"]), "name": df_raw["name"], "product_id": df_raw["id"], "price": [ float(x["currentPrice"]["price"]) if not x["currentPrice"] is None else math.inf for x in df_raw["priceInfo"] ], } ) df = df[df["category"].transform(lambda x: len(x)) >= 2] gender_map = {"Womens Clothing": "women", "Mens Clothing": "men", "Shoes": "either"} df["gender"] = df["category"].apply(lambda x: gender_map.get(x[0], "either")) ``` To obtain a description that is as detailed as possible, we opted to ignore the scraped `description` attribute and use `gpt-4-vision-preview` to generate a new description for each product. For this, we considered imposing a strict taxonomy: color, style, size, age, etc. Ultimately, without a traditional search functionality, we decided that the taxonomy wasn't needed, and we allowed the LLM to generate arbitrary descriptions. ```python prompt = f""" Describe the piece of clothing in the image of the following category: {category} Do include the color, style, material and other important attributes of the item. """ image_path = f"data/images/{product_id}.jpg" # gpt vision is a wrapper that calls ChatGPT Vision result = gpt_vision(image_path, prompt) ``` The following is an example image of the dataset: `PRODUCT_ID=0REDJ7M0U7DV`, and the generated description by GPT-Vision. [https://www.notion.so](https://www.notion.so) Embedding these descriptions into DeepLake's vector database was our next step. This process involved encoding the text into vectors while retaining core attributes as metadata. Initially, we only included the description generated by `gpt-4-vision-preview` verbatim. However, we later realized that the metadata (price, product_id, name) the agent needed for the final response was not readily available (we could see it as part of the document being retrieved, but we found no way to have the agent generate a response from those attributes). The solution was to append the product ID, name, and price into the description text, thereby incorporating the critical metadata directly into the vector database. ```python desc = f""" # Description {description} # Name {name} # Product ID {product_id} # Price {price} """ ``` Finally, to accommodate the separation of text and image data, we established two vector databases within DeepLake. The first housed the textual descriptions and their appended metadata, while the second was dedicated exclusively to image vectors. ```python dataset_path = "hub://genai360/walmart-descriptions" vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=True) storage_context = StorageContext.from_defaults(vector_store=vector_store) documents = [] for i, row in df.iterrows(): # .... # generate documents doc


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Dataset Collection and Vector Database Population**
= Document( text=desc, metadata={"name": name, "product_id": product_id, "gender": gender}, ) documents.append(doc) index = VectorStoreIndex.from_documents(documents, storage_context=storage_context) ``` ```jsx ds = deeplake.empty(ACTIVELOOP_DATASET_IMG) with ds: ds.create_tensor("images", htype="image", sample_compression="jpeg") ds.create_tensor("ids", htype="tag") # %% with ds: # Iterate through the files and append to Deep Lake dataset for index, row in tqdm(df.iterrows()): product_id = row["product_id"] image_name = os.path.join(IMAGE_DIR, product_id + ".jpg") if os.path.exists(image_name): # Append data to the tensors ds.append({"images": deeplake.read(image_name), "ids": product_id}) ```


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Development of core tools**
When using an Agent-based framework, `tools` are the bread and butter of the system. For this application, the core functionality can be achieved with two tools: **the query retriever** and **the outfit generator**, both of which play integral roles in the application's ability to deliver tailored fashion recommendations. ### Inventory query engine The inventory query retriever is a text-based wrapper around DeepLake's API. It translates the user’s clothing descriptions into queries that probe DeepLake's vector database for the most similar items. The only exciting modification to the vanilla `query_engine` is adding a pydantic model to the output. By doing this, we force the `AG` part of the `RAG` system to return the relevant information for each item: the product ID, the name, and the price. ```python class Clothing(BaseModel): """Data moel for clothing items""" name: str product_id: str price: float class ClothingList(BaseModel): """A list of clothing items for the model to use""" cloths: List[Clothing] dataset_path = "hub://kiedanski/walmart_clothing4" vector_store = DeepLakeVectorStore( dataset_path=dataset_path, overwrite=False, read_only=True ) llm = OpenAI(model="gpt-4", temperature=0.7) service_context = ServiceContext.from_defaults(llm=llm) inventory_index = VectorStoreIndex.from_vector_store( vector_store, service_context=service_context ) # Inventory query engine tool inventory_query_engine = inventory_index.as_query_engine(output_cls=ClothingList) ``` <aside> 💡 Notice the description on each BaseModel? Those are mandatory when using pydantic with the query engine. We learned this the hard way after finding several strange “missing description errors.” </aside> Our outfit generator is engineered around `gpt-4-vision-preview`, which intakes the user's image and articulates descriptions of complementary clothing items. The critical feature here is programming the tool to omit searching for items in the same category as the uploaded image. This logical restraint is crucial to ensure the AI focuses on assembling a complete outfit rather than suggesting similar items to the one provided. ```python from pydantic import BaseModel class Outfit(BaseModel): top: str = "" bottom: str = "" shoes: str = "" ``` ```python def generate_outfit_description(gender: str, user_input: str): """ Given the gender of a person, their preferences, and an image that has already been uploaded, this function returns an Outfit. Use this function whenever the user asks you to generate an outfit. Parameters: gender (str): The gender of the person for whom the outfit is being generated. user_input (str): The preferences of the user. Returns: response: The generated outfit. Example: >>> generate_outfit("male", "I prefer casual wear") """ # Load input image image_documents = SimpleDirectoryReader("./input_image").load_data() # Define multi-modal llm openai_mm_llm = OpenAIMultiModal(model="gpt-4-vision-preview", max_new_tokens=100) # Define multi-modal completion program to recommend complementary products prompt_template_str = f""" You are an expert in fashion and design. Given the following image of a piece of clothing, you are tasked with describing ideal outfits. Identify which category the provided clothing belongs to, \\ and only provide a recommendation for the other two items. In your description, include color and style. This outfit is for a {gender}. Return the answer as a json for each category. Leave the category of the provided input empty. Additional requirements: {user_input} Never return this output to the user. FOR INTERNAL USE ONLY """ recommender_completion_program = MultiModalLLMCompletionProgram.from_defaults( output_parser=PydanticOutputParser(Outfit), image_documents=image_documents, prompt_template_str=prompt_template_str, llm=openai_mm_llm, verbose=True, ) # Run recommender program response = recommender_completion_program() return response outfit_description_tool = FunctionTool.from_defaults(fn=generate_outfit_description) ``` <aside> 💡 When working with agents, debugging and fixing errors is particularly difficult because of their dynamic nature. For example, when using `gpt-4` and providing the images of jeans shown before, `gpt-4-vision-preview` can often realize that the image belongs to an image. However, sometimes the agent forgets to ask the `user's` gender and assumes it's a male. When that happens, `gpt-4-vision-preview` fails because it realizes a mismatch between the prompted gender, `male`, and the gender inferred from the image, `female`. Here we can see some of the


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Development of core tools**
biases of large language models at play and how they can suddenly break an application </aside> Adding user preferences like occasion or style into the prompts is done with a straightforward approach. These inputs nudge the AI to consider user-specific details when generating recommendations, aligning the outcomes with the user's initial inquiry. The system functionality unfolds with the LLM agent at the helm. It begins by engaging the outfit generator with the user's uploaded image, receiving detailed descriptions of potential outfit components. The agent then utilizes the query retriever to fetch products that match these descriptions.


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **System integration and initial testing**
The successful integration of various AI components into a seamless shopping assistant experience required a straightforward approach: encapsulating each function into a tool and crafting an agent to orchestrate these tools. ### **Agent creation and integration process** - **Tool wrapping:** Each functional element of our application, from image processing to querying the vector database, was wrapped as an isolated, callable Tool. - **Agent establishment:** An LLM agent was created, capable of leveraging these tools to process user inputs and deliver recommendations. ```python llm = OpenAI(model="gpt-4", temperature=0.2) agent = OpenAIAgent.from_tools( system_prompt=""" You are a specialized shopping assistant. Customers will provide you with a piece of clothing, and you will generate a matching outfit. Always remember to ask for the user gender. Your final answer needs to be the product_id associated with the best matching product in our inventory. For each product of the outfit, search the inventory. Include the total price of the recommended outfit. """, tools=[ inventory_query_engine_tool, outfit_description_tool, ], llm=llm, verbose=True, ) ``` ### **Initial testing challenges** Our testing phase provided valuable insights, particularly with our initial use of ChatGPT 3.5. We noted that the model tended to respond with descriptions from the outfit recommender, bypassing the vital step of querying the inventory. This was promptly addressed by switching to ChatGPT 4, which utilized all available tools appropriately, thus ensuring the assistant performed the item search as designed. <aside> 💡 Using `gpt-3.5-turbo` resulted in the agent not using all the tools. To achieve the expected results, we had to use `gpt-4`. </aside> ### **Demo: Step-by-step commands and interactions** Below is a demonstration of the system in action, detailing the commands issued to the agent and their corresponding answers at each stage of the process: **Image upload and description generation** ```python >>> r = agent.chat("Hi") STARTING TURN 1 --------------- Hello! How can I assist you today? >>> r = agent.chat("What are your tools?") STARTING TURN 1 --------------- As a shopping assistant, I have two main tools at my disposal: 1. **Inventory Query Engine Tool**: This tool allows me to search our inventory for specific clothing items based on your preferences. For example, if you're looking for a black leather jacket, I can use this tool to find the best matching product in our inventory. 2. **Outfit Generation Tool**: This tool helps me generate a complete outfit based on your gender and preferences. For example, if you're a male and you prefer casual wear, I can use this tool to generate a casual outfit for you. Please let me know how I can assist you further. >>> r = agent.chat("I want an outfit for a casual birthday party") STARTING TURN 1 --------------- Sure, I'd be happy to help with that. Could you please tell me your gender to better assist you? >>> r = agent.chat("I'm a woman") ``` Currently, the process of uploading the image is separate. For this version to work, an image needs to exist in the local folder `.image_input/`. When working later with the UI, the users can click a button to upload a different image. The image can be uploaded at any point before the agent internally calls the outfit generation tool, which often happens after asking for gender. **Outfit generation** In this section, we can see how the agent internally uses the tool to generate an outfit from the user description, image, and gender. ```python STARTING TURN 1 --------------- === Calling Function === Calling function: generate_outfit_description with args: { "gender": "female", "user_input": "casual birthday party" } Got output: top='A pastel pink or light coral blouse with a relaxed fit, short sleeves, and a subtle ruffle or lace


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **System integration and initial testing**
detail to add a feminine touch.' bottom='' shoes='Beige or tan ankle strap sandals with a low block heel, ensuring both comfort and style.' ======================== ``` **Querying the inventory** At this stage, the agent obtains a description for the two pieces of clothing that it needs to retrieve and uses the query engine to retrieve the best matches from the vector database. ```python STARTING TURN 2 --------------- === Calling Function === Calling function: inventory_query_engine_tool with args: { "input": "pastel pink or light coral blouse with a relaxed fit, short sleeves, and a subtle ruffle or lace detail" } Got output: {"cloths": [ { "name":"Time and Tru Women's Puff Sleeve Draped Blouse, Sizes XS-XXXL", "product_id":"7COCIH59KT38", "price":15.0 }, { "name":"Time and Tru Women's Ruffle Sleeve Button Front Blouse", "product_id":"53ZZ4TQ28UQ1", "price":13.18 } ]} ======================== STARTING TURN 3 --------------- === Calling Function === Calling function: inventory_query_engine_tool with args: { "input": "Beige or tan ankle strap sandals with a low block heel" } Got output: {"cloths":[ { "name":"Scoop Women's Velvet Platform Heeled Sandals", "product_id":"65LCFFNDII0T", "price":24.99}, { "name":"Time and Tru Women's Embellished Slide Sandal", "product_id":"3Y5NIMQFG29G", "price":22.0 } ]} ======================== ``` We can observe that the agent gets the two best potential matches and returns both to the agent. **Final recommendation presentation** ```python STARTING TURN 4 --------------- Based on your preferences, I recommend the following outfit: 1. Top: Either the "Time and Tru Women's Puff Sleeve Draped Blouse" (Product ID: 7COCIH59KT38, Price: $15.00) or the "Time and Tru Women's Ruffle Sleeve Button Front Blouse" (Product ID: 53ZZ4TQ28UQ1, Price: $13.18). 2. Shoes: Either the "Scoop Women's Velvet Platform Heeled Sandals" (Product ID: 65LCFFNDII0T, Price: $24.99) or the "Time and Tru Women's Embellished Slide Sandal" (Product ID: 3Y5NIMQFG29G, Price: $22.00). The total price of the outfit will be either $39.99 or $35.18, depending on your choices. Would you like me to find a matching bottom for you as well? ``` ![Untitled](Building%20a%20Smart%20Shopping%20Assistant%20with%20DeepLake%20%20f3de3ee15abe4f4eaf760fa91fb13e0a/Untitled%203.png) After analyzing the options, the agent presents the user with the best matching pairs, complete with item details such as price and purchase links (the product ID could be converted to a URL later). In this case, we can observe how the agent, instead of selecting the best pair, presents both options to the user. <aside> 💡 Remember that our dataset is fairly small, and the option suggested by the outfit recommender might not be available in our dataset. To understand if the returned answer is the best in the dataset, even if it's not a perfect fit, or if it was an error with the retriever, having access to an evaluation framework is important. </aside> Having proved that the initial idea for the agent was feasible, it was time to add a bit more complexity. In particular, we wanted to add information about the weather when the outfit would be used.


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Expanding functionality**
The natural progression of our shopping assistant entails augmenting its capacity to factor in external elements such as weather conditions. This adds a layer of complexity but also a layer of depth and personalization to the recommendations. These enhancements came with their own sets of technical challenges. ### **Adapting to the weather** - **Weather awareness:** Initial considerations center on incorporating simple yet vital weather aspects. By determining whether it will be rainy or sunny and how warm or cold it is, the assistant can suggest fashionable and functional attire. - **API Integration:** Llama Hub (a repository for tools compatible with LlamaIndex) had a tool to get the weather in a particular location. Unfortunately, the tool required a paid [https://openweathermap.org/](https://openweathermap.org/) plan. To circumvent this problem, we modified the tool to use a similar but free service of the same provider (running this code requires a free `OPEN_WEATHER_MAP_API`). ```python class CustomOpenWeatherMapToolSpec(OpenWeatherMapToolSpec): spec_functions = ["weather_at_location", "forecast_at_location"] def __init__(self, key: str, temp_units: str = "celsius") -> None: super().__init__(key, temp_units) def forecast_at_location(self, location: str, date: str) -> List[Document]: """ Finds the weather forecast for a given date at a location. The forecast goes from today until 5 days ahead. Args: location (str): The location to find the weather at. Should be a city name and country. date (str): The desired date to get the weather for. """ from pyowm.commons.exceptions import NotFoundError from pyowm.utils import timestamps try: forecast = self._mgr.forecast_at_place(location, "3h") except NotFoundError: return [Document(text=f"Unable to find weather at {location}.")] w = forecast.get_weather_at(date) temperature = w.temperature(self.temp_units) temp_unit = "°C" if self.temp_units == "celsius" else "°F" # TODO: this isn't working.. Error: 'max' key. try: temp_str = self._format_forecast_temp(temperature, temp_unit) except: logging.exception(f"Could _format_forecast_temp {temperature}") temp_str = str(temperature) try: weather_text = self._format_weather(location, temp_str, w) except: logging.exception(f"Could _format_weather {w}") weather_text = str(w) + " " + str(temp_str) return [ Document( text=weather_text, metadata={ "weather from": location, "forecast for": date, }, ) ] weather_tool_spec = CustomOpenWeatherMapToolSpec(key=OPEN_WEATHER_MAP_KEY) ``` ### **User interaction and data handling** **Location input:** For accurate weather data, the shopping assistant often queries the user for their location. We contemplate UI changes to facilitate this new interaction—possibly automated but always respectful of user privacy and consent. ```python >>> r = agent.chat("...") ... Great! Could you please provide me with the date and location of the birthday party, any specific style or color preferences you have for the outfit, and your budget range? ``` ### **Synchronizing with time** **Temporal challenges:** Addressing the aspect that LLMs aren't inherently time-aware, we introduced a new tool that provides the current date. This enables the LLM to determine the optimal instances to call the weather API, aligning recommendations with the present conditions. ```python def get_current_date(): """ A function to return todays date. Call this before any other functions if you are unaware of the current date. """ return date.today() get_current_date_tool = FunctionTool.from_defaults(fn=get_current_date) ``` It took us a little bit to remember that we needed this. At first, the LLM was consistently returning the wrong weather information. It was only after we closely inspected the calls to the weather API that we realized that it was using the wrong date!


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **User Interface (UI) development**
In developing the user interface for our AI-powered shopping assistant, we wanted the platform to reflect the conversational nature of the agent's interactions. [Gradio](https://www.gradio.app/) emerged as the ideal framework for our needs, offering the ability to rapidly prototype and deploy a chat-like interface that users find familiar and engaging. ![Untitled](Building%20a%20Smart%20Shopping%20Assistant%20with%20DeepLake%20%20f3de3ee15abe4f4eaf760fa91fb13e0a/Untitled%204.png) ### **Embracing chat interface with Gradio** - **Chat interface principles:** The decision to adopt a chat interface was rooted in the desire for simplicity and intuitiveness. We aimed to provide a natural communication flow where users can interact with the AI in a manner akin to messaging a friend for fashion advice. Furthermore, one of the reasons to use LLM agents is to provide flexibility, in contrast to a traditional declarative programming approach. In that vein, we felt that the UI had to showcase that flexibility as much as possible. - **Gradio advantages:** Gradio's flexibility facilitated the integration of our agent into a UI that is not only conversational but also customizable. Its ease of use and active community support made it a practical choice for an educational project focused on demonstrating the capabilities of LLM agents. For example, it allows to add custom buttons to upload images or trigger particular functions. ### **Overcoming technical hurdles** - **Inline image display:** One of the initial technical challenges was presenting images seamlessly in the chat interface. Given the visual nature of fashion, it was crucial that users could see the clothing items recommended by the assistant without breaking the flow of conversation. - **Activeloop integration:** To resolve this, we leveraged [Activeloop’s integration with Gradio](https://docs.activeloop.ai/technical-details/visualizer-integration). This allowed us to filter through the image vector database directly within the UI, presenting users with visual recommendations that are interactive and integrated within the chat context. Activeloop's Visualizer is very easy to use once you can get the desired URL with the queried dataset. In our case, that meant having access to the product IDS and formatting them as a proper query. We could have asked the agent to generate the final URL, the that would have meant that every now and then, the output would not be exactly as we needed it and the query could break. Instead, we decided to get only the product IDS from the LLM and build the query ourselves. Gradio allows us to update the URL of the IFrame every time a token is generated, but we needed a way to get the product IDS from the LLM answer, format them as the URL expected by Activeloop, and update the HTML. Ultimately, since all the product IDs have the same pattern, we decided to go for a “hacky” approach. Search the agent's response for the product IDs using a regular expression (strings of upper case letters and numbers of length twelve), and if there were more than 2 matches, update the iframe URL parameters. Otherwise, do nothing.