# 1. Baseline evaluation
## Evaluation for Relevancy and Faithfulness metrics.
**Relevancy** evaluates whether the retrieved context and answer are relevant to the query. **Faithfulness** evaluates if the answer is faithful to the retrieved contexts or, in other words, whether there’s a hallucination. LlamaIndex includes functions that evaluate both metrics using an LLM as the judge. **GPT4 will be used as the judge.** Now, let's see how the top_k value affects these two metrics. ```python from llama_index.evaluation import RelevancyEvaluator, FaithfulnessEvaluator, BatchEvalRunner for i in [2, 4, 6, 8, 10]: # Set Faithfulness and Relevancy evaluators query_engine = vector_index.as_query_engine(similarity_top_k=i) # While we use GPT3.5-Turbo to answer questions # we can use GPT4 to evaluate the answers. llm_gpt4 = OpenAI(temperature=0, model="gpt-4-1106-preview") service_context_gpt4 = ServiceContext.from_defaults(llm=llm_gpt4) faithfulness_evaluator = FaithfulnessEvaluator(service_context=service_context_gpt4) relevancy_evaluator = RelevancyEvaluator(service_context=service_context_gpt4) # Run evaluation queries = list(qc_dataset.queries.values()) batch_eval_queries = queries[:20] runner = BatchEvalRunner( {"faithfulness": faithfulness_evaluator, "relevancy": relevancy_evaluator}, workers=8, ) eval_results = await runner.aevaluate_queries( query_engine, queries=batch_eval_queries ) faithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness']) print(f"top_{i} faithfulness_score: {faithfulness_score}") relevancy_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['relevancy']) print(f"top_{i} relevancy_score: {relevancy_score}") ``` ``` top_2 faithfulness_score: 0.95 top_2 relevancy_score: 0.95 top_4 faithfulness_score: 0.95 top_4 relevancy_score: 0.95 top_6 faithfulness_score: 0.95 top_6 relevancy_score: 0.95 top_8 faithfulness_score: 1.0 top_8 relevancy_score: 1.0 top_10 faithfulness_score: 1.0 top_10 relevancy_score: 1.0 ``` We can notice the relevancy and faithfulness scores increase as the Top_k value increases. We also get a perfect score using eight retrieved chunks as context. <aside> 💡 Remember that these scores come from another LLM, such as GPT4. Since LLM outputs are not deterministic, you can expect different results if you run the evaluations repeatedly. </aside> This is the LlamaIndex Relevancy prompt default template. <aside> 🛠 DEFAULT_EVAL_TEMPLATE = PromptTemplate( "Your task is to evaluate if the response for the query \ is in line with the context information provided.\n" "You have two options to answer. Either YES/ NO.\n" "Answer - YES, if the response for the query \ is in line with context information otherwise NO.\n" "Query and Response: \n {query_str}\n" "Context: \n {context_str}\n" "Answer: " ) </aside>


Iterative Optimization of LlamaIndex RAG Pipeline

# 2. **Changing the embedding model**
Now that we have the baseline evaluation score, we can start changing some modules of our LlamaIndex RAG pipeline. We can start by changing the embedding model. Here, we will be testing the [cohere embedding model](https://docs.llamaindex.ai/en/stable/examples/embeddings/cohereai.html) `embed-english-v3.0` instead of OpenAI’s `text-embedding-ada-002`. ```python import os from llama_index import VectorStoreIndex, ServiceContext, StorageContext from llama_index.vector_stores import DeepLakeVectorStore from llama_index.embeddings.cohereai import CohereEmbedding from llama_index.llms import OpenAI # Create another local DeepLakeVectorStore to store the embeddings dataset_path = "./data/paul_graham/deep_lake_db_1" vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=False, exec_option="compute_engine") llm = OpenAI(model="gpt-3.5-turbo-1106") embed_model = CohereEmbedding( cohere_api_key=os.getenv('COHERE_API_KEY'), model_name="embed-english-v3.0", input_type="search_document", ) service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm,) storage_context = StorageContext.from_defaults(vector_store=vector_store) vector_index = VectorStoreIndex(nodes, service_context=service_context, storage_context=storage_context, show_progress=True) ``` ``` Generating embeddings: 100% 58/58 [00:02<00:00, 23.68it/s] Uploading data to deeplake dataset. 100%|██████████| 58/58 [00:00<00:00, 315.69it/s]Dataset(path='./data/paul_graham/deep_lake_db_1', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (58, 1) str None metadata json (58, 1) str None embedding embedding (58, 1024) float32 None id text (58, 1) str None ``` We run the retrieval evaluation using these new embeddings. ```python from llama_index.evaluation import RetrieverEvaluator embed_model.input_type = "search_query" retriever = vector_index.as_retriever(similarity_top_k=10, embed_model=embed_model) retriever_evaluator = RetrieverEvaluator.from_metric_names( ["mrr", "hit_rate"], retriever=retriever ) eval_results = await retriever_evaluator.aevaluate_dataset(qc_dataset) print(display_results_retriever(f"Retriever_cohere_embeds", eval_results)) ``` ``` Retriever Name Hit Rate MRR 0 Retriever_cohere_embeds 0.965517 0.754823 ``` These embeddings show a lower Hit Rate but a better MRR value. <aside> 💡 In this tutorial, we test the cohere embeddings, but you can also try any [embedding models](https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#local-embedding-models) from the Hugging Face hub. The models at the top of the [mteb/leaderboard](https://huggingface.co/spaces/mteb/leaderboard) are a good choice to try. </aside> <aside> 💡 If pre-trained embedding models do not perform well on your data, consider [fine-tuning](https://docs.llamaindex.ai/en/stable/examples/finetuning/embeddings/finetune_embedding.html) your own embedding model. </aside>


# 3. **Incorporating a Reranker**
Here, we will be testing three different Rerankers that we learned about in previous lessons. - `cross-encoder/ms-marco-MiniLM-L-6-v2` from the [Hugging Face hub](https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-6-v2). - [LlamaIndex’s](https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/LLMReranker-Gatsby.html) `LLMRerank` - [Cohere’s](https://docs.cohere.com/docs/reranking) `CohereRerank`. ```python from llama_index.postprocessor.cohere_rerank import CohereRerank from llama_index.indices.postprocessor import SentenceTransformerRerank, LLMRerank st_reranker = SentenceTransformerRerank( top_n=5, model="cross-encoder/ms-marco-MiniLM-L-6-v2" ) llm_reranker = LLMRerank( choice_batch_size=4, top_n=5, ) cohere_rerank = CohereRerank(api_key=os.getenv('COHERE_API_KEY'), top_n=10) for reranker in [cohere_rerank, st_reranker, llm_reranker]: retriever_with_reranker = vector_index.as_retriever(similarity_top_k=10, postprocessor=reranker, embed_model=embed_model) retriever_evaluator_1 = RetrieverEvaluator.from_metric_names( ["mrr", "hit_rate"], retriever=retriever_with_reranker ) eval_results1 = await retriever_evaluator_1.aevaluate_dataset(qc_dataset) print(display_results_retriever("Retriever with added Reranker", eval_results1)) ``` ``` config.json: 100% 794/794 [00:00<00:00, 23.6kB/s] pytorch_model.bin: 100% 90.9M/90.9M [00:00<00:00, 145MB/s] tokenizer_config.json: 100% 316/316 [00:00<00:00, 11.0kB/s] vocab.txt: 100% 232k/232k [00:00<00:00, 3.79MB/s] special_tokens_map.json: 100% 112/112 [00:00<00:00, 3.88kB/s] Retriever Name Hit Rate MRR 0 Retriever with added Reranker 0.965517 0.754823 Retriever Name Hit Rate MRR 0 Retriever with added Reranker 0.965517 0.754823 Retriever Name Hit Rate MRR 0 Retriever with added Reranker 0.965517 0.754823 ``` Here, we unfortunately don't see a significant improvement in the retriever's performance. We suspect it is mainly caused by the evaluation dataset we’ve built. Rerankers can nonetheless offer great benefits depending on your application and are easy to implement.


# 4. **Employing Deep Memory**
[Activeloop's Deep Memory](https://www.activeloop.ai/resources/use-deep-memory-to-boost-rag-apps-accuracy-by-up-to-22/) is a feature that introduces a tiny neural network layer trained to match user queries with relevant data from a corpus. While this addition incurs minimal latency during search, it can **boost retrieval accuracy by up to 27%.** First, let's reuse and convert our generated dataset into a format Deep Memory expects. We need queries and relevant IDs. ```python def create_query_relevance(qa_dataset): """Function for converting LlamaIndex dataset to correct format for deep memory training""" queries = [text for _, text in qa_dataset.queries.items()] relevant_docs = qa_dataset.relevant_docs relevance = [] for doc in relevant_docs: relevance.append([(relevant_docs[doc][0], 1)]) return queries, relevance train_queries, train_relevance = create_query_relevance(qc_dataset) print(len(train_queries)) ``` Now, let's upload our baseline Vectore Store on Activeloop's cloud platform and convert it into a managed database. ```python import deeplake local = "./data/paul_graham/deep_lake_db" hub_path = "hub://genai360/optimization_paul_graham" hub_managed_path = "hub://genai360/optimization_paul_graham_managed" # First upload our local vector store deeplake.deepcopy(local, hub_path, overwrite=True) # Create a managed vector store deeplake.deepcopy(hub_path, hub_managed_path, overwrite=True, runtime={"tensor_db": True}) ``` You can replace the paths using your organization name and database name. Let’s create a LlamaIndex RAG pipeline using our new managed vector store. ```python import os from llama_index import VectorStoreIndex, ServiceContext, StorageContext from llama_index.vector_stores import DeepLakeVectorStore from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.llms import OpenAI vector_store = DeepLakeVectorStore(dataset_path=hub_managed_path, overwrite=False, runtime={"tensor_db": True}, read_only=True) llm = OpenAI(model="gpt-3.5-turbo-1106") embed_model = OpenAIEmbedding() service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm,) storage_context = StorageContext.from_defaults(vector_store=vector_store) vector_index = VectorStoreIndex.from_vector_store(vector_store,service_context=service_context, storage_context=storage_context, use_async=False, show_progress=True) ``` ``` Deep Lake Dataset in hub://genai360/optimization_paul_graham_managed already exists, loading from the storage ``` And now we can launch the Deep Memory training. ```python from langchain.embeddings.openai import OpenAIEmbeddings embeddings = OpenAIEmbeddings() job_id = vector_store.vectorstore.deep_memory.train( queries=train_queries, relevance=train_relevance, embedding_function=embeddings.embed_documents, ) ``` ``` Your Deep Lake dataset has been successfully created! creating embeddings: 100%|██████████| 1/1 [00:02<00:00, 2.27s/it] 100%|██████████| 100/100 [00:00<00:00, 158.16it/s] Dataset(path='hub://genai360/optimization_paul_graham_managed_queries', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (100, 1) str None metadata json (100, 1) str None embedding embedding (100, 1536) float32 None id text (100, 1) str None DeepMemory training job started. Job ID: 652dceeed7d1579bf6abf3df ``` With the job_id, you can keep track of deep memory training. ```python vector_store.vectorstore.deep_memory.status('652dceeed7d1579bf6abf3df') ``` To evaluate our Deep Memory-enabled vector store, we can generate a test dataset. Here, we only send 20 chunks to make things fast, but a bigger dataset size would be recommended for a stronger evaluation. ```python from llama_index.evaluation import generate_question_context_pairs # Generate test dataset test_dataset = generate_question_context_pairs( nodes[:20], llm=llm, num_questions_per_chunk=1 ) test_dataset.save_json("test_dataset.json") # We can also load the dataset from a json file if already done previously. from llama_index.finetuning.embeddings.common import ( EmbeddingQAFinetuneDataset, ) test_dataset = EmbeddingQAFinetuneDataset.from_json( "test_dataset.json" ) test_queries, test_relevance = create_query_relevance(test_dataset) ``` ``` 100%|██████████| 20/20 [00:29<00:00, 1.49s/it] ``` Let’s evaluate the recall on the generated test dataset using the Deep Lakes evaluation Python function. Recall measures the proportion of relevant items successfully retrieved by the system from all relevant items available in the dataset. **Formula**: Recall is calculated as: $$ \text{Recall} = \frac{\text{Number of Relevant Items Retrieved}}{\text{Total Number of Relevant Items in the Dataset}} $$ It focuses on the system's ability to find all relevant items. A high recall means the system is good at not missing relevant items. <aside> 💡 **Compared to Hit Rate:** Recall is about the system's thoroughness in retrieving all relevant items, and Hit Rate is about its effectiveness in ensuring that each query retrieves something relevant. </aside> ```python # Evaluate recall on the generated test dataset recalls = vector_store.vectorstore.deep_memory.evaluate( queries=test_queries, relevance=test_relevance, embedding_function=embeddings.embed_documents, ) ``` ``` Embedding queries took 1.24 seconds ---- Evaluating without Deep Memory ---- Recall@1: 55.2% Recall@3: 87.1% Recall@5: 90.5% Recall@10: 97.4% Recall@50: 100.0% Recall@100: 100.0% ---- Evaluating with Deep Memory ---- Recall@1: 56.0% Recall@3:


# 4. **Employing Deep Memory**
87.1% Recall@5: 92.2% Recall@10: 99.1% Recall@50: 100.0% Recall@100: 100.0% ``` Now, let’s get the Hit Rate and MRR scores of our Deep Memory enabled vector store. We start measuring the Hit Rate and MRR of our base vector store: ```python import os from llama_index.postprocessor.cohere_rerank import CohereRerank from llama_index.evaluation import ( RetrieverEvaluator, ) base_retriever = vector_index.as_retriever(similarity_top_k=10) deep_memory_retriever = vector_index.as_retriever( similarity_top_k=10, vector_store_kwargs={"deep_memory": True} ) base_retriever_evaluator = RetrieverEvaluator.from_metric_names( ["mrr", "hit_rate"], retriever=base_retriever ) eval_results = await base_retriever_evaluator.aevaluate_dataset(test_dataset) print(display_results_retriever("Retriever Results", eval_results)) ``` ``` Retriever Name Hit Rate MRR 0 Retriever Results 0.974138 0.717809 ``` Now, the same evaluation for the Deep Memory Vector Store ```python deep_memory_retriever = vector_index.as_retriever( similarity_top_k=10, vector_store_kwargs={"deep_memory": True} ) dm_retriever_evaluator = RetrieverEvaluator.from_metric_names( ["mrr", "hit_rate"], retriever=deep_memory_retriever ) dm_eval_results = await dm_retriever_evaluator.aevaluate_dataset(test_dataset) print(display_results_retriever("Retriever Results", dm_eval_results)) ``` ``` Retriever Name Hit Rate MRR 0 Retriever Results 0.991379 0.72865 ``` We can see a small increase in the MRR score compared to the baseline RAG pipeline while our Hit Rate stays the same. Note that this is again mainly due to our evaluation test set and the fact that we only chose 20 chunks. You can experiment with more chunks or manually build a different test set for improved results, especially in your application!


# Conclusion
In this lesson, optimizing a LlamaIndex RAG pipeline involved a structured approach to improve information retrieval and generation quality. We adjusted retrieval top_k values, evaluated two embedding models, introduced reranking mechanisms, and integrated Active Loop’s Deep Memory, some leading to performance enhancements. The improvements were somewhat negligible in this short demo. Still, it is crucial to try these more advanced improvements as they could have high impacts on a real application and improved evaluation dataset. We also highlight the importance of a good evaluation set of tools, such as a well-curated and large enough evaluation dataset. --- **RESOURCES** - Colab notebook for the lesson: [Google Colaboratory](https://colab.research.google.com/drive/1BZGMXoQVEt7__6dNfRyBAHXwr4kQw4PZ#scrollTo=I93VcS630cbQ) - LlamaIndex and Deep Memory integration: [How to get +15% RAG hit_rate improvement for question answering on documentation? - LlamaIndex 🦙 0.9.15.post2](https://docs.llamaindex.ai/en/stable/examples/retrievers/deep_memory.html) --- This lesson is based on the Llamaindex AI-engineer-workshop https://github.com/run-llama/ai-engineer-workshop posted by [Disiok](https://github.com/Disiok). [GitHub - run-llama/ai-engineer-workshop](https://github.com/run-llama/ai-engineer-workshop)


RAG - Metrics & Evaluation

# **Introduction**
So far, we've observed that LLMs' response generation ability is improved by incorporating context from a vector database, a typical design approach used in RAG systems for chatbots and question-answering systems. RAG applications strive to produce factually grounded outputs supported by the context they retrieve. Creating an evaluation pipeline for these systems is crucial, as it allows for measuring the effectiveness of the added techniques. The evaluation process should focus on ensuring that the output meaningfully incorporates the context, avoiding mere repetition, and aiming to create responses that are comprehensive, non-repetitive, and devoid of redundancy.


RAG - Metrics & Evaluation

# **RAG Metrics**
A holistic approach in RAG system evaluation is to present a detailed assessment of individual components and the system as a whole. Setting the baseline values for elements like chunking logic and embedding models and then examining each part independently and in an end-to-end manner is key for understanding the impact of modifications on the system’s overall performance. The holistic modules don't always need ground-truth labels, as they can be evaluated based on the query, context, response, and LLM interpretations. Here are five metrics commonly used to evaluate RAG systems: - **[Correctness](https://docs.llamaindex.ai/en/latest/examples/evaluation/correctness_eval.html)**: Checks if the answer generated matches the reference answer for the given query (labels required). The accuracy of the generated answer is verified by comparing it directly to a reference answer provided for the question. - **[Faithfulness](https://docs.llamaindex.ai/en/latest/examples/evaluation/faithfulness_eval.html)**: Determines if the answer is accurate and doesn't contain fabrication, relative to the retrieved contexts. The faithfulness metric evaluates the integrity of the answer, ensuring it faithfully represents the information in the retrieved context by checking that the answer is accurate and free from distortions or fabrications that could misrepresent the source material. - **[Context Relevancy](https://docs.llamaindex.ai/en/latest/examples/evaluation/relevancy_eval.html)**: Measures the relevance of the retrieved context and the resulting answer to the original query. It does so by ensuring that the system only retrieves information in a way that is pertinent to the user's request. - **[Guideline Adherence](https://docs.llamaindex.ai/en/latest/examples/evaluation/guideline_eval.html)**: Determines if the predicted answer follows a set of guidelines and whether the response meets predefined criteria, encompassing stylistic, factual, and ethical standards, so the answer responds to the query while also aligning with specific established norms. - [**Embedding Semantic Similarity](https://docs.llamaindex.ai/en/latest/examples/evaluation/semantic_similarity_eval.html#):** Calculates the similarity score between embeddings of the generated answer and the reference answer (reference labels required). The analysis starts broadly, focusing on the overarching goal of RAG applications to produce helpful outputs supported by contextually relevant facts from the retrievers. It is then narrowed down to specific evaluation metrics, including faithfulness, answer relevancy, and the [Sensibleness and Specificity Average (SSA)](https://arxiv.org/abs/2001.09977), focusing on avoiding hallucination in responses. Google's SSA metric evaluates open-domain chatbot responses for sensibleness (contextual coherence) and specificity (detailed and direct responses). Initially involving human evaluators, this approach aims to ensure outputs are comprehensive yet not overly vague. A high faithfulness score does not guarantee high relevance. For example, an answer that accurately reflects the context but lacks direct relevance to the question would score lower in answer relevance, especially if it includes incomplete or redundant information. ### Faithfulness Evaluator Avoiding vague responses is essential, but preventing LLMs from “hallucinating” is equally crucial. Hallucination refers to generating responses not grounded in factual content or context. LlamaIndex's `FaithfulnessEvaluator` assesses responses based on their alignment with the retrieved context, measuring this aspect. Faithfulness evaluation considers whether the response matches the retrieved context, aligns with the query, and adheres to the reference answer or guidelines. The result returns a boolean value indicating whether the response passed the accuracy and faithfulness checks. To execute the following codes, we must initially install the necessary libraries. This can be done using the Python package manager. Afterward, set the API keys for both OpenAI and the Activeloop service, as we will access indexes from a dataset hosted on Deep Lake. Remember to replace the placeholders with your API keys. The code for this lesson is also available through a [Colab notebook](https://colab.research.google.com/drive/1XcCcoUi5ZbUNmTCLYXL51zeD-6mXON5T?usp=sharing), where you can follow along. ```bash pip install -q llama-index==0.9.14.post3 deeplake==3.8.12 openai==1.3.8 cohere==4.37 ``` ```python import os os.environ["OPENAI_API_KEY"] = "<YOUR_OPENAI_KEY>" os.environ["ACTIVELOOP_TOKEN"] = "<YOUR_ACTIVELOOP_TOKEN>" ``` Here's an example illustrating how to evaluate a single response for faithfulness. ```python from llama_index import ServiceContext from llama_index.llms import OpenAI # build service context


RAG - Metrics & Evaluation

# **RAG Metrics**
llm = OpenAI(model="gpt-4", temperature=0.0) service_context = ServiceContext.from_defaults(llm=llm) from llama_index.vector_stores import DeepLakeVectorStore from llama_index.storage.storage_context import StorageContext from llama_index import VectorStoreIndex vector_store = DeepLakeVectorStore(dataset_path="hub://genai360/LlamaIndex_paulgraham_essay", overwrite=False) storage_context = StorageContext.from_defaults(vector_store=vector_store) index = VectorStoreIndex.from_vector_store( vector_store, storage_context=storage_context ) from llama_index.evaluation import FaithfulnessEvaluator # define evaluator evaluator = FaithfulnessEvaluator(service_context=service_context) # query index query_engine = index.as_query_engine() response = query_engine.query( "What does Paul Graham do?" ) eval_result = evaluator.evaluate_response(response=response) print( "> response:", response ) print( "> evaluator result:", eval_result.passing ) ``` ``` > response: Paul Graham is involved in various activities. He is a writer and has given talks on topics such as starting a startup. He has also worked on software development, including creating software for generating websites and building online stores. Additionally, he has been a studio assistant for a beloved teacher who is a painter. > evaluator result: True ``` Most of the previously mentioned code should be recognizable, as it involves generating an index from the Deep Lake vector store, using this index to query the LLM, and conducting the evaluation procedure. The query engine processes the question, and its response is then forwarded to the evaluator for analysis. Let’s focus on the evaluation process, which starts by setting up an evaluator to assess the accuracy of responses based on the service context. - The code initializes a `FaithfulnessEvaluator`, a tool designed to assess the accuracy of responses generated by the language model (GPT-4 in this case). - The evaluator uses the `service_context` created earlier, which includes the configured GPT-4 model. This context provides the necessary environment and parameters for the language model to function. - The primary role of the `FaithfulnessEvaluator` is to determine how closely the language model's responses adhere to accurate and reliable information. It uses a set of criteria or algorithms to compare the generated responses against known factual data or expected outputs. The evaluator then checks the response for its faithfulness to factual information. This means it evaluates whether the response accurately and reliably reflects historical facts about the queried topic. The result of this evaluation (`eval_result`) is then checked to see if it meets the standards of accuracy set by the evaluator, indicated by `eval_result.passing`. The result returns a boolean value indicating whether the response passed the accuracy and faithfulness checks.


RAG - Metrics & Evaluation

# ****Retrieval Evaluation Metrics****
The evaluation of retrieval in RAG systems involves determining the relevance of documents to specific queries. <aside> ✴ RAG system outputs need to encompass three critical aspects: factual accuracy, direct relevance to the query, and inclusion of essential contextual information. </aside> In information retrieval, the main goal is to identify unstructured data that meets a specific information requirement within a database. ![ the evaluation metrics of retrieval in RAG systems.](RAG%20-%20Metrics%20&%20Evaluation%205f19e1a52ec04b1c9f69052c39bc9ddc/image_2023-12-17_064848082.png) the evaluation metrics of retrieval in RAG systems. Metrics for evaluating a retriever include Mean Reciprocal Rank (MRR), Hit Rate, MAP and NDCG. - **[MRR](https://en.wikipedia.org/wiki/Mean_reciprocal_rank)** measures the retrieval system's ability to return the best result as high up in the ranking as possible. - **Hit Rate** evaluates the presence of relevant items within the top results returned, which is crucial where users only consider the first few results. - **[MAP (Mean Average Precision)](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Mean_average_precision)**: is a measure of ranking quality across multiple queries. MAP calculates the mean of the average precisions for each query, where the average precision is computed as the mean of the precision scores after each relevant document is retrieved. - **[NDCG (Normalized Discounted Cumulative Gain)](https://en.wikipedia.org/wiki/Discounted_cumulative_gain)**: This metric evaluates the ranking of documents based on their relevance, giving more importance to relevant documents that appear higher in the ranking. It is normalized so that the perfect ranking's score is 1, allowing for comparison across different sets of queries. The `RetrieverEvaluator` is a more advanced technique from LlamaIndex, which can calculate metrics such as Mean Reciprocal Rank (MRR) and Hit Rate. It is designed to evaluate the efficacy of a retrieval system, which retrieves data pertinent to user queries from a database or index. This class assesses the retriever's performance in relation to specific questions and expected results, providing benchmarks for evaluation. Inferred from the evaluator's definition, it's necessary to compile an evaluation dataset comprising the contents, a collection of questions, and corresponding nodes that serve as references for answering these questions. The `generate_question_context_pairs` function in LlamaIndex can take care of generating an evaluation dataset. The rest of the process involves passing a query and using the dataset as a reference to ensure the chatbot retrieves the right documents. Please read the [following tutorial](https://docs.llamaindex.ai/en/stable/examples/evaluation/retrieval/retriever_eval.html) for an example. <aside> 💡 While the single-query evaluation is presented, the practical implementation in real-world scenarios often requires batch evaluations. A large set of queries and expected results are extracted from the retriever to determine its overall reliability. In batch testing, the retriever undergoes an assessment involving many queries and their corresponding expected results. Systematically querying the retriever with these varied inputs and evaluating its outputs against predefined correct answers is the method employed to measure its consistency accurately. </aside>


RAG - Metrics & Evaluation

# **Golden Context Dataset**
The [Golden Context dataset](https://github.com/microsoft/promptflow-resource-hub/blob/main/sample_gallery/golden_dataset/copilot-golden-dataset-creation-guidance.md) would consist of carefully selected queries paired with an ideally matched set of sources that contain the answers. Optionally, it could also include the perfect answers that are expected to be generated by the LLM. For our purposes, 177 representative user queries have been manually curated. For each query, the most relevant source within our documentation has been diligently identified so that these sources directly address the queries in question. The Golden Context Dataset serves as our benchmark for precision evaluation. The dataset is structured around 'question' and 'source' pairings. To create a Golden Dataset, gather a set of realistic customer questions and pair them with expert answers, then use this dataset to compare against responses from a language model for quality assurance, ensuring the LLM's answers align closely with the expert ones for accuracy and relevance. If you want to learn more about the dataset's creation, follow [the link in the resources](https://github.com/microsoft/promptflow-resource-hub/blob/main/sample_gallery/golden_dataset/copilot-golden-dataset-creation-guidance.md) section. Once the golden dataset is ready, the next step is to use it to measure the quality of LLM responses. After each evaluation, metrics like the following will be available to quantify the user experience. For example: | Similarity | Relevance | Coherence | Grounded-ness | | --- | --- | --- | --- | | 3.7 | 77 | 88 | 69 | <aside> ⚠ Note: While synthetic question generation has benefits, it's not recommended for evaluating metrics related to real-world user experiences. Questions used in such evaluations should ideally mirror what actual users might ask, something not typically achieved with questions generated by an LLM. Hand-crafting questions with a focus on the users’ perspective is advised for a more accurate reflection. </aside>


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
LlamaIndex incorporates various evaluation tools designed to foster community engagement and collaborative endeavors. The devices are structured to support a shared process of assessing and enhancing the system, empowering users and developers to play an active role in the evaluation. Through the use of tools shaped by community input, LlamaIndex creates a collaborative environment where constant feedback is smoothly incorporated, contributing to continual development. Notable tools in this ecosystem include: - [Ragas](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/llamaindex.ipynb): Another key tool that provides a framework for evaluating and integrating with LlamaIndex, offering detailed metrics. - [DeepEval](https://docs.llamaindex.ai/en/latest/module_guides/evaluating/root.html#../../../community/integrations/deepeval.md): A tool designed for in-depth evaluation, facilitating comprehensive assessments of various aspects of the system.


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
## **Evaluating with Ragas**
The evaluation process involves importing specific metrics from Ragas, such as *faithfulness, answer relevancy, context precision, context recall, and harmfulness.* When evaluating using Ragas, the following elements are essential: - **Query Engine**: This is the primary component and acts as the core of the evaluation process, where its performance is assessed. - **Metrics**: Ragas provides a range of metrics specifically designed to evaluate a nuanced assessment of the engine's capabilities. - **Questions**: A curated set of questions is required, which are used to probe the engine's ability to retrieve and generate accurate responses. We must first set up a query engine to demonstrate how to use the Ragas library, which involves loading a document. For this, we will use the contents of the "New York City" Wikipedia page as our source document. Additionally, installing two more libraries is necessary: one for processing the webpage content and the other for the evaluation library itself. ```bash pip install html2text==2020.1.16 ragas==0.0.22 ``` At this point, we can use the `SimpleWebPageReader` class by providing a URL as its argument to load the content. These documents can then create the index and the query engine. Then, it is possible to ask questions about the document! ```bash from llama_index.readers.web import SimpleWebPageReader from llama_index import VectorStoreIndex, ServiceContext documents = SimpleWebPageReader(html_to_text=True).load_data( ["https://en.wikipedia.org/wiki/New_York_City"] ) vector_index = VectorStoreIndex.from_documents( documents, service_context=ServiceContext.from_defaults(chunk_size=512) ) query_engine = vector_index.as_query_engine() response_vector = query_engine.query("How did New York City get its name?") print(response_vector) ``` ``` New York City got its name in honor of the Duke of York, who later became King James II of England. The Duke of York was appointed as the proprietor of the former territory of New Netherland, including the city of New Amsterdam, when England seized it from Dutch control. ``` Returning to our goal of evaluating the models, the next step involves composing a series of questions, ideally derived from the original document, to ensure a more accurate performance assessment. ```python eval_questions = [ "What is the population of New York City as of 2020?", "Which borough of New York City has the highest population?", "What is the economic significance of New York City?", "How did New York City get its name?", "What is the significance of the Statue of Liberty in New York City?", ] eval_answers = [ "8,804,000", # incorrect answer "Queens", # incorrect answer "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.", "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.", "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.", ] eval_answers = [[a] for a in eval_answers] ``` This


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
## **Evaluating with Ragas**
stage is the setup phase of the evaluation process. QueryEngine's proficiency is assessed based on how effectively it processes and responds to these specific questions, utilizing the answers as a standard for measuring performance. We need to import the metrics from the Ragas library. ```python from ragas.metrics import ( faithfulness, answer_relevancy, context_precision, context_recall, ) from ragas.metrics.critique import harmfulness metrics = [ faithfulness, answer_relevancy, context_precision, context_recall, harmfulness, ] ``` The `metrics` list compiles the metrics into a collection, which can then be used in the evaluation process to assess various aspects of the `QueryEngine`'s performance. The results, which include scores for each metric, can be further analyzed. Finally, let's run the evaluation: ```python from ragas.llama_index import evaluate result = evaluate(query_engine, metrics, eval_questions, eval_answers) # print the final scores print(result) ``` ```python evaluating with [faithfulness] 100%|██████████| 1/1 [00:16<00:00, 16.95s/it] evaluating with [answer_relevancy] 100%|██████████| 1/1 [00:03<00:00, 3.54s/it] evaluating with [context_precision] 100%|██████████| 1/1 [00:02<00:00, 2.73s/it] evaluating with [context_recall] 100%|██████████| 1/1 [00:07<00:00, 7.06s/it] evaluating with [harmfulness] 100%|██████████| 1/1 [00:02<00:00, 2.16s/it] {'faithfulness': 0.8000, 'answer_relevancy': 0.7634, 'context_precision': 0.6000, 'context_recall': 0.8667, 'harmfulness': 0.0000} ``` The metrics analysis quantifies different aspects of the RAG system's performance: 1. `faithfulness`: **0.8000** - measures how accurately the system's responses adhere to the factual content of the source material. A score of 0.7 indicates *relatively high* faithfulness, meaning the responses are mostly accurate and true to the source. 2. `answer_relevancy`: **0.7634** - measures how relevant the system's responses are to the given queries. A high score of 0.955 suggests that the majority of the system's responses are *closely aligned* with the queries' intent. 3. `context_precision`: **0.6000** - evaluates the precision of the context used by the system to generate responses. A lower score of 0.2335 indicates that the context used *often includes irrelevant* information. 4. `context_recall`: **0.8667** - measures the recall rate of relevant context determined by the system. A high score of 0.98 suggests that the system is *very effective* in retrieving most of the relevant context. 5. `harmfulness`: **0.0000** - measures the system for harmful or inappropriate content generation. A score of 0 implies that *no harmful content* was generated in the evaluated responses.


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
## **The Custom RAG Pipeline Evaluation**
For a practical evaluation of a custom RAG system, it is essential to employ a range of assessment benchmarks instrumental in assessing various facets of the RAG system, such as its effectiveness and reliability. The variety of measures guarantees a detailed evaluation and in-depth insight into the system's overall capabilities. This section involves developing a tailored evaluation pipeline, beginning with loading a dataset, forming an evaluation dataset from its contents, and then calculating the metrics we have previously discussed. Initially, we download the text file to serve as the dataset. ```bash wget 'https://raw.githubusercontent.com/idontcalculate/data-repo/main/venus_transmission.txt' ``` ``` --2023-12-18 20:21:53-- https://raw.githubusercontent.com/idontcalculate/data-repo/main/venus_transmission.txt Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 19241 (19K) [text/plain] Saving to: ‘venus_transmission.txt’ venus_transmission. 100%[===================>] 18.79K --.-KB/s in 0.001s 2023-12-18 20:21:53 (32.6 MB/s) - ‘venus_transmission.txt’ saved [19241/19241] ``` Then, the text file can be loaded as a `Document` object, identifiable by LlamaIndex. ```python from llama_index import SimpleDirectoryReader reader = SimpleDirectoryReader(input_files=["/content/venus_transmission.txt"]) docs = reader.load_data() print(f"Loaded {len(docs)} docs") ``` ``` Loaded 1 docs ``` The `SimpleNodeParser`, in this context, converts documents into a structured format known as nodes and serves for customization in parsing documents, specifically in terms of defining the chunk size, managing overlap, and incorporating metadata. Each chunk of the document is treated as a node. In this case, the parser is set with a `chunk_size` of 512, meaning each node will consist of 512 characters from the original document. These chunks can then be used to generate the indexes. ```python from llama_index.node_parser import SimpleNodeParser from llama_index import VectorStoreIndex # Build index with a chunk_size of 512 node_parser = SimpleNodeParser.from_defaults(chunk_size=512) nodes = node_parser.get_nodes_from_documents(docs) vector_index = VectorStoreIndex(nodes) ``` The indexes can now be used as a query engine to query a specific question concerning the source document. ```python query_engine = vector_index.as_query_engine() response_vector = query_engine.query("What was The first beings to inhabit the planet?") print( response_vector.response ) ``` ``` The first beings to inhabit the planet were a dinoid and reptoid race from two different systems outside our solar system. ``` The response generated by the query engine is stored in `response_vector`. So, the document is processed into nodes, indexed, and then queried using a language model. To investigate the response further, we can use the `.source_nodes` key to access the retrieved document from the index used to answer the question. ```python # First retrieved node response_vector.source_nodes[0].get_text() ``` ``` They had heard of this beautiful new planet. At this time, Earth had two moons to harmonize the weather conditions and control the tides of the large bodies of water. The first beings to inhabit the planet were a dinoid and reptoid race from two different systems outside our solar system. They were intelligent and walked on two legs like humans and were war-like considering themselves to be superior to all other life forms. In the past, the four races of humans had conflicts with them before they outgrew such behavior. They arrived on Earth to rob it of its minerals and valuable gems. Soon they had created a terrible war. They were joined by re- 1 enforcements from their home planets. One set up its base on one of the Earth's moons, the other on Earth. It was a terrible war with advanced nuclear and laser weapons like you see in your science fiction movies. It lasted very long. Most of the life forms lay in singed waste and the one moon was destroyed. No longer interested in Earth, they went back to their planets leaving their wounded behind, they had no use for them. The four races sent a few forces


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
## **The Custom RAG Pipeline Evaluation**
to see if they could help the wounded dinoids and reptilians and to see what they could do to repair the Earth. They soon found that due to the nuclear radiation it was too dangerous on Earth before it was cleared. Even they had to remain so as not to contaminate their own planets. Due to the radiation, the survivors of the dinoids and reptoids mutated into the Dinosaurs and giant reptilians you know of in your history. The humans that were trapped there mutated into what you call Neanderthals. The Earth remained a devastated ruin, covered by a huge dark nuclear cloud and what vegetation was left was being devoured by the giant beings, also humans and animals by some. It was this way for hundreds of years before a giant comet crashed into one of the oceans and created another huge cloud. This created such darkness that the radiating heat of the Sun could not interact with Earth's gravitational field and an ice age was created. This destroyed the mutated life forms and gave the four races the chance to cleanse and heal the Earth with technology and their energy. Once again, they brought various forms of life to the Earth, creating again a paradise, except for extreme weather conditions and extreme tidal activities. ``` We can index the second item on the list to view the content of the second node that contributed to the generation of the content. ```python # Second retrieved node response_vector.source_nodes[1].get_text() ``` ``` Due to the radiation, the survivors of the dinoids and reptoids mutated into the Dinosaurs and giant reptilians you know of in your history. The humans that were trapped there mutated into what you call Neanderthals. The Earth remained a devastated ruin, covered by a huge dark nuclear cloud and what vegetation was left was being devoured by the giant beings, also humans and animals by some. It was this way for hundreds of years before a giant comet crashed into one of the oceans and created another huge cloud. This created such darkness that the radiating heat of the Sun could not interact with Earth's gravitational field and an ice age was created. This destroyed the mutated life forms and gave the four races the chance to cleanse and heal the Earth with technology and their energy. Once again, they brought various forms of life to the Earth, creating again a paradise, except for extreme weather conditions and extreme tidal activities. During this time they realized that their planets were going into a natural dormant stage that they would not be able to support physical life. So they decided to colonize the Earth with their own people. They were concerned about the one moon, because it is creating earthquakes and tidal waves and storms and other difficulties for the structure of the Earth. They knew how to drink fluids to protect and balance themselves. These were the first colonies like Atlantis and Lemuria. The rest of the people stayed on their planets to await their destiny. They knew that they would perish and die. They had made the decision only to bring the younger generation with some spiritual teachers and elders to the Earth. The planet was too small for all of them. But they had no fear of death. They had once again created a paradise. They were instructed to build special temples here as doorways to the other dimensions. Because of the aggressive beings, the temples were hidden for future times when they will be important. There they could do their meditations and the higher beings. They


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
## **The Custom RAG Pipeline Evaluation**
were informed to build two shields around the Earth out of ice particles to balance the influence of the one moon. They created a tropical climate for the Earth. There were no deserts at that time. They have special crystals for these doorways and they were able to lower their vibration to enter through these doorways. The news spread of the beautiful planet. ``` You can view the textual information from the second node that the query engine found relevant, providing additional context or information in response to the query. This helps understand the breadth of knowledge the query engine pulls from and how different parts of the indexed documents contribute to the overall response. As previously mentioned, our task involves creating an evaluation dataset. This essentially entails generating a series of questions and their respective answers, all in relation to the content we have loaded. The `generate_question_context_pairs` class leverages the LLM to create questions based on the content of each node: For each node, two questions will be created, resulting in a dataset where each item consists of a context (the node's text) and a corresponding set of questions. The Q&A dataset will serve us to evaluate the capabilities of an RAG system in question generation and context understanding tasks. You can see the first ten questions in the output. ```python from llama_index.llms import OpenAI from llama_index.evaluation import generate_question_context_pairs # Define an LLM llm = OpenAI(model="gpt-3.5-turbo") qa_dataset = generate_question_context_pairs( nodes, llm=llm, num_questions_per_chunk=2 ) queries = list(qa_dataset.queries.values()) print( queries[0:10] ) ``` ``` 100%|██████████| 13/13 [00:31<00:00, 2.46s/it] ['Explain the role of different alien races in the history of our solar system according to the information provided. How did these races contribute to the transformation process and why was Earth considered a special planet?', 'Describe the advanced abilities and technology possessed by the Masters and beings mentioned in the context. How did their understanding of creation and their eternal nature shape their perspective on life and death?', 'How did the four races of humans demonstrate their mastery of creativity and what were the potential consequences of using this power for selfish reasons?', 'Describe the initial state of Earth before it became a planet and how the four races of humans contributed to its transformation into a unique paradise.', 'How did the arrival of the dinoid and reptoid races on Earth lead to a devastating war? Discuss the reasons behind their conflict with the four races of humans and the impact it had on the planet.', "Explain the process of mutation that occurred among the survivors of the dinoids and reptoids, resulting in the emergence of dinosaurs and Neanderthals. Discuss the role of nuclear radiation and its effects on the Earth's environment and living organisms.", 'How did the survivors of the dinoids and reptoids mutate into the dinosaurs and giant reptilians we know of in history? Explain the role of radiation in this process.', 'Describe the events that led to the creation of an ice age on Earth. How did this ice age affect the mutated life forms and provide an opportunity for the four races to cleanse and heal the Earth?', 'Explain the purpose and significance of building special temples as doorways to other dimensions in the context of the given information. How did these temples serve the people and protect them from the dark forces?', 'Discuss the actions taken by the colonies in response to the war declared by another race of humans. How did the colonies ensure the preservation of their knowledge and technology, and what measures did they take to protect themselves from the dark forces?', 'How did the


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
## **The Custom RAG Pipeline Evaluation**
inhabitants of Lemuria and Atlantis ensure that their knowledge and technology would not be misused by the dark forces?', 'What measures were taken by the controlling forces to prevent the people from communicating with other dimensions and remembering their past lives or the hidden temples?', 'How has the manipulation and control of human beings by the rich and powerful impacted society throughout history? Discuss the role of religion, race, and power in perpetuating this control and the potential consequences for humanity.', 'Explain the role of the Galactic Brotherhood and other spiritually evolved beings in the transformation of Earth. How have they worked to change the energy of the planet and its inhabitants? Discuss the potential risks they aim to mitigate, such as genetic manipulation and the use of destructive technologies.', "Explain the role of the Galactic Brotherhood in the transformation of the planet's energy and the introduction of new technologies. How are different beings, such as the Spiritual Hierarchy, Ascended Masters, and nature spirits, cooperating in this process?", 'Discuss the significance of the hidden temples and the space ships in the frequency change of the Earth. How do these elements contribute to the gradual transformation and what effects do they have on the environment?', 'Explain the concept of chakras and their role in the transformation process described in the context information. How do chakras relate to the abilities of mental telepathy, intuition, and past life recollection?', "Discuss the significance of the Earth's future purpose as mentioned in the context information. How does it differ from its past role? How does the concept of yin and yang, as well as the negative and positive energies, tie into this transformation?", 'How does the concept of division into good and bad energies contribute to the perpetuation of negative forces and selfishness among individuals?', 'Discuss the shift in power dynamics from feminine qualities to male energy in societies after genetic manipulation. How does the future vision of equal and balanced male and female powers impact the purpose of Earth for human beings?', 'How has the balance of feminine and masculine energies shifted throughout human history, and what is the envisioned future for this balance on Earth?', 'In the future described in the context information, how will individuals govern themselves and what role will manmade laws play in society?', 'How does the concept of obeying spiritual laws contribute to living in harmony on other planets for millions of years? Provide examples or evidence from the context information to support your answer.', 'According to the context information, what are some key aspects of the future living style and awareness on Earth after the transformation is complete? How do these aspects differ from the current state of existence?', "How does the concept of eternity and the ability to overcome time and aging impact one's perspective on life and the enjoyment of experiences?", 'In what ways can individuals create a balance and harmony within themselves, and why is it important for them to do so?'] ``` The `RetrieverEvaluator` class can now use this QA dataset to evaluate the retriever's performance. It queries each question using the retriever and evaluates which chunks are returned as the answer. The higher MRR and Hit rate numbers represent the retriever’s ability to identify the chunk with the correct answer. ```python from llama_index.evaluation import RetrieverEvaluator retriever = vector_index.as_retriever(similarity_top_k=2) retriever_evaluator = RetrieverEvaluator.from_metric_names( ["mrr", "hit_rate"], retriever=retriever ) # Evaluate eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset) def display_results(name, eval_results): """Display results from evaluate.""" metric_dicts = [] for eval_result in eval_results: metric_dict = eval_result.metric_vals_dict metric_dicts.append(metric_dict) full_df = pd.DataFrame(metric_dicts) hit_rate = full_df["hit_rate"].mean() mrr = full_df["mrr"].mean() metric_df = pd.DataFrame( {"Retriever Name":


RAG - Metrics & Evaluation

# **Community-Based Evaluation Tools**
## **The Custom RAG Pipeline Evaluation**
[name], "Hit Rate": [hit_rate], "MRR": [mrr]} ) return metric_df display_results("OpenAI Embedding Retriever", eval_results) ``` ```bash | | Retriever Name | Hit Rate | MRR | | ---- | -------------------------- | -------- | --------- | | 0 | OpenAI Embedding Retriever | 0.884615 | 0.730769 | ``` We can now enhance our analysis of the application's performance by including additional metrics like faithfulness and relevancy. To achieve this, we utilize a subset of the generated Q&A dataset and create instances of both GPT-3.5 and GPT-4. It's advisable to employ a more advanced model like GPT-4 for evaluation purposes while using the less expensive model for the generation process. ```python # gpt-3.5-turbo gpt35 = OpenAI(temperature=0, model="gpt-3.5-turbo") service_context_gpt35 = ServiceContext.from_defaults(llm=gpt35) # gpt-4 gpt4 = OpenAI(temperature=0, model="gpt-4") service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4) vector_index = VectorStoreIndex(nodes, service_context = service_context_gpt35) query_engine = vector_index.as_query_engine() eval_query = queries[10] response_vector = query_engine.query(eval_query) print( "> eval_query: ", eval_query ) print( "> response_vector:", response_vector ) ``` ``` > eval_query: How did the colonies respond to the declaration of war by the dark forces, and what measures did they take to protect their knowledge and technology? > response_vector: The colonies did not fight back against the dark forces when they declared war. Instead, they sent most of their people into hiding in order to rebuild the colonies later. They also destroyed everything to ensure that their knowledge and technology would not fall into the hands of the dark forces. Additionally, Lemuria and Atlantis were destroyed by their inhabitants to prevent the misuse of their knowledge and technology by the dark forces. ``` Now, we can establish the evaluator classes responsible for measuring each metric. We'll then use a sample response to determine if it meets the test criteria. ```python from llama_index.evaluation import RelevancyEvaluator from llama_index.evaluation import FaithfulnessEvaluator relevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4) faithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4) # Compute faithfulness evaluation eval_result = faithfulness_gpt4.evaluate_response(response=response_vector) # check passing parameter in eval_result if it passed the evaluation. print( eval_result.passing ) # Relevancy evaluation eval_result = relevancy_gpt4.evaluate_response( query=eval_query, response=response_vector ) # You can check passing parameter in eval_result if it passed the evaluation. print( eval_result.passing ) ``` ``` True True ``` We must perform a for-loop to feed each sample from the evaluation dataset and get the appropriate results. In this situation, we can use the LlamaIndex `BatchEvalRunner` class, which runs the evaluation process in batches and concurrently. It means the evaluation can be done faster. ```python #Batch Evaluator: #BatchEvalRunner to compute multiple evaluations in batch wise manner. from llama_index.evaluation import BatchEvalRunner # Let's pick top 10 queries to do evaluation batch_eval_queries = queries[:10] # Initiate BatchEvalRunner to compute FaithFulness and Relevancy Evaluation. runner = BatchEvalRunner( {"faithfulness": faithfulness_gpt4, "relevancy": relevancy_gpt4}, workers=8, ) # Compute evaluation eval_results = await runner.aevaluate_queries( query_engine, queries=batch_eval_queries ) # get faithfulness score faithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness']) # get relevancy score relevancy_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['relevancy']) print( "> faithfulness_score", faithfulness_score ) print( "> relevancy_score", relevancy_score ) ``` ``` > faithfulness_score 1.0 > relevancy_score 1.0 ``` The batch processing method helps in quickly assessing the system’s performance over a range of different queries. A faithfulness score of 1.0 signifies that the generated answers contain no hallucinations and are entirely based on retrieved context. Additionally, the Relevance score of 1.0 suggests that the answers generated consistently align with the retrieved context and the queries.


RAG - Metrics & Evaluation

# Conclusion
This lesson has guided us through constructing and assessing an RAG pipeline using LlamaIndex, concentrating mainly on evaluating the retrieval system and the responses generated within the pipeline. Assessing LLMs and chatbots presents a challenge due to the subjective nature of their outputs. Perceptions of what constitutes a great response can vary significantly from one person to another. Therefore, examining various facets of an RAG application and evaluating each aspect individually based on specific metrics is prudent. --- >> [Notebook](https://colab.research.google.com/drive/1XcCcoUi5ZbUNmTCLYXL51zeD-6mXON5T?usp=sharing). --- ### RESOURCES: - [Response Evaluation](https://docs.llamaindex.ai/en/latest/module_guides/evaluating/usage_pattern.html) - [Retrieval](https://docs.llamaindex.ai/en/latest/module_guides/evaluating/usage_pattern_retrieval.html) Evaluation - **openai-cookbook-eval** [](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb) - **llamaindex** [Evaluating - LlamaIndex 🦙 0.9.15](https://docs.llamaindex.ai/en/stable/module_guides/evaluating/root.html) - **golden-dataset** [](https://github.com/microsoft/promptflow-resource-hub/blob/main/sample_gallery/golden_dataset/copilot-golden-dataset-creation-guidance.md) - **RAGAS** [](https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/llamaindex.ipynb) - `RagEvaluatorPack` [Downloading a LlamaDataset from LlamaHub - LlamaIndex 🦙 0.9.15.post2](https://docs.llamaindex.ai/en/stable/examples/llama_dataset/downloading_llama_datasets.html)


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
[https://lh7-us.googleusercontent.com/CMrM62nvjTbLm41Dq5ILsc9PhkQgm7QyKmnpMinLQeN7kh7kUQy4Z5J6oC2cV-_Vt3JwXUmgoBbD2va6UqhPw5ZkFub75tc6awr56kqbP_0ztvpUq0GVLqjUz-CJ8K0q9SwAwK6pWE3_b0wdnHOGMQU](https://lh7-us.googleusercontent.com/CMrM62nvjTbLm41Dq5ILsc9PhkQgm7QyKmnpMinLQeN7kh7kUQy4Z5J6oC2cV-_Vt3JwXUmgoBbD2va6UqhPw5ZkFub75tc6awr56kqbP_0ztvpUq0GVLqjUz-CJ8K0q9SwAwK6pWE3_b0wdnHOGMQU) **Abstract** This technical report describes the development and deployment of a Retrieval Augmented Generation (RAG) based AI Tutor to provide scalable, personalized support for students enrolled in the Gen AI 360 online courses. The solution leverages Activeloop’s database for AI, 4th Gen Intel® Xeon® processors, and Intel® oneAPI Math Kernel Library (oneMKL). The RAG AI Tutor project showcases the capabilities of Intel's 4th Gen Xeon® processors and oneAPI Math Kernel Library (oneMKL) in driving high-performance computing for AI applications. Intel's technology enables real-time AI interactions and efficient cosine similarity computations, crucial for the RAG system's embedding retrieval, demonstrating Intel® hardware and software's synergy in advanced AI tasks.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Introduction**
The Gen AI 360, facilitated by Activeloop, Towards AI, & Intel Disruptor Initiative, faces challenges in scaling student support for its extensive online courses. A RAG-based AI Tutor has been developed to augment human tutor efforts by providing immediate, accurate responses to student inquiries.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Problem Statement**
Scalability issues arise when providing support for thousands of students. There is a need for an AI-based solution that can deliver individualized support for routine queries while ensuring knowledge relevance accuracy and reducing hallucinations.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **The RAG AI Tutor Solution**
A partnership between Towards AI, Activeloop, and Intel has resulted in an RAG AI Tutor that supports tens of thousands of students. ### **Knowledge Base** - Content: Access to a library of ~4,000 AI tutorials and articles, including updated information on LLMs and articles from the GenAI360 course. - Storage: 21,000 chunks of data (~500 words each) transformed into embedding vectors. ### **Relevancy, Accuracy, and Hallucinations** - Technique: Utilizes RAG and Deep Memory by Activeloop. - Implementation: Strictly answer questions from the embedded knowledge base to reduce misinformation. ### **Cost, Latency, and Scalability** - Hardware: Utilization of 4th Gen Intel® Xeon® processors. - Optimization: Balancing the number of sources and data chunks to enhance speed and reduce costs.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Technology Stack**
- LLMs: OpenAI’s ada-002 embeddings and GPT-Turbo 16k LLM. - Intel Inside: 4th Gen Intel® Xeon® processors for low-latency LLM inference with Intel® AVX-512, Intel® oneAPI Math Kernel Library (oneMKL). - Data Storage: Activeloop Deep Lake is a database for AI that hosts embeddings and metadata and enables deep memory for better retrieval.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Technical Achievements**
- Compute Efficiency: A 22.93% increase in cosine similarity computations was achieved with 4th Gen Intel® Xeon® compared to 3rd Gen Xeon® Processors. - Accuracy Improvement: The implementation of Deep Memory noted a 20% increase in recall@10 for embedding retrieval. [https://lh7-us.googleusercontent.com/JgQAdnzCc94GLaADelYAxDkJ-K3SBG_vYq-J9OjosIFwtFHbBht4pdT5LSFBVGGtIFa8zz2YSE1zrB9rvn2dBDBPtpUBMEPbqTSSJCAM-oKGgHxkzJ-6XnLp06MZERn_C-J9l2pJDUGiKWhKnpLaflk](https://lh7-us.googleusercontent.com/JgQAdnzCc94GLaADelYAxDkJ-K3SBG_vYq-J9OjosIFwtFHbBht4pdT5LSFBVGGtIFa8zz2YSE1zrB9rvn2dBDBPtpUBMEPbqTSSJCAM-oKGgHxkzJ-6XnLp06MZERn_C-J9l2pJDUGiKWhKnpLaflk)


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Implementation in Educational Context**
The RAG AI Tutor is integrated with Gen AI 360 course offerings, providing in-depth lessons and tutorials, practical coding projects, real-time AI assistance for technical Q&A, and access to updated AI-related content and community support.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Product Outcome**
- Response Time: An AI Tutor delivers answers with a 0.0243-second response time. - Accuracy: The utilization of pre-filtered knowledge sources and the Deep Memory by Activeloop dramatically reduces the risk of hallucinations. - Efficiency: Demonstrated speed improvements and accurate query handling.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Future Steps**
- Deployment: Introduction of a Discord bot for real-time student interaction. - Optimization: Continual refinement of the data sources for the RAG model. - Experimentation: Potential fine-tuning of open-source LLMs and new embedding models.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Key Learnings**
- Integrated AI solutions are vital for scalable student support in online education. - The selection of hardware and database management systems is crucial for the performance and cost-effectiveness of AI applications.


# Creating a RAG AI Tutor Bot with TowardsAI & Activeloop
## **Conclusion**
The collaborative development of the RAG AI Tutor represents an advancement in AI-assisted education, focusing on scalability, accuracy, and speed. Implementing efficient hardware and vector databases is critical to the success of AI applications in education. **Disclaimers** Performance varies by use, configuration, and other factors. Learn more on the/Performance Index site. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for configuration details. No product or component can be absolutely secure. Your costs and results may vary. For workloads and configurations, visit 4th Gen Xeon® Scalable processors at www.intel.com/processorclaims. Results may vary. Intel technologies may require enabled hardware, software or service activation. Intel does not control or audit third-party data. You should consult other sources to evaluate accuracy. Intel® technologies may require enabled hardware, software, or service activation. © Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.


LangSmith Introduction

# LangSmith Introduction
In this lesson, we will recap LangChain components, review its essential concepts, and discuss how to use the newly introduced LangSmith platform. Additionally, we will create a basic Large Language Model application to understand its capabilities better.


LangSmith Introduction

# LangChain Recap
LangChain is a specialized framework designed for building LLM-powered applications. It streamlines the development of intelligent, responsive LLMs and Libraries for handling chains and agents with integrated components. It also offers [Templates](https://python.langchain.com/docs/templates) for deployable task-specific architectures and [LangSmith](https://python.langchain.com/docs/langsmith) for debugging in a testing environment. The key features of LangChain, like Models, Vector Stores, Chains, etc., have been explained in detail in the previous lesson. <aside> 💡 While LangChain is suitable for prototyping, LangSmith enables an environment for debugging, testing, and optimizing LLM apps. </aside>


LangSmith Introduction

# **LangChain Hub**
LangChain Hub is a centralized repository for community-sourced prompts tailored to various use cases like classification or summarization. The Hub supports both public contributions and private organizational use, fostering a collaborative development environment. The platform's version control system enables users to track prompt modifications and maintain consistency across applications. The Hub offers features like **Prompt Exploration**, ideal for fresh interactions with language models or specific prompts to achieve particular objectives. It also simplifies the process of finding and utilizing effective prompts for various models. Additionally, the user can share, modify, and track prompt versions with **Prompt Versioning**. It allows for the easy management of different versions of prompts, a highly relevant feature in real-world projects where reverting to earlier versions may be necessary. The user-friendly interface of the Hub allows for prompt testing, customization, and iteration in a playground environment. <aside> 💡 Discover, version control, and experiment with different prompts for LangChain and LLMs directly in your browser: [docs.smith.langchain.com](https://docs.smith.langchain.com/hub/faq). </aside>


LangSmith Introduction

# LangSmith
[LangSmith](https://www.langchain.com/langsmith) provides an environment for evaluating and monitoring the quality of LLM outputs. An integral part of its functionality includes metadata monitoring, token usage, and execution time, which are crucial for resource management. This platform facilitates the refinement of new chains and tools, potentially enhancing their efficiency and performance. Users can create diverse testing environments tailored to specific needs, enabling thorough evaluation under various conditions. Additionally, the service provides visualization tools that can aid in identifying response patterns and trends, thereby supporting a deeper understanding and assessment of performance. Lastly, the platform supports tracing the runs associated with an active instance and testing and evaluating any prompts or answers generated. LangSmith is designed with user-friendliness in mind. The platform offers a range of tutorials and documentation to help you get started. The setup for LangChain requires installing the necessary libraries and configuring the required environment variables, which we will cover in the following section. For certain functionalities like tracing, you need to have a LangSmith account. Please follow the steps outlined below to set up a new account. - Head over to the [LangSmith](https://www.langchain.com/langsmith) website and sign up for an account. You can use various supported login methods. - Once your account is set up, go to the settings page. Here, you'll find the option to create an API key. - Click the 'Generate API Key' button to receive your API key.


LangSmith Introduction

# LangSmith
## Versioning
You can commit a prompt after implementing and debugging your chain. Add this prompt under your handle's namespace to view it in the Hub. ```python from langchain import hub from langchain.prompts.chat import ChatPromptTemplate prompt = ChatPromptTemplate.from_template("tell me a joke about {topic}") handle = "<YOUR_USERNAME>" hub.push(f"{handle}/rag", prompt) ``` During evaluation, if you come up with a better idea after trying the prompt, you can push the updated prompt to the same key to "commit" a new version of the prompt. For instance, let's add a system message to the prompt. ```python # You may try making other changes and saving them in a new commit. from langchain import schema prompt.messages.insert(0, schema.SystemMessage( content="You are a precise, autoregressive question-answering system." ) ) ``` With the saved changes, we can analyze how the change reflects the model performance. The newest version of the prompt is saved as the latest version. ```python # Pushing to the same prompt "repo" will create a new commit hub.push(f"{handle}/rag-prompt", prompt) ```


LangSmith Introduction

# LangSmith
## Tracing
LangSmith allows users to review the inputs and outputs of each element in the chain by simplifying the process of logging runs for your LLM applications. This feature is useful when debugging your application or understanding the behavior of specific components. The following section will explore the optional environment variables that enable the tracing feature. For more information, you can [visit the documentation](https://docs.smith.langchain.com/tracing#log-runs).


LangSmith Introduction

# LangSmith
## **Serving (LangServe)**
[LangServe](https://www.langchain.com/langserve) helps developers deploy LangChain-powered applications and chains as a REST API. It is integrated with FastAPI, which makes the process of creating API endpoints easy and accessible. It is possible to quickly deploy applications by using the `langserve` package. The deployment process is out of the scope of this course. However, you can learn more about the process from the [Github repository](https://github.com/langchain-ai/langserve?ref=blog.langchain.dev).


LangSmith Introduction

# QuestionAnswering Chain & LangChain Hub
The next steps are loading data from a webpage, splitting it into smaller chunks, transforming them into embeddings, storing them on the Deep Lake vector store, and utilizing the prompt templates from the [LangSmith Hub](https://smith.langchain.com/hub). Before exploring the code, installing the essential libraries from the Python package (pip) manager is necessary. ```bash pip install -q langchain==0.0.346 openai==1.3.7 tiktoken==0.5.2 cohere==4.37 deeplake==3.8.11 langchainhub==0.1.14 ``` The next step is to set the API keys in the environment for OpenAI, utilized in the embedding generation process, and the Activeloop key, required for storing data in the cloud. ```python import os os.environ["OPENAI_API_KEY"] = "<YOUR_OPENAI_API_KEY>" os.environ["ACTIVELOOP_TOKEN"] = "<YOUR_ACTIVELOOP_API_KEY>" ``` You can optionally use the following environment variables to keep track of the runs in the LangSmith dashboard under the projects section. ```python os.environ["LANGCHAIN_TRACING_V2"]=True os.environ["LANGCHAIN_ENDPOINT"]="https://api.smith.langchain.com" os.environ["LANGCHAIN_API_KEY"]="<YOUR_LANGSMITH_API_KEY>" os.environ["LANGCHAIN_PROJECT"]="langsmith-intro" # if not specified, defaults to "default" ``` Now, we can read the content of a webpage using the `WebBaseLoader` class. It will return a single instance of the `Document` class containing all the textual information from the mentioned address. Subsequently, the lengthy text is divided into smaller segments of 500 characters each, with no overlap, resulting in 130 chunks. ```python from langchain.document_loaders import WebBaseLoader from langchain.text_splitter import RecursiveCharacterTextSplitter # Loading loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/") data = loader.load() print(len(data)) # Split text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0) all_splits = text_splitter.split_documents(data) print(len(all_splits)) ``` ``` 1 130 ``` Chunks can be saved to the Deep Lake vector store through LangChain integration. The DeepLake class handles converting texts into embeddings via OpenAI's API and then stores these results in the cloud. The dataset can be loaded from the GenAI360 course organization, or you can use your organization name (which defaults to your username) to create the dataset. Note that this task incurs the associated costs of using OpenAI endpoints. ```python from langchain.embeddings import OpenAIEmbeddings from langchain.vectorstores import DeepLake vectorstore = DeepLake.from_documents( all_splits, dataset_path="hub://genai360/langsmith_intro", embedding=OpenAIEmbeddings(), overwrite=False) ``` ```python Your Deep Lake dataset has been successfully created! Creating 130 embeddings in 1 batches of size 130:: 100%|██████████| 1/1 [00:05<00:00, 5.81s/it] dataset (path='hub://genai360/langsmith_intro', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (130, 1) str None metadata json (130, 1) str None embedding embedding (130, 1536) float32 None id text (130, 1) str None ``` Once the data is processed, we can retrieve a prompt from the LangChain hub, which provides a `ChatPromptTemplate` instance. This eliminates the need for designing a prompt through trial and error, allowing us to build upon already tested implementations. The following code tagged a specific prompt version so future changes would not impact the active deployment version. ```python from langchain import hub prompt = hub.pull("rlm/rag-prompt:50442af1") print(prompt) ``` ``` ChatPromptTemplate(input_variables=['context', 'question'], messages=[HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context', 'question'], template="You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\nQuestion: {question} \nContext: {context} \nAnswer:"))]) ``` Finally, we can employ the `RetrievalQA` chain to fetch related documents from the database and utilize the `ChatOpenAI` model to use these documents to generate our final response. ```python # LLM from langchain.chains import RetrievalQA from langchain.chat_models import ChatOpenAI llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0) # RetrievalQA qa_chain = RetrievalQA.from_chain_type( llm, retriever=vectorstore.as_retriever(), chain_type_kwargs={"prompt": prompt} ) question = "What are the approaches to Task Decomposition?" result = qa_chain({"query": question}) result["result"] ``` ``` The approaches to task decomposition include using LLM with simple prompting, task-specific instructions, and human inputs. ``` Prompt versioning supports ongoing experimentation and collaboration, effectively preventing the accidental deployment of chain components that haven't been sufficiently validated.


LangSmith Introduction

# Conclusion
In this lesson, we discussed how to use the LangChain [Hub](https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fsmith.langchain.com%2Fhub%3Fpage%3D1) to store and share prompts for a retrieval QA chain. The Hub is a centralized location to manage, version, and share prompts. LangSmith excels in diagnosing errors, comparing prompt effectiveness, assessing output quality, and tracking key metadata like token usage and execution time for optimizing LLM applications. The platform also provides a detailed analysis of how different prompts affected the LLM performance. The intuitive UI and the valuable insights the platform offers make the iterative process of refining LLMs more transparent and manageable. It's evident that the LangSmith platform, even in its beta phase, has the potential to be a significant tool for developers aiming to leverage the full potential of LLMs. LangSmith provides an immediate functionality to sift through your runs and presents metrics. These metrics are essential for quickly assessing latency and the total token count throughout your application. >> [Notebook](https://colab.research.google.com/drive/1scwUUFOqgjqCE0CRFe19Llv86GcBt7Sh?usp=sharing). --- ### **RESOURCES**: - hub-examples: LangSmith cookbook [](https://github.com/langchain-ai/langsmith-cookbook/blob/main/hub-examples/retrieval-qa-chain/retrieval-qa.ipynb) - the art of LangSmith article [The Art of LangSmithing](https://betterprogramming.pub/the-art-of-langsmithing-42dcd191a220)


Fine-tuning vs RAG; Introduction to Activeloop’s D

# Introduction
In this lesson, we will explore optimization techniques that maximize large language model performance. We will learn about the appropriate use of prompt engineering, retrieval augmented generation (RAG), and fine-tuning, distinguishing how each method contributes and their specific challenges. A significant portion of the lesson will be dedicated to addressing the limitations of RAG systems in real-world applications. These mainly include maintaining high retrieval accuracy and ensuring accurate responses from LLMs. Much of our discussion will include [Activeloop's Deep Memory](https://docs.activeloop.ai/performance-features/deep-memory), a technique designed to augment the retrieval precision of embeddings for user queries. We will also perform a detailed comparison of empirical data, analyzing the differences in retrieval recall rates between systems employing Deep Memory and those that do not.


# Overview of RAG E**nhancement Techniques**
Expanding on the discussion surrounding fine-tuning, retrieval-augmented generation, and prompt engineering, it's essential to understand each approach's distinct strengths, weaknesses, and most suitable applications.


# Overview of RAG E**nhancement Techniques**
## Prompt engineering
Prompt engineering is often the first step in enhancing the performance of an LLM for specific tasks. This approach alone can be sufficient, especially for simpler or well-defined tasks. Techniques like [few-shot prompting](https://www.promptingguide.ai/techniques/fewshot) can notably improve task performance. This method involves providing small task-specific examples to guide the LLM. [Chain of Thought (CoT)](https://arxiv.org/abs/2201.11903) prompting can also improve reasoning capabilities and encourage the model to generate more detailed responses. Combining Few-shot with RAG—using a tailored dataset of examples to retrieve the most relevant information for each query—can be more effective.


# Overview of RAG E**nhancement Techniques**
## **Fine-tuning**
Fine-tuning enhances LLM’s capabilities in the following areas: 1. Modifying the **structure** or **tone** of responses. 2. Teaching the model to follow complex instructions. For example, fine-tuning enables models to perform tasks like extracting JSON-formatted data from text, translating natural language into SQL queries, or adopting a specific writing style. Fine-tuning demands a large, high-quality, task-specific dataset for effective training. You can start with a small dataset and training to see if the method works for your task. Fine-tuning is less effective in adapting to new, rapidly changing data or unfamiliar queries beyond the training dataset. It's also not the best choice for incorporating new information into the model. Alternative methods, such as Retrieval-Augmented Generation, are more suitable. ![From [A Survey of Techniques for Maximizing LLM Performance](https://www.youtube.com/watch?v=ahnGLM-RC1Y&t=344s)](Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20b072ea133ad2469693dcd5e53fd4c87c/Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20fc5328546cb14262a1afada413848a28/Untitled.png) From [A Survey of Techniques for Maximizing LLM Performance](https://www.youtube.com/watch?v=ahnGLM-RC1Y&t=344s)


# Overview of RAG E**nhancement Techniques**
## **Retrieval-Augmented Generation**
RAG specializes in incorporating **external knowledge**, enabling the model to access current and varied information. **Real-Time Updates**: It is more adept at dealing with evolving datasets and can provide more up-to-date responses. **Complexity in Integration**: Setting up a RAG system is more complex than basic prompting, requiring extra components like a Vector Database and retrieval algorithms. **Data Management**: Managing and updating the external data sources is crucial for maintaining the accuracy and relevance of its outputs. **Retrieval accuracy:** Ensuring precise embedding retrieval is crucial in RAG systems to guarantee reliable and comprehensive responses to user queries. For that, we will demonstrate how Activeloop’s Deep Memory method can greatly increase the recall of embedding retrieval.


# Overview of RAG E**nhancement Techniques**
## RAG + Fine-tuning
Fine-tuning and RAGs are not mutually exclusive techniques. Fine-tuning brings the advantage of customizing models for a specific style or format, which can be useful when using LLMs for specific domains such as medical, financial, or legal, requiring a highly specialized tone of writing. When combined with RAG, the model becomes adept in its specialized area and gains access to a vast range of external information. The resulting model provides accurate responses in the niche area. Implementing these two methods can demand considerable resources for setup and ongoing upkeep. It involves multiple training runs of fine-tuning with the data handling requirements inherent to RAG. ![From [A Survey of Techniques for Maximizing LLM Performance](https://www.youtube.com/watch?v=ahnGLM-RC1Y&t=344s) ](Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20b072ea133ad2469693dcd5e53fd4c87c/Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20fc5328546cb14262a1afada413848a28/Screenshot_2023-12-13_at_3.43.57_PM.png) From [A Survey of Techniques for Maximizing LLM Performance](https://www.youtube.com/watch?v=ahnGLM-RC1Y&t=344s) [data:image/svg+xml;charset=utf-8,%3Csvg height='1254' width='2398' xmlns='http://www.w3.org/2000/svg' version='1.1'%3E%3C/svg%3E](data:image/svg+xml;charset=utf-8,%3Csvg height='1254' width='2398' xmlns='http://www.w3.org/2000/svg' version='1.1'%3E%3C/svg%3E)


# Enhanced RAG with Deep Memory
[Deep Memory](https://docs.activeloop.ai/performance-features/deep-memory) is a method developed by Activeloop to boost the accuracy of embedding retrieval for RAG systems integrated into the Deep Lake vector store database. Central to its functionality is an embedding transformation process. Deep Memory trains a model that transforms embeddings into a space optimized for your use case. This reconfiguration significantly improves vector search accuracy. Deep Memory is effective where query reformulation, query transformation, or document re-ranking might cause latency and increased token usage. It boosts retrieval capabilities without negatively impacting the system's performance. The figure below shows the recall performance for different algorithms compared to Deep Memory. **Recall@1**: This measures whether the top result (i.e., the first result) returned by the retrieval system is relevant to the query. **Recall@10**: This metric assesses whether the relevant document is within the top 10 results returned by the retrieval system. ![From [Activeloop’s Deep Memory documentation](https://docs.activeloop.ai/performance-features/deep-memory)](Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20b072ea133ad2469693dcd5e53fd4c87c/Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20fc5328546cb14262a1afada413848a28/Untitled%201.png) From [Activeloop’s Deep Memory documentation](https://docs.activeloop.ai/performance-features/deep-memory) **Comparison to Lexical search** [BM25](https://en.wikipedia.org/wiki/Okapi_BM25) is considered a state-of-the-art approach for "lexical search," based on the explicit presence of words (or lexicons) from the query in the documents. It's particularly effective for applications where the relevance of documents depends heavily on the presence of specific terms, such as in traditional search engines. However, BM25 does not account for the semantic relationships between words, where more advanced techniques like vector search with neural embeddings and semantic search come into play.


# Enhanced RAG with Deep Memory
## Overview of Deep Memory
![[From Activeloop Docs - How Deep Memory Works](https://docs.activeloop.ai/performance-features/deep-memory/how-it-works)](Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20b072ea133ad2469693dcd5e53fd4c87c/Fine-tuning%20vs%20RAG;%20Introduction%20to%20Activeloop%E2%80%99s%20D%20fc5328546cb14262a1afada413848a28/deepmemory_diagram.png) [From Activeloop Docs - How Deep Memory Works](https://docs.activeloop.ai/performance-features/deep-memory/how-it-works) In the figure above, we see the Inference and Training workflow: 1. **Embeddings**: Vector representation of a text sentence or set of words. We can create them using embedding models such as OpenAI’s [text-embedding-ada-002](https://platform.openai.com/docs/guides/embeddings/embedding-models) or open-source models. 2. **Deep Memory Training**: A dataset of **query and context pairs** trains the Deep Memory model. This training process runs on the Deep Lake service, which provides the computational resources and infrastructure for handling the training. 3. **Deep Memory Inference**: The model enters the inference phase after training, which transforms query embeddings. We can use the [Tensor Query Language (TQL)](https://docs.activeloop.ai/performance-features/querying-datasets) when running an inference/querying in the Vector Store. 4. **Transformed Embeddings**: The result of the inference process is a set of transformed embeddings optimized for a specific use case. This optimization means that the embeddings are now in a more conducive space for returning accurate results. 5. **Vector Search**: These optimized embeddings are used in a vector search, utilizing standard similarity search techniques (e.g., cosine similarity). The vector search is retrieving information, leveraging the refined embeddings to find and retrieve the most relevant data points for a given query.


# Enhanced RAG with Deep Memory
## Step by Step - Training a **Deep Memory Model**
Moving forward in our lesson, let's implement Deep Memory within our experimental workflow to see firsthand how it impacts retrieval recall. You can follow along with this [Colab notebook.](https://colab.research.google.com/drive/1dgNwwCuA4zqtQDRuyzqd53PbG3PlRHul#scrollTo=lXJYCiwNPbOL) 1. Install the required libraries ```python !pip3 install deeplake langchain openai tiktoken llama-index ``` 1. Set your ACTIVELOOP_TOKEN and OPENAI_API_KEY ```python import os, getpass os.environ['ACTIVELOOP_TOKEN'] = getpass.getpass() ``` ```python os.environ['OPENAI_API_KEY'] = getpass.getpass() ``` 1. Download the data or use your own. Here, we download a [text file](https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt) hosted on GitHub. ```python !mkdir -p 'data/paul_graham/' !curl 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -o 'data/paul_graham/paul_graham_essay.txt' ``` 1. Create the Llama-index nodes/chunks ```python from llama_index.node_parser import SimpleNodeParser from llama_index import SimpleDirectoryReader documents = SimpleDirectoryReader("./data/paul_graham/").load_data() node_parser = SimpleNodeParser.from_defaults(chunk_size=512) nodes = node_parser.get_nodes_from_documents(documents) # By default, the node/chunks ids are set to random uuids. To ensure same id's per run, we manually set them. for idx, node in enumerate(nodes): node.id_ = f"node_{idx}" print(f"Number of Documents: {len(documents)}") print(f"Number of nodes: {len(nodes)} with the current chunk size of {node_parser.chunk_size}") ``` ``` Number of Documents: 1 Number of nodes: 58 with the current chunk size of 512 ``` 1. Create a local Deep Lake vector store ```python from llama_index import VectorStoreIndex, ServiceContext, StorageContext from llama_index.vector_stores import DeepLakeVectorStore from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.llms import OpenAI # Create a DeepLakeVectorStore locally to store the vectors dataset_path = "./data/paul_graham/deep_lake_db" vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=True, exec_option="compute_engine") # LLM that will answer questions with the retrieved context llm = OpenAI(model="gpt-3.5-turbo-1106") embed_model = OpenAIEmbedding() service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm,) storage_context = StorageContext.from_defaults(vector_store=vector_store) vector_index = VectorStoreIndex(nodes, service_context=service_context, storage_context=storage_context, show_progress=True) ``` ``` Uploading data to deeplake dataset. 100%|██████████| 58/58 [00:00<00:00, 274.94it/s]Dataset(path='./data/paul_graham/deep_lake_db', tensors=['text', 'metadata', 'embedding', 'id']) tensor htype shape dtype compression ------- ------- ------- ------- ------- text text (58, 1) str None metadata json (58, 1) str None embedding embedding (58, 1536) float32 None id text (58, 1) str None ``` 1. Now, let's upload the local Vectore Store to Activeloop's platform and convert it into a managed database. ```python import deeplake local = "./data/paul_graham/deep_lake_db" hub_path = "hub://genai360/LlamaIndex_paulgraham_essay" hub_managed_path = "hub://genai360/LlamaIndex_paulgraham_essay_managed" # First upload our local vector store deeplake.deepcopy(local, hub_path, overwrite=True) # Create a managed vector store under a different name deeplake.deepcopy(hub_path, hub_managed_path, overwrite=True, runtime={"tensor_db": True}) ``` 1. Instantiate a Vector Store with the managed dataset that we just created. ```python db = DeepLakeVectorStore(dataset_path=hub_managed_path, overwrite=False, exec_option="compute_engine", read_only=True,) ``` Now, let’s generate a dataset of Queries and Documents 1. Fetching our `docs` and `ids` from the vector store. ```python # Fetch dataset docs and ids docs = db.vectorstore.dataset.text.data(fetch_chunks=True, aslist=True)['value'] ids = db.vectorstore.dataset.id.data(fetch_chunks=True, aslist=True)['value'] print(len(docs)) ``` 1. Generating a synthetic training dataset. We need labeled data (`query` and `document_id` pairs) to train a Deep Memory model. Sometimes, it can be difficult to get labeled data when you are starting from scratch. This tutorial generates queries/questions using `gpt-3.5-turbo` from our existing documents. ```python from openai import OpenAI client = OpenAI() def generate_question(text): try: response = client.chat.completions.create( model="gpt-3.5-turbo-1106", messages=[ {"role": "system", "content": "You are a world class expert for generating questions based on provided context. \ You make sure the question can be answered by the text."}, { "role": "user", "content": text, }, ], ) return response.choices[0].message.content except: question_string = "No question generated" return question_string ``` ```python import random from tqdm import tqdm def generate_queries(docs: list[str], ids: list[str], n: int): questions = [] relevances = [] pbar = tqdm(total=n) while len(questions) < n: # 1. randomly draw a piece of text and relevance id r = random.randint(0, len(docs)-1) text, label = docs[r], ids[r] # 2. generate queries and assign and relevance id generated_qs = [generate_question(text)] if generated_qs == ["No question generated"]: print("No question generated") continue questions.extend(generated_qs) relevances.extend([[(label, 1)] for _ in generated_qs]) pbar.update(len(generated_qs)) return questions[:n], relevances[:n] ``` 5.1 Launch