# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Limitations**
The agent framework is a novel and fascinating methodology to build complex applications. Nevertheless, it comes with its own set of challenges. When thinking about deploying agents in production, you should consider these additional steps: - Make individual tools as reliable as possible. In this project, most tools are simply wrappers of functions. Those functions assume that inputs behave fairly well, and otherwise fail. This can pose a problem if an agent does not correctly format the inputs as the tool expect them. One solution to this would be to add input checks inside the function and return and error with what went wrong prompting the agent to try again. - Monitoring, monitoring, monitoring. Agents can be quite complicated to debug. Using a framework to quickly log all calls to the underlying LLM, i.e. `gpt-4`, can greatly improve the building experience and allow you to identify issues with your agent quickly. In this regard, [LangSmith](https://www.langchain.com/langsmith) looks very promising. - Finally, if your agent is customer-facing, you may want to add guardrails to guarantee that the LLM remains on topic. Otherwise, your chat might be used for third-party purposes without your consent. Don't trust me? Look at what happened to Chevrolet. ![Untitled](Building%20a%20Smart%20Shopping%20Assistant%20with%20DeepLake%20%20f3de3ee15abe4f4eaf760fa91fb13e0a/Untitled%205.png)


# Building a Smart Shopping Assistant with DeepLake and LlamaIndex
## **Conclusion**
As we've journeyed through the intricate process of developing an AI-powered shopping assistant, the roles of Deep Lake and LlamaIndex have proven pivotal. From the versatile data handling in Deep Lake's vector database to the adaptive LLM agents orchestrated by LlamaIndex, these technologies have showcased their robust capabilities and the potential for innovation in the AI space. Deep Lake has demonstrated its capacity to seamlessly manage and retrieve complex structures, enabling efficient and precise item matchings. Its architecture has been the backbone of the shopping assistant, proving that even complex data interactions can be handled with elegance and speed. LlamaIndex, on the other hand, has been instrumental in empowering the shopping assistant with natural language processing and decision-making abilities. Its framework has enabled the LLM agents to interpret, engage, and personalize the shopping experience, charting new courses in user-AI interaction. Looking beyond the shopping assistant itself, the potential uses for these technologies span myriad domains. The flexibility and power of Deep Lake and LlamaIndex could drive innovation in fields ranging from healthcare to finance and from educational tools to creative industries. Your insights and feedback are crucial as we continue to navigate and expand the frontiers of artificial intelligence. We are particularly eager to hear your views on the innovative applications of vector databases and LLM frameworks. Additionally, your suggestions for topics you'd like us to delve into in future segments are highly appreciated. 🙂 Diego & Lucas from [Tryolabs](https://tryolabs.com/) ---


# How We Finetuned a Large Language Model & Built A Retrieval Engine to Search & Generate Patents
[https://lh7-us.googleusercontent.com/uama5dKPeETNBxyacA_s3TxBQUR6V9qSeSNmrsKiTtu4cX-tOWou-0iyGB-4Rn8RQGm1h7nT3sR6JfeaL2yCgQQszLeE47ho1CyyyewBCPdN3HJIf3xO30Ln3KFohE-kUHpOwYLJ4CbJ7I2tZnE0Cyw](https://lh7-us.googleusercontent.com/uama5dKPeETNBxyacA_s3TxBQUR6V9qSeSNmrsKiTtu4cX-tOWou-0iyGB-4Rn8RQGm1h7nT3sR6JfeaL2yCgQQszLeE47ho1CyyyewBCPdN3HJIf3xO30Ln3KFohE-kUHpOwYLJ4CbJ7I2tZnE0Cyw) *The technical details behind how we built PatentPT* As research in the language modeling space advances, there need to be more accompanying practical guides on how to finetune and deploy large language models on a custom text corpus. Yet, in practice, finetuned LLMs (like [BloombergGPT](https://www.bloomberg.com/company/press/bloomberggpt-50-billion-parameter-llm-tuned-finance/)) and ensembles of these finetuned LLMs are precisely where the industry's future is heading - instead of a singular general LLM API to rule them all.


# **PatentPT Features**
If you have ever used [https://www.uspto.gov/patents](https://www.uspto.gov/patents), you may have been unimpressed with the rigidly structured search provided, likely running on Cobalt servers without any neural network execution. So, there was ample opportunity and demand from the legal industry for an LLM-enabled approach to leveraging and search/retrieval of the US patent corpus. Specifically, we wanted our PatentPT to have the following features: - Autocomplete - Patent search on abstract - Patent search on claims - Abstract generation - Claim generation - General chat To provide a simple, unified chat experience, we elected to have a meta agent sitting on top of these capabilities, routing the user to the PatentPTs capability. [https://lh7-us.googleusercontent.com/LHb6uNOeIsVmcevw785YK9-gsDa0KC2rkRDVHoub3Edr4rbHVrHXzD0Yme2GF5yL3aT-egyl3WbCEBusw1P5EZsM2LRMv35l579cWP4qe94UcssWnWxmV86-PCZpxAJXIy_grnMP5B31OKzuRPk1o-E](https://lh7-us.googleusercontent.com/LHb6uNOeIsVmcevw785YK9-gsDa0KC2rkRDVHoub3Edr4rbHVrHXzD0Yme2GF5yL3aT-egyl3WbCEBusw1P5EZsM2LRMv35l579cWP4qe94UcssWnWxmV86-PCZpxAJXIy_grnMP5B31OKzuRPk1o-E) *PatentPT features*


# **PatentPT Technical Architecture**
To stand up these APIs, we needed to create an ensemble of finetuned LLM models and search indices to provide the richest PatentPT experience. To accomplish this, we chose to first domain train a base LLM on the patent corpus of text, providing a base for our finetuning routines and our autocompletion API out of the box - this network also provides a custom featurizer for our search indices. Next, we finetuned LLMs for generation and chat off the base domain-trained LLM using PEFT techniques. Using our custom featurizer, we index patents on abstract and claim to create search indices to power our various APIs - hooking them up to the search APIs and providing context for generation and chat. [https://lh7-us.googleusercontent.com/gMsa_qNzAmRmVJo3GfI8chF7FIusiR7Kv2GbNMrXWs5LTHGDzIYtsOHJowbSPziwbWoh4K-chnP0Jgghaq6Vm5ApVC0AFi4Iyz6N1ksjvbhx6MN9j1XBYTwE1cPsqRDJCglB-CmUw_zxLhBN_M_j4ws](https://lh7-us.googleusercontent.com/gMsa_qNzAmRmVJo3GfI8chF7FIusiR7Kv2GbNMrXWs5LTHGDzIYtsOHJowbSPziwbWoh4K-chnP0Jgghaq6Vm5ApVC0AFi4Iyz6N1ksjvbhx6MN9j1XBYTwE1cPsqRDJCglB-CmUw_zxLhBN_M_j4ws) *PatentPT Technical Architecture*


# **PatentPT Dataset**
The USPTO dataset consists of over 8 million patents, each broken down into fields - title, classification, publication_date, abstract, description, claims , etc. ```python dict_keys(['bibliographic_information', 'source_file', 'abstract', 'citations', 'assignees', 'classifications', 'inventors', 'brief_summary', 'foreign_priority', 'other_citations', 'detailed_description', 'claim_information']) dict_keys(['bibliographic_information', 'source_file', 'abstract', 'citations', 'assignees', 'classifications', 'inventors', 'detailed_description', 'claim_information']) dict_keys(['bibliographic_information', 'source_file', 'abstract', 'citations', 'assignees', 'classifications', 'inventors', 'detailed_description', 'claim_information']) ``` The corpus of text in the USPTO dataset spans 40 billion words, filling a 350 GB text file, which, while short of the full size of the large pretraining LLM datasets, like [the stack](https://huggingface.co/datasets/bigcode/the-stack-dedup), it is enough to make a full finetuning pass on an LLM without losing the semantic richness as you would with a smaller domain training dataset. As is typical with a project like this, we spent much time preparing our dataset for training from the USPTO XML files, using this open-source [USPTO parser](https://github.com/USPTO) as a base.


# **Domain Training Patent GPT**
To domain train our base patent GPT LLM, we used the entire patent corpus of 40 billion words. For hardware, we chose Habana® Labs' (an Intel® company) first-generation [Intel Gaudi® AI deep learning processor](https://habana.ai/aws-ec2-dl1-instances/), an instance with 8 HPUs. Habana HPUs are competitive with the latest and greatest GPUs, particularly for training transformer models. We trained our model with a CLM objective using the Optimum library from Huggingface, which has excellent Habana API bindings for running training on HPU. Optimum Intel interfaces the HuggingFace Transformers and Diffusers libraries and Habana HPUs. It provides tools that facilitate effortless model loading, training, and inference on single- and multi-HPU settings for tasks such as text classification, question answering, or language modeling! For our dataloader, we used the [Deep Lake performant dataloader](https://docs.activeloop.ai/performance-features/performant-dataloader) to stream data loading into our model. Tokenizing the dataset alone ran for 18 hours on our 8 HPU machine, while training ran for 24 days, at which point validation loss had fully converged. We used our domain-trained LLM for the autocompletion API as is. And we used the domain-trained LLM as a base for our downstream finetuning.


# **Finetuning Generation Models**
The next step in our training routines was to finetune generation models for abstracts and claims lists. To do so, we constructed datasets of description by abstract and claims and fed those datasets through a generation objective of our LLM. Again, we loaded our dataset into Deep Lake for this process and took advantage of the [Deep Lake dataloader](https://docs.activeloop.ai/performance-features/performant-dataloader). During this process, we did not tune all of our LLMs model weights; instead, we used the [HuggingFace PEFT library](https://huggingface.co/docs/transformers/pad_truncation) to tune LORA weights for each objective.


# **Finetuning General Chat**
We used PEFT techniques to fine-tune the chat to keep the general patent knowledge learned from our domain training routine.


# **Creating Custom Featurizers**
We pulled a custom featurizer from our domain-trained open-source model to set up our search APIs. To do so, we pull out the representations from the last hidden layer. We found these features more robust in practice than general sentence embedders.


# **Standing up Search Indices**
With our custom featurizer, we indexed the corpus of patents on the abstract and claim fields. We found that indexing the entire list of claims concatenated did not provide good signals to our chat interface in practice. For a Vector DB, we chose the managed version of [Deep Lake vector database](https://github.com/activeloopai/deeplake), that provides features such as [Deep Memory for increased retrieval accuracy](https://www.activeloop.ai/resources/use-deep-memory-to-boost-rag-apps-accuracy-by-up-to-22/), as well as an optimized [HNSW index for up to 75% lower cost without impact on the speed](https://www.activeloop.ai/resources/deep-lake-hnsw-index-rapidly-query-35-m-vectors-save-80/) - we chose this database due to its native [Deep Lake LangChain integration](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/deeplake.html), cloud deployment, fast query time, and TQL language that made filtered queries easy relative to competitors like Weaviate, Opensearch, Elasticsearch, and Pinecone. To create our index, we extended the fields list in the [Deep Lake object in Langchain](https://github.com/hwchase17/langchain/blob/master/langchain/vectorstores/deeplake.py) but otherwise used that integration as is. Our vectorization process ran for eight days on a single V100 GPU.


# **Deploying Search APIs**
Once our search indices stood up, we wrote APIs around them to provide context to our chat queries and provide patent searches. We used the Deep Lake TQL queries to filter our search queries, which allows you to filter your vector database for metadata efficiently. Here is a snippet of what one of those TQL queries looks like: ```python search_deeplake_claims_time = time.time() embedding = model.encode([query_text])[0] embedding_search = ",".join([str(item) for item in embedding]) if "year" in filters.keys(): year_filter = f"filing_year = '{filters['year']}'" else: year_filter = "" if "classification" in filters.keys(): classification_filter = "classification = '" + filters["classification"] + "'" else: classification_filter = "" if year_filter != "" and classification_filter != "": tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score WHERE {year_filter} and {classification_filter} ) order by score desc limit {top_k}" elif year_filter != "" and classification_filter == "": tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score WHERE {year_filter}) order by score desc limit {top_k}" elif year_filter == "" and classification_filter != "": tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score WHERE {classification_filter}) order by score desc limit {top_k}" else: tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score) order by score desc limit {top_k}" ds_view = ds.query(tql_query) patents = [] for i in range(len(ds_view)): patents.append(json.loads(ds_view.patent[i].data()))search_deeplake_claims_time = time.time() embedding = model.encode([query_text])[0] embedding_search = ",".join([str(item) for item in embedding]) if "year" in filters.keys(): year_filter = f"filing_year = '{filters['year']}'" else: year_filter = "" if "classification" in filters.keys(): classification_filter = "classification = '" + filters["classification"] + "'" else: classification_filter = "" if year_filter != "" and classification_filter != "": tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score WHERE {year_filter} and {classification_filter} ) order by score desc limit {top_k}" elif year_filter != "" and classification_filter == "": tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score WHERE {year_filter}) order by score desc limit {top_k}" elif year_filter == "" and classification_filter != "": tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score WHERE {classification_filter}) order by score desc limit {top_k}" else: tql_query = f"select * from (select *, cosine_similarity(embedding, ARRAY[{embedding_search}]) as score) order by score desc limit {top_k}" ds_view = ds.query(tql_query) patents = [] for i in range(len(ds_view)): patents.append(json.loads(ds_view.patent[i].data())) ``` On receiving patents for a search, we return that list in the search APIs or pass those patents on as context to our chat APIs.


# **Deploying LLM Inference APIs**
Once our search APIs were deployed, the last thing remaining for the backend was to deploy our LLMs to the cloud for scalable inference. We deployed our finetuned models on top of the HuggingFace DLC to Amazon Sagemaker.


# **The Final Application**
With our final ensemble of finetuned LLMs and search APIs constructed, we could put our meta agent LLM in front of our APIs to route user queries to their proper location. We wrote a simple flask server to run each inference and deployed it onto an AWS p3.2xlarge machine to make the vectorization calls locally with the rest of the app. [https://lh7-us.googleusercontent.com/7aPMqd22_E5g3zNsTBI6RY22R_EIt_qYIUjPDnYXcoo_gUkKR9GeUTEgZ2b3_js3tOkggvykDLtSC59hxzeF3Fwj8pVwBHMN0uYymnqIowsYzNvxa_OlVGnO8zJ8_D5LfpVYq6uxq2pVgiMcCMFgfMM](https://lh7-us.googleusercontent.com/7aPMqd22_E5g3zNsTBI6RY22R_EIt_qYIUjPDnYXcoo_gUkKR9GeUTEgZ2b3_js3tOkggvykDLtSC59hxzeF3Fwj8pVwBHMN0uYymnqIowsYzNvxa_OlVGnO8zJ8_D5LfpVYq6uxq2pVgiMcCMFgfMM) Generating claims for an electrical circuit invention [https://lh7-us.googleusercontent.com/7u8WkwABAbTL5Ae-0C6KD-Xm8My8EekQTmTzrKdjfPk9Wl0RG57VB4vCpYduKugksIWSrEBr_Z5PGxSeYjTdQHcUtCTnfxXBPnOwAHjz5G3eIySwr-vv2TTDAPrNl9LKfwIE5edpJfW4aQqJ_NzzSLI](https://lh7-us.googleusercontent.com/7u8WkwABAbTL5Ae-0C6KD-Xm8My8EekQTmTzrKdjfPk9Wl0RG57VB4vCpYduKugksIWSrEBr_Z5PGxSeYjTdQHcUtCTnfxXBPnOwAHjz5G3eIySwr-vv2TTDAPrNl9LKfwIE5edpJfW4aQqJ_NzzSLI) Searching for ironing board patents


# **Conclusion**
While the stack for training and deploying finetuned LLMs in practice is far from solidified, we have summarized an efficient approach in this post, working with the latest and most remarkable technologies in the space right now, including: - [Deep Lake from Activeloop](https://github.com/activeloopai/deeplake) - [Hugging Face Optimum](https://huggingface.co/docs/optimum/main/en/intel/index) by Intel and [training routines](https://huggingface.co/docs/optimum/index) - [Habana Gaudi HPU hardware](https://habana.ai/products/) PatentPT is one use case for finetuning LLMs, where you are likely to find much greater accuracy and control over your LLMs output than simply constructing prompts consumed by general AI APIs. **Disclaimers** Performance varies by use, configuration, and other factors. Learn more on the/Performance Index site. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for configuration details. No product or component can be absolutely secure. Your costs and results may vary. For workloads and configurations, visit 4th Gen Xeon® Scalable processors at www.intel.com/processorclaims. Results may vary. Intel technologies may require enabled hardware, software or service activation. Intel does not control or audit third-party data. You should consult other sources to evaluate accuracy. Intel® technologies may require enabled hardware, software, or service activation. © Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.


LangChain Basic Concepts Recap

# LangChain: Basic Concepts Recap
--- In this lesson, we'll review the essential features of LangChain. We will examine the architecture comprising various components, such as data loading, processing, and segmentation, to provide optimal information to language models. Additionally, we will highlight the significance of indexing and retrieval. The material in this lesson was covered in depth with code and project examples in our earlier course **[LangChain & Vector Databases in Production](https://learn.activeloop.ai/courses/langchain).** This overview is structured to clearly convey LangChain's features, providing foundational knowledge for advanced Generative AI and LLM-based projects. Here’s a [Notebook](https://colab.research.google.com/drive/1rDAJyqfVkZR-HQGlzeMi1tNekqyTgXgy?usp=sharing) with all the code we will go through in this lesson.


LangChain Basic Concepts Recap

# LangChain: Basic Concepts Recap
## 1. Preprocessing the Data
LangChain's approach to structuring documents is particularly favorable for developers and researchers. It provides tools that help structure documents for convenient use with LLMs. Document loaders simplify the process of loading data into documents, and text splitters break down lengthy pieces of text into smaller chunks for better processing. Finally, the indexing process involves creating a structured database of information that the language model can query to enhance its understanding and responses. ### 1-1. [Document Loaders](https://python.langchain.com/docs/modules/data_connection/document_loaders) Document Loaders are responsible for loading documents into structured data. They handle various types of documents, including PDFs, and convert them into a data type that can be processed by the other LangChain functions. It enables loading data from multiple sources into `Document` objects. LangChain provides over 100 document loaders and integrations with other major providers in the space, like [AirByte](https://docs.airbyte.com/integrations/destinations/langchain) and [Unstructured](https://python.langchain.com/docs/integrations/providers/unstructured), and from all sources, such as private [S3](https://aws.amazon.com/s3/) buckets and public websites. **Read from Files/Directories** Handling various input formats and transforming them into the Document format is easy. For instance, you can load the CSV data using the [CSVLoader](https://python.langchain.com/docs/modules/data_connection/document_loaders/csv). Each row in the CSV file will be transformed into a separate `Document`. ```python from langchain.document_loaders import CSVLoader # Load data from a CSV file using CSVLoader loader = CSVLoader("./data/data.csv") documents = loader.load() # Access the content and metadata of each document for document in documents: content = document.page_content metadata = document.metadata ``` Some of the popular loaders include the `[TextLoader](https://python.langchain.com/docs/modules/data_connection/document_loaders/)` for text files, the `[DirectoryLoader](https://python.langchain.com/docs/modules/data_connection/document_loaders/file_directory)` for loading all the files in a directory, the `[UnstructuredMarkdownLoader](https://python.langchain.com/docs/modules/data_connection/document_loaders/markdown)` for markdown files, and the `[PyPDFLoader](https://python.langchain.com/docs/modules/data_connection/document_loaders/pdf)` for loading PDF files. **Public Source Loaders** Loaders for popular public sources allow the data to be transformed into `Document` objects. For example, the `[WikipediaLoader](https://python.langchain.com/docs/integrations/document_loaders/wikipedia)` retrieves the content of the specified Wikipedia page and loads it into a `Document`. ```python from langchain.document_loaders import WikipediaLoader # Load content from Wikipedia using WikipediaLoader loader = WikipediaLoader("Machine_learning") document = loader.load() ``` Another popular loader is `[UnstructuredURLLoader](https://python.langchain.com/docs/integrations/document_loaders/url)`, which allows reading from public web pages. **Proprietary Data loaders** These loaders are designed to handle proprietary sources that may require additional authentication or setup. For example, a loader could be created to load custom data from an internal database or an API with proprietary access. Popular loaders of this category are `[GoogleDriveLoader](https://python.langchain.com/docs/integrations/document_loaders/google_drive)` for loading documents from Google Drive and `[MongodbLoader](https://python.langchain.com/docs/integrations/document_loaders/mongodb)` for loading documents from a MongoDB database. ### 1-2. **[Document transformers](https://python.langchain.com/docs/modules/data_connection/document_transformers/) (chunking methods)** A crucial part of retrieval is fetching only the relevant details of documents. This involves several transformation steps to prepare the documents for retrieval. One of the primary steps here is splitting (or chunking) a large document into smaller segments. LangChain provides several transformation algorithms and optimized logic for specific document types. <aside> 💡 Chunking is essential when the source documents are too large for the maximum input size imposed by models and involves partitioning the content into chunks that the LLM can process and then be used for indexing and queries. For instance, the GPT-4 model initially had context window of 8,000 tokens and the ada-002 embedding model is still limited to 8,000 tokens. Considering an average text document contains roughly 500 tokens per page, a 8000 token context window can handle a maximum of 16 pages. To address a query using a 100-page document, we must, therefore, divide the document into smaller chunks. </aside> LangChain offers several key chunking transformation strategies. - **Fixed-size chunks** that define a fixed size that's sufficient for semantically meaningful paragraphs (for example, 300 words) and allows for some overlap (for example, an additional 30 words). Overlapping ensures continuity and context preservation between adjacent chunks of data,


LangChain Basic Concepts Recap

# LangChain: Basic Concepts Recap
## 1. Preprocessing the Data
improving the coherence and accuracy of the created chunks. For example, you may use the `[CharacterTextSplitter](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/split_by_token#hugging-face-tokenizer)` splitter to split every N character or token if configured with a tokenizer. - **Variable-sized chunks** partition the data based on content characteristics, such as end-of-sentence punctuation marks, end-of-line markers, or using features in the NLP libraries. It ensures the preservation of coherent and contextually intact content in all chunks. An example is the `[RecursiveCharacterTextSplitter](https://python.langchain.com/docs/modules/data_connection/document_transformers/)` splitter. - **Customized chunking** when dealing with large documents, you might use variable-sized chunks but also append the document title to chunks from the middle of the document to prevent context loss. This can be done, for example, with the `[MarkdownHeaderTextSplitter](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/markdown_header_metadata)`. Chunking offers a mixed bag of strengths and weaknesses in processing extensive documents with LLMs. A key advantage is its ability to manage documents that exceed the context window of an LLM. This capability enables the model to handle and analyze significantly larger texts than it could in a single pass, expanding its applicability and utility in processing lengthy documents. However, this approach comes with a notable drawback. In dividing a document into chunks, there's a risk of losing vital context related to the overall document. While individually coherent, each chunk might only partially capture the nuances and interconnected elements present in the full text. This can lead to a fragmented or incomplete understanding of the document, as important details and subtleties might be overlooked or misinterpreted when the text is not viewed cohesively. ### 1-3. **[Indexing](https://python.langchain.com/docs/modules/data_connection/indexing)** Indexing is a process that involves storing and organizing data from various sources into a vector store, which is essential for efficient storing and retrieving. The process typically consists of storing the chunk along with an embedding representation of it, which captures the meaning of the text and makes it easy to retrieve chunks by semantic similarity. Embeddings are usually generated by [embedding models](https://python.langchain.com/docs/modules/data_connection/text_embedding/), such as the `OpenAIEmbeddings` models.


LangChain Basic Concepts Recap

# 2. Models
### 2-1. LLMs LangChain provides an `LLM` class for interfacing with various language model providers, such as [OpenAI](https://openai.com/), [Cohere](https://cohere.com/), and [Hugging Face](https://huggingface.co/). Here's an example of some integrations. The rest of the list can be found in the conclusion section. Every LLM listed has a direct link to the document page with more detailed information. | LLM | DESCRIPTION | | --- | --- | | https://openai.com/?ref=davidgentile.net | An AI research organization, very popular for the https://chat.openai.com/ product | | https://huggingface.co/?ref=davidgentile.net | A platform that hosts thousands of pre-trained models and datasets | | https://cohere.ai/?ref=davidgentile.net | A platform that provides natural language understanding APIs powered by large-scale neural networks | | https://github.com/Llama-Group/Project-Llama?ref=davidgentile.net | A library that makes it easy to load (small) language models locally on your PC | | https://azure.microsoft.com/en-us/services/openai/?ref=davidgentile.net | A cloud service that provides access to OpenAI’s LLMs | Before seeing an example of interacting with a “chat model”, you must install the required packages to set up the environment. ```bash pip install langchain==0.0.346 openai==1.3.7 tiktoken==0.5.2 cohere==4.37 ``` The next step is setting the OpenAI API key in your environment to use the endpoints. Please remember to replace the `<YOUR_OPENAI_API_KEY>` placeholder with your key. Alternatively, you can load environment variables from a `.env` file using the `[dotenv](https://pypi.org/project/python-dotenv/)` library. ```python import os os.environ["OPENAI_API_KEY"] = "<YOUR_OPENAI_API_KEY>" ``` Now, you can run the following code to interact with the OpenAI GPT-3.5 Turbo chat model. ```python from langchain.chat_models import ChatOpenAI from langchain.schema import HumanMessage, SystemMessage chat = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0) messages = [ SystemMessage( content="You are a helpful assistant." ), HumanMessage( content="What is the capital of France?" ), ] chat(messages) ``` ``` AIMessage(content='The capital of France is Paris.') ``` Chat models are a variation of language models that use a message-based input and output system. Chat models operate using LLMs but have a different interface that uses “messages” instead of raw text input/output. A Chat model in LangChain has three types of messages: - `SystemMessage` sets the behavior and objectives of the chat model. You would give specific instructions here like, “Act like a Marketing Manager.” or “Return only a JSON response and no explanation text.” - The `HumanMessage` type is where you would input the user’s prompts to be sent to the model. - Lastly, the `AIMessage` represents the responses from the models, which can be used to pass the history of previous messages to the model. ### 2-2. **Embedding Models** [Text embedding models](https://python.langchain.com/docs/modules/data_connection/text_embedding/) are a standardized interface for various embedding model providers like OpenAI, Cohere, and HuggingFace. These models transform text into vector representations, enabling operations like semantic search through text similarity in vector space. In LangChain, the `embed_documents` method is used to embed multiple texts, providing a list of vector representations. ```python from langchain.embeddings import OpenAIEmbeddings # Initialize the model embeddings_model = OpenAIEmbeddings() # Embed a list of texts embeddings = embeddings_model.embed_documents( ["Hi there!", "Oh, hello!", "What's your name?", "My friends call me World", "Hello World!"] ) print("Number of documents embedded:", len(embeddings)) print("Dimension of each embedding:", len(embeddings[0])) ``` ``` Number of documents embedded: 5 Dimension of each embedding: 1536 ``` The above code snippet illustrates how the embedding model can effectively convert phrases such as "Hi there!" into a 1536-dimensional vector. The remarkable aspect of these embeddings lies in their consistent output dimensionality, regardless of the input's length, while capturing the meaning of the sequences. These attributes enable us to measure sentence similarity using similarity metrics, such as [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity).


LangChain Basic Concepts Recap

# 2. Models
## 3. ****The Role of [Vector Stores](https://python.langchain.com/docs/modules/data_connection/vectorstores/)**
The increasing use of embeddings has created a demand for databases that can effectively store and search these embeddings. Vector stores tailored for managing vector data, such as [Deep Lake by Activeloop](https://www.activeloop.ai/), have become essential. They are a fundamental framework for storing and retrieving embeddings produced by LLMs. Embeddings are high-dimensional vectors that capture the semantics of textual data. They are generated using LLMs and are crucial for tasks like text similarity, clustering, and retrieval. Traditional databases are not optimized for high-dimensional data. Vector stores, on the other hand, are built to handle such data, offering faster and more accurate retrieval. The advantages of using vector stores in LangChain: - **Speed**: Vector stores are optimized for quick data retrieval, which is essential for real-time applications. - **Scalability**: As your application grows, so does your data. Vector stores can handle this growth efficiently. - **Precision**: Vector stores ensure you get the most relevant results using specialized algorithms for nearest neighbor search. ### 3-1. **[Retrievers](https://python.langchain.com/docs/modules/data_connection/retrievers/)** Once the data is in the database, you need to retrieve it. Retrievers in LangChain are interfaces that return documents in response to a query. The most straightforward approach is to use basic similarity metrics such *as cosine similarity*, which compares the angle between vector embeddings of the question and potential answers, ensuring that the responses generated are semantically aligned with the query. This method effectively narrows down the most contextually relevant information from a vast dataset, improving the precision and relevance of the responses. However, there are more advanced retrieval approaches to increase precision. LangChain offers a variety of retrieval methods; some examples include: - [Parent Document Retriever](https://python.langchain.com/docs/modules/data_connection/retrievers/parent_document_retriever): It creates multiple embeddings, allowing you to look up smaller chunks but return larger contexts. It becomes easier to discover related content when dealing with smaller chunks through cosine similarity, and then the parent document (e.g., the whole document containing that chunk) will be used to offer additional context to the LLM for generating the final answer. - [Self-Query Retriever](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query): User questions often reference something that isn't just semantic but contains logic that can be represented as metadata filters. Self-query allows us to generate several filters based on the user's input prompt and apply them to the document's metadata. It improves performance by getting the most out of users' prompts and using the document and its metadata to retrieve the most relevant content. For example, a query like “What were Microsoft's revenues in 2021?” would generate the filter “datetime = 2021,” which would then be used to filter the documents (assuming that we know their publish dates).


LangChain Basic Concepts Recap

# 2. Models
## 4. [Chains](https://python.langchain.com/docs/modules/chains/)
Chains consist of powerful, reusable components that can be linked together to perform complex tasks. Integrating prompt templates with LLMs using chains allows a powerful synergy. Taking the output of one LLM and using it as input for the next makes it feasible to connect multiple prompts sequentially. Additionally, it allows us to integrate LLMs with other components, such as long-term memory and output guarding. Chains can enhance the overall quality and depth of interactions. The following two chain classes can be helpful in different situations: `LLMChain` and `SequentialChain`. ### 4-1. **[LLMChain](https://python.langchain.com/docs/modules/chains/foundational/llm_chain)** LLMChain is the simplest form of chain in LangChain that transforms user inputs using a `PromptTemplate`, a fundamental and widely used tool for interacting with LLMs. The following code receives the user’s input and a parser class to create a `PromptTemplate` object that can then interact with a model. The prompt object defines the parameters of our request to the model and determines the expected format of the output. Then, we can use the `LLMChain` class to tie the prompt and model to make predictions. Note that the `StrOutputParser` class will ensure that we receive a string containing the responses from the LLM. ```python from langchain.chains import LLMChain from langchain.chat_models import ChatOpenAI from langchain.prompts import PromptTemplate from langchain.schema import StrOutputParser template = """List all the colors in a rainbow""" prompt = PromptTemplate( template=template, input_variables=[], output_parser=StrOutputParser() ) chat = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0) llm_chain = LLMChain(prompt=prompt, llm=chat) llm_chain.predict() ``` ``` The colors in a rainbow are: 1. Red 2. Orange 3. Yellow 4. Green 5. Blue 6. Indigo 7. Violet ``` It is also possible to use the newly introduced [LangChain Expression Language (LCEL)](https://python.langchain.com/docs/expression_language/) scheme to make the code more readable. The following code will generate the same output as the previous, but it is easier to interpret the data flow from the user prompt and parse the output. ```python from langchain.chat_models import ChatOpenAI from langchain.prompts import PromptTemplate from langchain.schema import StrOutputParser prompt = PromptTemplate.from_template( "List all the colors in a {item}." ) runnable = prompt | ChatOpenAI(temperature=0) | StrOutputParser() runnable.invoke({"item": "rainbow"}) ``` ``` The colors in a rainbow are: 1. Red 2. Orange 3. Yellow 4. Green 5. Blue 6. Indigo 7. Violet ``` ### 4-2. **[Sequential](https://python.langchain.com/docs/modules/chains/foundational/sequential_chains)** After invoking a language model using `LLMChain`, we can now experiment with making a series of subsequent calls to an LLM. This approach is especially beneficial for using the output of one call as the input for the next, streamlining the process, and enabling complex interactions across various applications. Here's an example of how to use the `SequentialChain`. ```python from langchain.prompts import PromptTemplate post_prompt = PromptTemplate.from_template( """You are a business owner. Given the theme of a post, write a social media post to share on my socials. Theme: {theme} Content: This is social media post based on the theme above:""" ) review_prompt = PromptTemplate.from_template( """You are an expert social media manager. Given the presented social media post, it is your job to write a review for the post. Social Media Post: {post} Review from a Social Media Expert:""" ) from langchain.chat_models import ChatOpenAI from langchain.schema import StrOutputParser llm = ChatOpenAI(temperature=0.0) chain = ( {"post": post_prompt | llm | StrOutputParser()} | review_prompt | llm | StrOutputParser() ) chain.invoke({"theme": "Having a black friday sale with 50% off on everything."}) ``` ``` This social media post is highly effective in promoting the Black Friday sale. The use of emojis and exclamation marks adds excitement and grabs the attention of the audience. The post clearly states the offer - a 50% off on everything in-store and online, which is a great deal. It also highlights the variety


LangChain Basic Concepts Recap

# 2. Models
## 4. [Chains](https://python.langchain.com/docs/modules/chains/)
of products available, from trendy fashion pieces to must-have accessories, appealing to a wide range of customers. The post encourages urgency by mentioning that the sale is for a limited time only and advises customers to arrive early to catch the best deals. The call to action is clear, directing customers to visit the website or head to the store to explore the products. The post also encourages customers to spread the word and tag their shopping buddies, which can help increase the reach and engagement of the post. Overall, this social media post effectively communicates the details of the Black Friday sale, creates excitement, and encourages customers to take advantage of the unbeatable offer. It is well-written, visually appealing, and likely to generate interest and engagement from the audience. ``` The example above uses the mentioned LCEL language to create two distinct chains. The first chain will generate a social media post based on a theme, and the other one will act as a social media expert to review the generated post. It is evident that the output of the `post_prompt` will be marked as the `post` variable and passed to the `review_prompt` template. You can see a variation of this code in the lesson notebook, where the model returns both social media posts and the review. Also, refer to [the documentation](https://python.langchain.com/docs/modules/chains/foundational/sequential_chains#legacy-sequentialchain) for the use of the `SequenceChain` class.


LangChain Basic Concepts Recap

# 2. Models
## 5. [Memory](https://python.langchain.com/docs/modules/memory/)
Memory is the backbone for maintaining context in ongoing dialogues, ensuring that the LLM can provide coherent and contextually relevant responses. Memory in LangChain is essential for context preservation and enhancing user experience. Traditional conversational models often struggle with maintaining context. LangChain’s Memory module addresses this by storing both input and output messages in a structured manner. The system can offer more personalized and relevant responses by remembering and referring to past interactions, significantly improving the user experience. This flexibility makes it suitable for conversational applications. Memory module methods save the current conversational context, including both user input and system output.


LangChain Basic Concepts Recap

# Conclusion
In this lesson, we explored LangChain's essential components. LangChain stands out with its versatile document loaders, which are capable of importing varied data types. We discussed its document transformation techniques, especially chunking, which breaks down long texts into manageable segments. We also learned about vector stores and their benefits, including faster data retrieval, scalability to handle data growth, and precision in retrieving the most relevant results. Additionally, we covered LangChain's use of retrievers and chains. Retrievers provide various methods to return documents in response to queries. Chains facilitate the creation of complex workflows by linking multiple components and integrating LLMs with external data and long-term memories like chat history. This results in more effective responses and coherent conversations. Overall, it is important to have a good grasp of LangChain's capabilities to pursue advanced projects in the field. >> [Notebook](https://colab.research.google.com/drive/1rDAJyqfVkZR-HQGlzeMi1tNekqyTgXgy?usp=sharing). --- **RESOURCES** [Simple guide to Text Chunking for Your LLM Applications](https://medium.com/aimonks/simple-guide-to-text-chunking-for-your-llm-applications-bddfe8ad7892) - more on chunking [Understanding Text Chunking for the LLM Application](https://medium.com/@ashu.goel_9925/understanding-text-chunking-for-the-llm-application-da59cbc2855b) - complete guide [LangChain: A Complete Guide & Tutorial](https://nanonets.com/blog/langchain/#module-ii-retrieval) - more on models [LangChain Models: Simple and Consistent Interfaces for LLMs, Chat, and Text Embeddings](https://www.davidgentile.net/langchain-models/) - more on llms [LangChain: Quickly Build Apps with OpenAI API – Kanaries](https://docs.kanaries.net/articles/langchain-openai#chains-combining-llms-and-prompts) - Here’s a list of LLMs integrated with LangChain along with a description of them (source is [https://www.davidgentile.net/langchain-models/](https://www.davidgentile.net/langchain-models/)): | LLM | DESCRIPTION | | --- | --- | | https://www.ai21.com/?ref=davidgentile.net | A platform for building AI applications that comprehend and generate natural language powered by Jurassic-1 language models | | https://aleph-alpha.de/?ref=davidgentile.net | A company that develops large-scale language models for the European market | | https://azure.microsoft.com/en-us/services/openai/?ref=davidgentile.net | A cloud service that provides access to OpenAI’s GPT-3 language model | | https://www.banana.dev/?ref=davidgentile.net | A platform that helps you build web applications using natural language commands and templates | | https://www.cerebrium.ai/?ref=davidgentile.net | A platform that enables data scientists and developers to build and deploy AI solutions faster and easier | | https://cohere.ai/?ref=davidgentile.net | A platform that provides natural language understanding APIs powered by large-scale neural networks | | https://deepinfra.ai/?ref=davidgentile.net | A platform that simplifies the deployment and management of deep learning models on cloud infrastructure | | https://forefront.ai/?ref=davidgentile.net | A platform that helps businesses leverage AI to optimize their operations and customer experiences | | https://goose.ai/?ref=davidgentile.net | A platform that provides natural language generation APIs for various domains and use cases | | https://gpt4all.com/?ref=davidgentile.net | A platform that allows anyone to interact with GPT-3 and other language models without coding or API keys | | https://github.com/huggingface/huggingface_hub/tree/main/api-inference-community/docker_images/generic/app?ref=davidgentile.net | A tool that allows you to run Hugging Face pipelines locally on your machine or server | | https://github.com/Llama-Group/Project-Llama?ref=davidgentile.net | A library that provides fast and easy-to-use data structures and algorithms for C++ developers | | https://manifest.ai/?ref=davidgentile.net | A platform that helps you create engaging and interactive content using natural language generation and computer vision | | https://modal.ai/?ref=davidgentile.net | A platform that helps you build conversational AI applications using natural language understanding and dialog management | | https://nlpcloud.io/?ref=davidgentile.net | A platform that provides high-performance NLP APIs for various tasks such as sentiment analysis, named entity recognition, summarization, etc. | | https://github.com/bigscience-workshop/petals?ref=davidgentile.net | Petals run 100B+ language models at home, BitTorrent-style. | | https://pipeline.ai/?ref=davidgentile.net | A platform that helps you build, train, deploy, and monitor machine learning models at scale on any cloud or edge device | | https://predictionguard.com/?ref=davidgentile.net | A platform that helps you monitor and improve the performance of your machine-learning models in production | | https://promptlayer.com/openai?ref=davidgentile.net | A tool that helps you craft effective prompts for OpenAI’s GPT-3 language model using best practices and examples | | https://replicate.ai/?ref=davidgentile.net | A tool that helps you


LangChain Basic Concepts Recap

# Conclusion
version, package, and share your machine learning experiments with your team or the world | | https://runhouse.io/?ref=davidgentile.net | A platform that helps you run your machine learning models on any device with a web browser using WebAssembly technology | | https://github.com/aws-samples/sagemaker-endpoint-deployment-of-fastai-model-with-torchserve?ref=davidgentile.net | A tool that helps you deploy your machine learning models on AWS SageMaker with a few lines of code | | https://stochastic.ai/?ref=davidgentile.net | A platform that helps you optimize your machine learning workflows using probabilistic programming and Bayesian inference techniques | | https://writer.com/?ref=davidgentile.net | A platform that helps you create consistent and effective content using generative AI and brand guidelines |


LangChain Overview Agents, Tools, and OpenGPT Intr

# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## **Introduction**
In this lesson, we’ll recap the concepts of tools and agents in LangChain and its main applications. We’ll also learn about LangChain OpenGPTs, an open-source effort to create AI assistants similar to the OpenAI GPTs.


# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## **Agents**
[LangChain agents](https://python.langchain.com/docs/modules/agents/) integrate chains, prompts, memory, and tools to complete tasks. Agents can be used to perform a wide range of tasks, from executing a series of steps in a specific order to interacting with external systems like Gmail or SQL databases and more. They can be customized to fit a variety of use cases, and LangChain provides a suite of tools and functionalities to facilitate the process. Let’s review the relevant LangChain concepts: - **[Chain](https://python.langchain.com/docs/modules/chains/)**: A sequential application of models or tools, such as passing a prompt to an LLM and then parsing it. - **[Tool](https://python.langchain.com/docs/modules/agents/tools/)**: In LangChain, a tool is a function that performs a specific task that an LLM can leverage to get relevant information for its task completion. It can be a Google Search, a Database lookup, a Python [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop), or other chains. - **[Memory](https://python.langchain.com/docs/modules/memory/)**: Memory keeps track of past interactions with an LLM to be used as context for the next interactions.


# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## **Agent types**
LangChain has several agent types. Find a comprehensive list below. - **[Zero-shot ReAct](https://python.langchain.com/docs/modules/agents/agent_types/react)**: It leverages the [ReAct framework](https://arxiv.org/abs/2210.03629) to make tool decisions based exclusively on the descriptions of the available tools. It’s called “zero-shot” because it only leverages the tool descriptions without using specific examples. - **[Structured Input ReAct](https://python.langchain.com/docs/modules/agents/agent_types/structured_chat)**: Optimized for handling tools that require multiple inputs. - **[OpenAI Functions Agent](https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent)**: Tailored for models specifically fine-tuned for function calls, this agent is compatible with advanced models like `gpt-3.5-turbo` and `gpt-4`. - **[Conversational Agent](https://python.langchain.com/docs/modules/agents/agent_types/chat_conversation_agent)**: Focuses on conversational contexts, utilizing ReAct for tool selection and incorporating memory to recall past interactions. - **[Self-Ask with Search Agent](https://python.langchain.com/docs/modules/agents/agent_types/self_ask_with_search)**: Centering around the "Intermediate Answer" tool, it excels at finding factual answers to queries, mirroring the approach in the original *[self-ask with](https://ofir.io/self-ask.pdf) search* study. - **[ReAct Document Store Agent](https://python.langchain.com/docs/modules/agents/agent_types/react_docstore)**: This agent requires "Search" and "Lookup" tools, simultaneously using both features while providing a stream of thoughts. LangChain’s agents essentially provide the ‘reasoning’ behind the choice of the to-do action, deciding whether to involve multiple tools, just one, or none at all in the process.


# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## **Plan-and-Execute Agents**
The [plan-and-execute agents](https://blog.langchain.dev/plan-and-execute-agents/) first make a plan with multiple actions and then execute each action sequentially. They are more suited for complex or long-running tasks as they maintain focus on long-term objectives and focus. However, they may lead to more latency.


# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## Available tools and custom tools
You can find the list of tools that integrate LangChain with other tools [here](https://python.langchain.com/docs/integrations/toolkits/). Some examples are: - [The Python tool](https://python.langchain.com/docs/integrations/toolkits/python): It’s used to execute generated Python code to answer a question. - [The JSON tool](https://python.langchain.com/docs/integrations/toolkits/json): It’s used when interacting with a JSON blob that is too large to fit in the LLM context window. - [The CSV tool](https://python.langchain.com/docs/integrations/toolkits/csv): It’s used to interact with CSV files. [Custom tools](https://python.langchain.com/docs/modules/agents/tools/custom_tools) extend the capabilities of agents, making them adaptable to a wide range of specialized tasks and interactions. Custom tools provide task-specific functionality and flexibility for precise behaviors tailored to unique use cases. The level of customization depends on the creation of advanced interactions, where tools can be orchestrated to perform complex behaviors, such as generating questions, searching the web for answers, and summarizing the gathered information.


# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## ****LangChain OpenGPT****
LangChain OpenGPT is an open-source effort to create an experience similar to OpenAI’s assistants and GPTs. In contrast to OpenAI GPTs, LangChain OpenGPT allows you to configure the LLM used, the tools, the vector DB, the retrieval algorithm, and the chat history DB. Let’s see how to use them: 1. **Clone the Repository**: To interact with Langchain's OpenGPTs, follow these steps detailed in their [GitHub repository](https://github.com/langchain-ai/opengpts). The easiest way to launch OpenGPTs locally is through [Docker](https://www.docker.com/) and “[docker compose](https://docs.docker.com/compose/)”. First, clone the repo locally and `cd` into it. ```bash git clone https://github.com/langchain-ai/opengpts.git cd opengpts ``` You should now see a `.env` file with the following content. ```bash OPENAI_API_KEY=placeholder ANTHROPIC_API_KEY=placeholder YDC_API_KEY=placeholder TAVILY_API_KEY=placeholder AZURE_OPENAI_DEPLOYMENT_NAME=placeholder AZURE_OPENAI_API_KEY=placeholder AZURE_OPENAI_API_BASE=placeholder AZURE_OPENAI_API_VERSION=placeholder ``` By default, the app will be using the OpenAI models, so replace the `placeholder` of `OPENAI_API_KEY` with your OpenAI key. You can now launch everything with the following command. ```bash docker compose up ``` Now, visit [http://localhost:8100/](http://localhost:8100/); you should see the following page. ![Screenshot 2023-12-15 at 15.25.28.png](LangChain%20Overview%20Agents,%20Tools,%20and%20OpenGPT%20Intr%20d3b8525f4cfd43c097accae103c39486/LangChain%20Overview%20Agents,%20Tools,%20and%20OpenGPT%20Intr%20c3bced6325074bc4be074a75d1686d01/Screenshot_2023-12-15_at_15.25.28.png)


# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## ****Creating OpenGPTs****
We are now ready to create our first OpenGPT! To set a custom prompt in LangChain's OpenGPTs, you begin by defining a specific role or persona for the AI, like a 'Career Counselor'. > You guide the models' responses to fit the desired context and ensure that its advice, insights, or recommendations are aligned with the defined role. > This **`System Message`** outlines the OpenGPTs' responsibilities, tone, and the type of interactions it should engage in: ``` You are a Career Counselor. Your role is to provide insightful and personalized guidance as I navigate my professional path. Whether I'm facing career uncertainties, seeking job advancement, or contemplating a career shift, your expertise is aimed at offering constructive, individualized advice that helps me make informed decisions. Our sessions will be a platform for discussing my professional aspirations, skills, and potential barriers. In our interactions, I expect a supportive environment where I can share my professional experiences, goals, and concerns. Your role is to motivate and provide clear, practical strategies that align with my career objectives. By understanding my unique circumstances, you offer tailored advice and plans to aid my professional growth. This collaboration is crucial for my career development, with your guidance being a cornerstone of my journey towards achieving my career goals. ``` This prompt acts as a foundational script, directing the model's behavior to meet the specific needs of your application or service. Click on “New Bot”, select GPT-3.5-Turbo (or another default that you want to use), name the bot “**Career Counselor**” and provide the System Message we created. Click Save, and you are ready to chat!


# LangChain Overview: Agents, Tools, and OpenGPT Introduction
## Conclusion
In this lesson, we covered LangChain’s tools and agents, their applications, and the creation of LangChain OpenGPTs, an open-source initiative for AI assistants. We explored agent types, including Zero-shot ReAct and Conversational Agents, along with plan-and-execute agents. We discussed the integration of various tools like Python and CSV and the customization of these tools. Finally, we set up and used LangChain OpenGPTs, emphasizing their reconfigurability and role in facilitating AI interactions. RESOURCES: - OpenGPTs [OpenGPTs](https://opengpts-example-vz4y4ooboq-uc.a.run.app/) [Maximizing LangChain Efficiency: Agents and ReAct Method Review.](https://medium.com/towards-artificial-intelligence/maximizing-langchain-efficiency-agents-and-react-method-review-3abcf8a5c874) [GitHub - langchain-ai/opengpts](https://github.com/langchain-ai/opengpts) - tools docs [Defining Custom Tools | 🦜🔗 Langchain](https://python.langchain.com/docs/modules/agents/tools/custom_tools)


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
Build a RAG System with LLM and Deep Memory. Use three different datasets and test the quality of response with and without the Deep Memory feature to improve retrieval! Traditionally, retrieval methods have relied on standard techniques like RAG or query-based document retrieval. However, often the results are not fully satisfying as the retrieved chunks of text are not exactly what we expect. To solve this problem, many techniques are tested every day. In this guide, we will explore how to enhance the storage of indices in the retrieval pipeline in a more effective way through finetuning techniques. Unlike other techniques, Deep Memory enables an automatic and convenient finetuning of the retrieval step on the chunks of data provided, thus improving the solution when compared to a classic and agnostic RAG application. Deep Memory emerges as a fundamental solution in addressing the critical need for accurate retrieval in generating high-quality results. It is crucial to increase the accuracy of Deep Lake’s vector search by up to 22%, achieved through learning an index from labeled queries tailored to specific applications. Importantly, this improvement is achieved without compromising research time, demonstrating the efficacy of Deep Memory in fine-tuning the retrieval process. The enterprise application landscape, particularly the development of “chat with your data” solutions, highlights the importance of accurate retrieval. Current practices involve the integration of Retrieval Augmented Generation (RAG) systems with Large Language Models (LLMs) like GPT-4. The role of Deep Memory in significantly improving the accuracy of vector search becomes fundamental in this context, as it offers a potential solution to increase the reliability of these applications. By emphasizing the importance of accurate retrieval, the integration of technologies like Deep Memory becomes a focal point in achieving the desired level of consistency and precision in generating results. Let’s build a Deep Memory RAG application! We need three main components: A **dataset** containing the text chunks we want to retrieve, an **LLM model** to generate the text embeddings and the **Deep Memory** library. Let's delve into the details.


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
## Preparing the Dataset
In this guide, we have prepared three different datasets that could be downloaded and tested with and without the Deep Memory feature. To find out more about them, you can follow the links: 1. **Finance**: we chose the FinQA Dataset that contains text explaining the economy, acquisitions, etc. It also is a QA dataset, making it easier for us to embed as we already have questions and related answers without generating them! This work focuses on answering deep questions about financial data, aiming to automate the analysis of a large corpus of financial documents. In contrast to existing tasks in the general domain, the finance domain includes complex numerical reasoning and an understanding of heterogeneous representations. Source: [https://github.com/czyssrs/FinQA](https://github.com/czyssrs/FinQA) 2. **Legal**: the Legalbench Dataset contains questions and answers about legal subjects like a company's legal rights, policies, and such. That is a very tedious and specific topic that is not readable by everyone, so retrieving the right information for this task is very welcome! LegalBench tasks span multiple types (binary classification, multi-class classification, extraction, generation, entailment), multiple types of text (statutes, judicial opinions, contracts, etc.), and multiple areas of law (evidence, contracts, civil procedure, etc.). It is a benchmark consisting of different legal reasoning tasks. Source: [https://huggingface.co/datasets/nguha/legalbench?clone=true](https://huggingface.co/datasets/nguha/legalbench?clone=true) 3. **Biomedical**: To address a biomedical topic we chose the Cord19 Dataset, which is about Covid. As it is a very discussed topic, retrieving every possible information is crucial and so we wanted to test it. CORD-19 is a corpus of academic papers about COVID-19 and related coronavirus research. It's curated and maintained by the Semantic Scholar team at the Allen Institute for AI to support text mining and NLP research. Source: [https://github.com/allenai/cord19](https://github.com/allenai/cord19) These three datasets are in the Activeloop organization space, so you need to load them in a Tensor Database format to be able to take advantage of the Deep Memory functionality. In the code above the variable `user_hub` will be equal to the Organization name, in our case `"activeloop"`, and `name_db` will be the `<dataset_name>`. ```python def load_vector_store(user_hub, name_db): vector_store_db = DeepLakeVectorStore( f"hub://{user_hub}/{name_db}", embedding_function=embeddings_function.embed_documents, runtime={"tensor_db": True}, ) return vector_store_db ``` The datasets were created with a preprocessing consisting of 3 different steps: - **Gather** the data - **Divide** the data into **chunks** - **Create sample questions** The last point listed must be applied for every chunk. Specifically, it creates a relevance score that represents how relevant the question is when compared to the chunk of text (this is necessary for the most critical part: the Deep Memory Finetuning). ### Gather the data The simplest solution is to download a QA Dataset on whatever is our topic of interest so that it possesses everything we need. We can easily do it with the following command: ```bash wget <source_link> ``` But what if our data is just a long text? ### Chunk Generation To generate the chunks, we can use libraries like **Langchain**, which provide methods to divide our text into chunks automatically. Below are some examples of the generated chunks: - Legal: ![chunk_legal.webp](Deep%20Memory%20for%20RAG%20Applications%20Across%20Legal,%20Fin%202dfbaa4e47b84ce7a82b1996be45d0ef/chunk_legal.webp) - Biomedical: ![chunk_biomedical.webp](Deep%20Memory%20for%20RAG%20Applications%20Across%20Legal,%20Fin%202dfbaa4e47b84ce7a82b1996be45d0ef/chunk_biomedical.webp) - Finance: ![chunk_finance.webp](Deep%20Memory%20for%20RAG%20Applications%20Across%20Legal,%20Fin%202dfbaa4e47b84ce7a82b1996be45d0ef/chunk_finance.webp) There are multiple strategies to do this step most efficiently. For instance, we can use “.” as a separator character, or define the length of each chunk as a standard, or a combination of those! We suggest creating chunks that are not too short and overlapping them to keep relevant information intact. The main disadvantage of not overlapping chunks is the potential loss of information; depending on the nature of the data and the requirements of the analysis or modeling task, it's often beneficial to experiment with overlapping chunks. ```python from langchain.text_splitter import


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
## Preparing the Dataset
RecursiveCharacterTextSplitter # Divide text in chunks def create_chunks(context, chunk_size=300, chunk_overlap=50): # Initialize the text splitter with custom parameters custom_text_splitter = RecursiveCharacterTextSplitter( # Set custom chunk size chunk_size = chunk_size, chunk_overlap = chunk_overlap, # Use length of the text as the size measure length_function = len, ) chunks = custom_text_splitter.split_text(context) return chunks ``` ### Questions and Relevance Generation This is the most subtle step: how do we generate information like questions or scores without a properly trained model? The good (not so old) LLMs come to the rescue to accomplish this task! We can use prompt engineering on a LLM to generate a **question** for each chunk and at the same time generate the **relevance score** as a **classification task**. We just need to call the model and then parse the text output to obtain the necessary data. To do this we construct a dataset of questions and relevance. Relevance is a set of pairs ([corpus.id](http://corpus.id/): str, significance: str) that provides information where the answer is inside the corpus. Sometimes an answer can be found in multiple locations or have different significance. Relevance enables Deep Memory training to optimize the embedding space for higher accuracy. ![dataset_creation.webp](Deep%20Memory%20for%20RAG%20Applications%20Across%20Legal,%20Fin%202dfbaa4e47b84ce7a82b1996be45d0ef/dataset_creation.webp) An example of how to generate questions and relevance scores is the following: ```python #Sample Prompt message to generate Question and Answer for the provided context system_message = """ Generate a question related to the context and provide a relevance score on a scale of 0 to 1, where 0 is not relevant at all and 1 is highly relevant. The input is provided in the following format: Context: [The context that for the generated question] The output is in the following format: #Question#: [Text of the question] #Relevance#: [score number between 0 and 1] The context is: {context} """ def get_chunk_qa_data(context): # Generate the Question and Relevance Text with LLM llm = OpenAI(temperature=0) llm_chain = LLMChain(llm=llm, prompt=PromptTemplate.from_template(system_message)) output = llm_chain(context) # CHECK THE RELEVANCE STRING IN THE OUTPUT check_relevance = None relevance_strings = ["#Relevance#: ", "Relevance#: ", "Relevance: ", "Relevance"] for rel_str in relevance_strings: if rel_str in output["text"]: check_relevance = rel_str break if check_relevance is None: raise ValueError("Relevance not found in the output") messages = output["text"].split(check_relevance) relevance = messages[1] # CHECK THE QUESTION STRING IN THE OUTPUT question = None question_strings = ["#Question#: ", "Question#: ", "Question: ", "Question"] for qst_str in question_strings: if qst_str in messages[0]: question = messages[0].split(qst_str)[1] break if question is None: raise ValueError("Question not found in the output") return question, relevance ``` Now we are all set for the fun part!


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
## Deep Memory Tool
Now we dive into the core of this tutorial: **Deep Memory**. **Deep Memory** is one of the tools included into the High Performance Features of **Deep Lake**. It is very effective in improving the retrieval accuracy of an LLM model by optimizing your vector store for your use-case, **enhancing the performance** of you overall LLM app. This is possible by performing a **fine-tuning of the embeddings** of your embedding model using your own Dataset enriched by QA additional information that consists in **Questions** and **Relevance scores** (how much the question is related to the text). ![https://images.ctfassets.net/qtqp2awm2ktd/49raxwopyyZ4X8Bga1XwI0/12a59c6e56bd1edec954bad3ab4278d5/Untitled__1_.png](https://images.ctfassets.net/qtqp2awm2ktd/49raxwopyyZ4X8Bga1XwI0/12a59c6e56bd1edec954bad3ab4278d5/Untitled__1_.png) ### Creating the Deep Memory Vector Store Deep Memory leads an index from labeled queries tailored to your Dataset, without impacting search time. These results can be achieved with only a few hundred example pairs of prompt embeddings and the most relevant answers from the vector store. As we can see, Deep Memory uses the dataset text (corpus) along with the questions (queries) and relevance score we generated to train an enhanced retrieval model that can be used without any other modification and adds no latency while boosting the retrieval quality. Deep Memory increases retrieval accuracy without altering your existing workflow. ```python def load_vector_store(user_hub, name_db): vector_store_db = DeepLakeVectorStore( f"hub://{user_hub}/{name_db}", embedding_function=embeddings_function.embed_documents, runtime={"tensor_db": True}, ) return vector_store_db ``` In order to create a Deep Memory Dataset, we just need 2 things: the chunks of text 1. An **embedding function** to generate the text embeddings 2. The metadata we prepared (**questions and relevance scores**) For the first point, embeddings can still be computed using a model of your choice such as Open AI ada-002 or other OSS models BGE by BAAI. Furthermore, search results from Deep Memory can be further improved by combining them with lexical search or reranker. For the second point, we explained how to generate questions and relevance scores in the previous paragraphs. If you want to go deeper into details, there are some questions and relevance computed during the training phase: ***Legal Dataset:*** - Chunk: "*Confidential Information means all confidential information relating to the Purpose which the Disclosing Party or any of its Affiliates, discloses or makes available, to the Receiving Party or any of its Affiliates, before, on or after the Effective Date. This includes the fact that discussions and negotiations are taking place concerning the Purpose and the status of those discussions and negotiations.*" - Question: What is the definition of Confidential Information? ***Biomedical Dataset:*** - Chunk: "*The P2 64 and P3 regions encode the non-structural proteins 2B and 2C and 3A, 3B (1-3) (VPg), 3C pro and 4 structural protein-coding regions is replaced by reporter genes, allow the study of genome 68 replication without the requirement for high containment.*" - Question: What are the non-structural proteins encoded by the P2 64 and P3 regions? ***Finance Dataset:*** - Chunk: "*the deferred fuel cost revisions variance resulted from a revised unbilled sales pricing estimate made in december 2002 and a further revision made in the first quarter of 2003 to more closely align the fuel component of that pricing with expected recoverable fuel costs . the asset retirement obligation variance was due to the implementation of sfas 143 , "accounting for asset retirement obligations" adopted in january 2003 . see "critical accounting estimates" for more details on sfas 143 . the increase was offset by decommissioning expense and had no effect on net income . the volume variance was due to a decrease in electricity usage in the service territory . billed usage decreased 1868 gwh in the industrial sector including the loss of a large industrial customer to cogeneration.*" - Question:


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
## Deep Memory Tool
What was the impact of the asset retirement obligation variance on net income? Providing those inputs to the Vector Store, we can upload the Dataset we created to the Active Loop Dataset Repository. After this step, the Deep Memory feature will do 2 things automatically: 1. Generate the embeddings the embedding model we defined 2. Finetune the indices using the Deep Memory feature And we are all set to try our improved RAG applications! But how to try it out now? ### Deep Memory Search After creating the Deep Memory Dataset, we can search for the right piece of text for our question using the following code: ```python def get_answer(vector_store_db, user_question, deep_memory): # deep memory inside the vectore store ==> deep_memory=True answer = vector_store_db.search( embedding_data=user_question, embedding_function=embeddings_function.embed_query, deep_memory=deep_memory, return_view=False, ) return answer ```


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
## Developing a Deep Memory Search with Gradio
We created a Gradio application to test our application more easily. ![Untitled](Deep%20Memory%20for%20RAG%20Applications%20Across%20Legal,%20Fin%202dfbaa4e47b84ce7a82b1996be45d0ef/Untitled.png) The interface allows us to select the dataset we want to test, write a question, and instantly generate the answer. We can also compare the response returned by the Deep Memory model with that returned by the model without.


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
## Classic RAG vs Deep Memory
To test out the improvements of the Deep Memory step, we prepared and shared 3 datasets we mentioned earlier: **Legal, Medical, Finance**. In the output windows, you can see the benefits of this amazing tool when compared to more classical approaches. If you want to try these models, we suggest you to try one of these questions: - Legal Dataset: - What are the provisions of this Agreement regarding the disclosure of Confidential Information to third parties? - Biomedical Dataset: - What are the advantages of using the new package to visualize data? - Finance Dataset: - What were the primary factors that contributed to the improvement in net cash provided by operating activities during 2015? The following example shows how the model with deep memory and dataset finance is more efficient in the response generated: Deep Memory model: ``` The provisions of this Agreement state that disclosure of Confidential Information to third party consultants and professional advisors is allowed, as long as those third parties agree to be bound by this Agreement. Additionally, both parties are required to keep any confidential information they may have access to confidential, unless required by law or necessary to perform their obligations under this Agreement. This includes not only the information itself, but also the terms of the Agreement and the fact that the parties are considering a business arrangement. ``` Non Deep Memory model: ``` The provisions of this Agreement state that disclosure of Confidential Information to third party consultants and professional advisors is allowed, as long as those third parties agree to be bound by this Agreement. Additionally, the Confidential Information includes the terms of this agreement, the fact that the information is being made available, and the possibility of a business arrangement between the parties. ```


# Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
## Evaluation Metrics
After testing out our datasets, we can see that the Deep Memory contribution is visible in retrieving more suitable information to the query provided as a question by the user. The following metrics show how the **Deep Memory feature** can improve performance: **Legal Dataset:** ``` ---- Evaluating without Deep Memory ---- Recall@1: 12.0% Recall@3: 37.0% Recall@5: 47.0% Recall@10: 57.0% Recall@50: 87.0% Recall@100: 94.0% ---- Evaluating with Deep Memory ---- Recall@1: 19.0% Recall@3: 56.0% Recall@5: 66.0% Recall@10: 79.0% Recall@50: 88.0% Recall@100: 95.0% ``` **Biomedical Dataset:** ``` ---- Evaluating without Deep Memory ---- Recall@1: 59.0% Recall@3: 75.0% Recall@5: 78.0% Recall@10: 81.0% Recall@50: 91.0% Recall@100: 94.0% ---- Evaluating with Deep Memory ---- Recall@1: 69.0% Recall@3: 81.0% Recall@5: 83.0% Recall@10: 86.0% Recall@50: 97.0% Recall@100: 98.0% ``` **Financial Dataset:** ``` ---- Evaluating without Deep Memory ---- Recall@1: 18.0% Recall@3: 51.0% Recall@5: 65.0% Recall@10: 71.0% Recall@50: 98.0% Recall@100: 99.0% ---- Evaluating with Deep Memory ---- Recall@1: 26.0% Recall@3: 66.0% Recall@5: 75.0% Recall@10: 81.0% Recall@50: 99.0% Recall@100: 99.0% ``` In conclusion, it is crucial to recognize that in NLP, success is not only determined by the richness of the data, but also depends on the effectiveness of the retrieval strategy. Although collecting large and diverse datasets is undeniably valuable, how the information is retrieved and presented plays a key role in optimizing model performance. As we have seen in this brief guide, there are tools such as Deep Memory that allow us to be more accurate and efficient and thus generate more relevant answers.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
The "Advanced Retrieval Augmented Generation" module offers an in-depth exploration into optimizing large language models (LLMs) with advanced Retrieval-Augmented Generation (RAG) techniques. Across four lessons, it encompasses a range of topics including query transformation, re-ranking, optimization techniques like fine-tuning, the implementation of Activeloop's Deep Memory, and other advanced strategies using LlamaIndex. Students will gain practical experience in enhancing RAG system performance, from query refinement to production deployment and iterative optimization. The module is designed to provide a comprehensive understanding of building, refining, and deploying efficient RAG systems, integrating hands-on examples with theoretical knowledge to prepare students for real-world applications.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## Fine-tuning vs RAG; Introduction to Activeloop’s Deep Memory;
In this lesson, students will explore various optimization techniques to maximize the performance of large language models (LLMs), such as prompt engineering, fine-tuning, and retrieval-augmented generation (RAG). The lesson begins with identifying the benefits and challenges of each method. It further examines the limitations of RAG systems and introduces Activeloop's Deep Memory as a solution to these challenges, particularly in improving retrieval precision for user queries. Students will see a step-by-step guide on how to implement Deep Memory in experimental workflows, including creating a synthetic training dataset and running inference with the trained Deep Memory model. A significant portion of the lesson is dedicated to hands-on examples using code to demonstrate the increased recall rates when Deep Memory is applied in a RAG system. The lesson concludes with a comparison of empirical data, highlighting the advantages of Deep Memory over traditional methods and emphasizing its role in advancing the capabilities of LLMs.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## **Mastering Advanced RAG Techniques with LlamaIndex**
In this lesson, students will learn about the advanced techniques and strategies that enhance the performance of Retrieval-Augmented Generation (RAG) systems, using LlamaIndex as a framework. They will explore the concepts of query construction, query expansion, and query transformation to refine the information retrieval process. Students will also be introduced to advanced strategies like reranking with Cohere Reranker, recursive retrieval, and small-to-big retrieval to further improve the quality and relevance of search results. The lesson includes hands-on examples of setting up a query engine from indexing to querying, as well as creating custom retrievers and utilizing reranking. The conclusion underlines the importance of these techniques and strategies in developing more efficient RAG-based applications.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## Production-Ready RAG Solutions with LlamaIndex
In this lesson, students will learn about the challenges, optimization strategies, and best practices for Retrieval-Augmented Generation (RAG) systems in production. The discussion includes dealing with dynamic data management, diverse representation in latent space, regulatory compliance, and model selections for system efficiency. The lesson emphasizes the importance of fine-tuning both the embedding models and the Language Large Models (LLMs) to improve retrieval metrics and generate more accurate responses. Additionally, students will explore the role of Intel® technologies in optimizing neural network models on CPUs, and they will acquire knowledge on utilizing generative feedback loops, hybrid searches, and the continuous evaluation of RAG system performance. Practical use cases, data management tools, and integration of metadata in retrieval steps are also highlighted, with LlamaIndex being presented as a comprehensive framework for building data-driven LLM applications.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## Iterative Optimization of LlamaIndex RAG Pipeline: A Step-by-Step Approach
In this lesson, you will learn the process of iteratively optimizing a LlamaIndex Retrieval-Augmented Generation (RAG) pipeline to enhance its performance in information retrieval and generating relevant answers. The lesson guides you through establishing a baseline pipeline, experimenting with retrieval values and embedding models like "text-embedding-ada-002" and "cohere/embed-english-v3.0," and incorporating techniques like reranking and deep memory to refine document selection. Additionally, you will learn about performance metrics, such as Hit Rate and Mean Reciprocal Rank (MRR), and evaluate faithfulness and relevancy of answers using GPT-4 as a judge. The lesson provides hands-on code examples for each optimization step and concludes with the overall enhancement observed in the RAG pipeline's accuracy.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## Use Deep Memory to Boost RAG Apps' Accuracy by up to +22%
In this lesson, you will be introduced to a practical example of Deep Memory. Students will learn about the limitations of current RAG systems, such as suboptimal retrieval accuracy, and explore the benefits of implementing Deep Memory. The lesson explains how Deep Memory provides a significant accuracy boost by optimizing the vector search process using a tailored index from labeled queries. Throughout the lesson, students will be guided through hands-on examples for adopting Deep Memory in their applications, including data loading, creating a relevance dataset, training, and evaluating methods. The lesson emphasizes the practical advantages of this approach, like higher retrieval quality, cost savings from reduced context size needs and compatibility with existing workflows.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## **How to Use Deep Memory with LlamaIndex to Get +15% RAG hit_rate Improvement for Question Answering on Docs?**
In this comprehensive tutorial, students will learn about improving the hit rate of Retrieval-Augmented Generators (RAGs) when answering questions from documentation by up to 15% or more using Activeloop's Deep Memory. The lesson covers dataset creation and ingestion using BeautifulSoup and LlamaIndex, training deep memory with synthetic queries, evaluating the performance improvement, and leveraging deep memory for real-world inference. By integrating a small neural network layer into the retrieval process, the tutorial demonstrates how to precisely match user queries with relevant data, significantly boosting the accuracy of returned information while maintaining minimal search latency. Students will get hands-on experience with Python libraries and AI models such as OpenAI's GPT-4 and vector store operations to create a more efficient and accurate RAG system.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## **Use Deep Memory with LangChain to Get Up to +22% Increase in Accurate Questions Answers to LangChain Code DB**
In this lesson, students will learn how to utilize Activeloop Deep Memory with Langchain to enhance the efficiency and accuracy of Retrieval-Augmented Generation (RAG) systems by parsing documentation, creating datasets, generating synthetic queries, training a retrieval model, evaluating performance, and ultimately integrating Deep Memory into RAG-powered Language Learning Model (LLM) applications. They'll be guided through the practical steps involved in setting up this system, including library installation, data scraping and transformation, model training and evaluation, and even cost-saving measures, all while focusing on the balance between recall, cost, and latency in AI retrieval tasks.


# Module 2 Introduction - Advanced Retrieval Augmented Generation
## Deep Memory for RAG Applications Across Legal, Financial, and Biomedical Industries
In this comprehensive lesson, students will learn how to enhance RAG systems using Deep Memory in conjunction with LLMs for applications within legal, financial, and biomedical sectors. Students will be guided through the process of preparing datasets, including gathering and chunking data as well as question and relevance score generation using LLMs. The lesson emphasizes the significant performance improvements offered by Deep Memory, such as an increase in retrieval accuracy without compromising search time, and demonstrates how to integrate and test this feature with real datasets—Legalbench, FinQA, and CORD-19. Additionally, students will gain insight into the practical implementation of Deep Memory through code examples and explore the advantages of Deep Memory over classic retrieval methods.


# Module 1 Introduction - Basics of Retrieval Augmented Generation with Langchain and LlamaIndex
This module covers the basic concepts of Langchain and Llamaindex and prepares you to build a basic RAG application with both frameworks and help you decide which tool to select for your use case (spoiler: they both have their utility!). This is a recap of the Langchain concepts covered in the earlier **[LangChain & Vector Databases in Production](https://learn.activeloop.ai/courses/langchain)** course, together with a brief introduction to the Llamaindex framework. It also contains a clear summary of the different strengths and focus of each framework. This course will be focussed on advanced RAG topics so we recommend taking our earlier course and reading examples from LangChain and Llamaindex documentation to complement this module.


# Module 1 Introduction - Basics of Retrieval Augmented Generation with Langchain and LlamaIndex
## LangChain: ****Basic Concepts Recap****
In this lesson, students will recap on the functionalities and components of LangChain, a framework designed to work with generative AI and LLM-based applications. This material was covered in depth with code and project examples in our earlier course **[LangChain & Vector Databases in Production](https://learn.activeloop.ai/courses/langchain).** Students will be introduced to crucial preprocessing techniques like document loading and chunking, understanding the indexing of document segments and embedding models, as well as the structure and functionality of chains, memory modules, and vector stores. Additionally, students will gain insight into working with chat models, LLMs, embedding models, and constructing sequential chains to automate complex interactions.


# Module 1 Introduction - Basics of Retrieval Augmented Generation with Langchain and LlamaIndex
## **LlamaIndex Introduction: Precision and Simplicity in Information Retrieval**
In this lesson, students will learn about the LlamaIndex framework, designed to enhance the capabilities of Large Language Models by integrating them with Retrieval-Augmented Generation (RAG) systems. The framework allows LLM-based applications to fetch accurate and relevant information using vector stores, connectors, nodes, and index types for better-informed responses. The lesson covers vector stores and their importance in semantic search, the role of data connectors and LlamaHub in data ingestion, the creation of node objects from documents, and the indexing of data for quick retrieval. Students will also learn about the practical construction and usage of query engines, routers, and the distinction between saving indexes locally and on the cloud. Finally, it compares LlamaIndex with the LangChain frameworks and concludes by discussing the practical application and effectiveness of LlamaIndex in LLM applications.


# Module 1 Introduction - Basics of Retrieval Augmented Generation with Langchain and LlamaIndex
## Chat with Your Code: LlamaIndex and Activeloop Deep Lake for GitHub Repositories
In this project lesson, students will learn how to use LlamaIndex in conjunction with Activeloop Deep Lake to index GitHub repositories, enabling interaction with codebases through natural language queries. They will understand both tools' core functionalities, the synergy between data structuring and optimized storage, and the setup process for integrating these technologies. The lesson will guide students through installing necessary packages, setting up a Python virtual environment, loading and parsing GitHub repository data, building an index, and querying this index using a combination of LlamaIndex and Deep Lake. Additionally, the lesson covers the customization and flexibility of LlamaIndex's API for tailored data retrieval and response synthesis, and it concludes by comparing LlamaIndex with LangChain for building chatbots with external data.


Crafting AI Assistants via OpenAI and Hugging Face

# Introduction
In this lesson, we will explore the Assistant APIs from OpenAI. We will learn about the primary features of the Assistants API, including the Code Interpreter, Knowledge Retrieval, and Function Calling capabilities. We share a hands-on example to demonstrate the integration of the Code Interpreter with an existing Assistant. The example will show how to enhance an Assistant's ability to provide technical solutions by executing Python code, thus reducing LLM “hallucinations.” We will also introduce other advanced technologies from OpenAI, such as Whisper, Dalle-3, Speech to Text, and the GPT-4 vision API. These tools are useful for anyone looking to develop sophisticated AI assistants using a variety of APIs. Then, we will learn how to use the free Hugging Face Inference API to get access to the thousands of models hosted on their platform. By the end of this lesson, you will have gained a solid understanding of how to apply these technologies in your AI projects effectively.


# Open AI Assistant’s **Built-in Functionalities**
[The OpenAI Assistants API](https://platform.openai.com/docs/assistants/overview) includes three main functionalities: Code Interpreter, Retrieval, and Function Calling. **`[Code Interpreter](https://platform.openai.com/docs/assistants/tools/code-interpreter)`**: This functionality allows the Assistant to generate and run Python code in a sandboxed execution environment. The Assistant can use Code Interpreter automatically from your conversation or when you upload a file with data. It's a tool that transforms the LLM into a more accurate computational problem-solver that can handle tasks like solving complex math equations. It can also generate files with data and images of graphs from the same Python code. It's a useful way to trust the output from the assistant and a great tool when analyzing data. **`[Knowledge Retrieval](https://platform.openai.com/docs/assistants/tools/knowledge-retrieval)`**: This is OpenAI’s own [retrieval augmented generation (RAG)](https://arxiv.org/abs/2005.11401) system offered as part of the Assistants API. It allows multiple uploads. Once the files are uploaded and passed to the Assistant, OpenAI will automatically chunk your documents, index them, store the embeddings, and implement vector search to retrieve relevant content to answer user queries. **`[Function Calling](https://platform.openai.com/docs/assistants/tools/function-calling)`**: Function calling allows you to describe functions or tools to the Assistant and have it return the functions that need to be called along with their arguments. It's a powerful way to add new capabilities to your Assistant.


# How To Set Up an Assistant
You have two distinct pathways depending on your needs and expertise: - **[Assistants Playground](https://platform.openai.com/playground)**: Ideal for those looking to get a feel for the Assistant's capabilities without going into complex integrations. - **[Detailed Integration through the API](https://platform.openai.com/docs/assistants/overview)**: Best suited for those who require a more customized and in-depth setup. **STEP-BY-STEP ASSISTANT CREATION:** 1. **Creating an `Assistant`**: **Purpose**: An Assistant object represents an entity/agent that can be configured to respond to users’ messages in different ways using several parameters. **Model Selection**: you can specify any version of GPT-3.5 or GPT-4 models, including fine-tuned models. OpenAI recommends using its [latest models](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo) with the Assistants API for best results and maximum compatibility with tools. Thus, choose between **`gpt-3.5-turbo-1106`** or **`gpt-4-1106-preview`** models. **Tools:** The Assistant supports the **Code Interpreter** for technical queries that require Python code execution or **Knowledge Retrieval** to augment the Assistant with proprietary external information. 2. **Setting up a `Thread`**: **Role**: A Thread acts as the foundational unit of user interaction. It can be seen as a single **conversation**. Pass any user-specific context and files in this thread by [creating Messages](https://platform.openai.com/docs/api-reference/messages/createMessage). ```python thread = client.beta.threads.create() ``` **Customization**: In Thread, ingest user-specific contexts or attach necessary files so each conversation is unique and personalized. Threads don’t have a size limit. You can add as many messages as you want to a conversation/Thread. The Assistant will ensure that requests to the model fit within the maximum context window, using relevant optimization techniques used in ChatGPT, such as truncation. 3. **Adding a `Message`**: **Definition**: Messages are user inputs, and the Assistant’s answers are appended to a Thread. User inputs can be questions or commands. **Function**: They serve as the primary mode of communication between the user and the Assistant. ```python message = client.beta.threads.messages.create( thread_id=thread.id, role="user", content="I need to solve the equation `3x + 11 = 14`. Can you help me?" ) ``` Messages can include **text, images, and other files**. Messages are stored as a list on the Thread. Using GPT-4 with Vision is not supported here. You can upload images and have them [processed via retrieval](https://platform.openai.com/docs/assistants/tools/knowledge-retrieval). 4. **Executing with `Run`**: **Activation**: For the Assistant to respond to the user message, you must [create a Run](https://platform.openai.com/docs/api-reference/runs/createRun). The Assistant will then automatically decide what previous Messages to include in the context window for the model. <aside> ⚠ NOTE: You can [optionally pass additional instructions](https://platform.openai.com/docs/api-reference/runs/createRun#runs-createrun-instructions) to the Assistant while creating the Run, but these **will override** the default instructions of the Assistant! </aside> **Process**: The Assistant processes the entire Thread, employs its tools if required, and formulates an appropriate response. During its run, the Assistant can call tools or create Messages. Examining Run Steps allows you to check how the Assistant is getting to its final results. 5. **Displaying the `Response`**: **Outcome**: The assistant’s response to a Run: ```python messages = client.beta.threads.messages.list( thread_id=thread.id ) ``` These responses are displayed to the user! During this Run, the Assistant added two new Messages to the Thread.


# How To Set Up an Assistant
## **ASSISTANT’S CORE MECHANISM:**
Creating an Assistant only requires specifying the **`model`**. But you can further customize the behavior of the Assistant: 1. Use the **`instructions`** parameter to guide the personality of the Assistant and define its goals. Instructions are similar to system messages in the Chat Completions API. 2. Use the **`tools`** parameter to give the Assistant access to up to 128 tools in parallel. You can give it access to OpenAI-hosted tools (Conde Interpreter, Knowledge Retrieval) or call third-party tools via **`function calling`**. 3. Use the **`file_ids`** parameter to give the tools access to files. Files are uploaded using the **`File`** U[pload endpoint](https://platform.openai.com/docs/api-reference/files/create).