# Enhanced RAG with Deep Memory
## Step by Step - Training a **Deep Memory Model**
the query generation process with a desired size of 40 queries/questions. ```python questions, relevances = generate_queries(docs, ids, n=40) print(len(questions)) #40 print(questions[0]) ``` You will have a list of generated questions and the associated contexts by running the two cells above. 1. Launch Deep Memory Training ```python # Train deep memory job_id = db.vectorstore.deep_memory.train( queries=questions, relevance=relevances, ) ``` 1. Starting DeepMemory training job Your Deep Lake dataset has been successfully created! Preparing training data for DeepMemory: Creating 20 embeddings in 1 batches of size 20:: 100%|██████████| 1/1 [06:36<00:00, 396.77s/it] DeepMemory training job started. Job ID: 657b3083d528b0fd224173c6 ```python # During training you can check the status of the training run db.vectorstore.deep_memory.status(job_id="657b3083d528b0fd224173c6") ``` ```bash -------------------------------------------------------------- | 657b3083d528b0fd224173c6 | -------------------------------------------------------------- | status | completed | -------------------------------------------------------------- | progress | eta: 0.9 seconds | | | recall@10: 60.00% (+25.00%) | -------------------------------------------------------------- | results | recall@10: 60.00% (+25.00%) | -------------------------------------------------------------- ``` We see an increase of 25% in recall@10 after finetuning. 1. Run a Deep Memory-enabled inference by setting `deep_memory=True`. ```python # Define your question query = "What is the role of the 'encode' method in tokenizers?" # Run similarity search db.similarity_search(query=query, deep_memory=True, k=1) # Perform the search within search_results = db.similarity_search(query="What is the role of the 'encode' method in tokenizers?", deep_memory=False, k=3) # Print the search results print(search_results) # Search results: [Document(page_content='A tokenizer is in charge of preparing the inputs for a model. The library contains tokenizers for all the models. Most\nof the tokenizers are available in two flavors: a full python implementation and a “Fast” implementation based on the\nRust library 🤗 Tokenizers. The “Fast” implementations allows:\na significant speed-up in particular when doing batched tokenization and additional methods to map between the original string (character and words) and the token space (e.g. getting the\nindex of the token comprising a given character or the span of characters corresponding to a given token).\nThe base classes PreTrainedTokenizer and PreTrainedTokenizerFast\nimplement the common methods for encoding string inputs in model inputs (see below) and instantiating/saving python and\n“Fast” tokeni', metadata={'Unnamed: 0': 16245, 'title': 'Tokenizer', 'url': 'https://huggingface.co/docs/transformers/main/en/main_classes/tokenizer#tokenizer', 'source': 'hf_transformers'})] ``` 1. Now, let's run a quantitative evaluation on another set of synthetically generated test queries. ```python # Generate validation queries validation_questions, validation_relevances = generate_queries(docs, ids, n=100) # Launch the evaluation function recalls = db.vectorstore.deep_memory.evaluate( queries=validation_questions, relevance=validation_relevances, embedding_function=openai_embeddings.embed_documents, ) ``` ```bash Embedding queries took 0.82 seconds ---- Evaluating without Deep Memory ---- Recall@1: 27.0% Recall@3: 42.0% Recall@5: 42.0% Recall@10: 50.0% Recall@50: 67.0% Recall@100: 72.0% ---- Evaluating with Deep Memory ---- Recall@1: 32.0% Recall@3: 45.0% Recall@5: 48.0% Recall@10: 55.0% Recall@50: 69.0% Recall@100: 73.0% ``` Even with our new test dataset, we notice higher recall values using Deep Memory. Comparing these results with the training dataset highlights how a **query-context dataset** has better quality and represents your use case.


Fine-tuning vs RAG; Introduction to Activeloop’s D

# Conclusion
In this lesson, we explored the optimization techniques for large language models, covering prompt engineering as a first way to maximize LLM performance, fine-tuning, and Retrieval-Augmented Generation (RAG) for integrating external, up-to-date knowledge. We also discussed combining fine-tuning with RAG for complex, domain-specific applications requiring considerable resources. A significant focus was on Activeloop's Deep Memory, which was integrated into RAG systems to enhance embedding retrieval accuracy. Deep Memory outperforms traditional methods like BM25 using lexical search and vector search using cosine similarity. We demonstrated it by getting higher recall values. It also efficiently reduces token usage in LLM prompts compared to query reformulation or transformation. This approach addresses key embedding retrieval challenges and signals a promising future for increasingly capable and versatile LLMs. --- **RESOURCES** - Colab with the lesson code [Google Colaboratory](https://colab.research.google.com/drive/1dgNwwCuA4zqtQDRuyzqd53PbG3PlRHul#scrollTo=lXJYCiwNPbOL) - A Survey of Techniques for Maximizing LLM Performance from OpenAI [A Survey of Techniques for Maximizing LLM Performance](https://www.youtube.com/watch?v=ahnGLM-RC1Y) - Deep Memory Blog Post [Use Deep Memory to Boost RAG Apps' Accuracy by up to +22%](https://www.activeloop.ai/resources/use-deep-memory-to-boost-rag-apps-accuracy-by-up-to-22/) - Deep Memory Tutorial [Improving Search Accuracy using Deep Memory](https://docs.activeloop.ai/example-code/tutorials/vector-store/improving-search-accuracy-using-deep-memory) - Llama-index and Deep Memory [How to get +15% RAG hit_rate improvement for question answering on documentation? - LlamaIndex 🦙 0.9.15.post2](https://gpt-index.readthedocs.io/en/stable/examples/retrievers/deep_memory.html)


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
This project exploits the most advanced artificial intelligence techniques, specifically those relating to NLP and computer vision, and is made available to healthcare, allowing the user to take a photo of a pill and find information about it. The goal is to upload a photo of a pill and recognize it. To obtain noteworthy results it was decided to divide the problem into different phases which will be explained in the following paragraphs.


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Segmentation
Initially the image is segmented so that the background does not generate false positives or false negatives, for this phase an algorithm called FastSAM was used. This algorithm is able to perform well on both GPU and CPU and has some characteristics to consider: - **Real-time Solution:** FastSAM capitalizes on the computational prowess of Convolutional Neural Networks (CNNs) to offer a real-time solution for the 'segment anything' task. This feature is particularly beneficial for industrial applications where swift and efficient results are paramount. - **Practical Applications:** FastSAM introduces a novel and practical approach for a wide array of vision tasks, delivering results at a speed that is tens to hundreds of times faster than existing methods, revolutionizing the field. - **Based on YOLOv8-seg:** At its core, FastSAM utilizes YOLOv8-seg, a sophisticated object detector with an integrated instance segmentation branch, enabling it to efficiently generate segmentation masks for all instances in an image. - **Efficiency and Performance:** FastSAM stands out by dramatically lowering computational and resource usage while maintaining high-quality performance. It rivals the performance of SAM but requires substantially Here is an example of how segmentation is performed, which will allow the algorithm of the next phase to focus only on the important part of the image: ![https://github.com/efenocchi/PillSearch-Activeloop/raw/main/normal_image.webp](https://github.com/efenocchi/PillSearch-Activeloop/raw/main/normal_image.webp) ![https://github.com/efenocchi/PillSearch-Activeloop/raw/main/image_masked.webp](https://github.com/efenocchi/PillSearch-Activeloop/raw/main/image_masked.webp) After performing the segmentation, we examine which image in our dataset is similar to the one we just segmented. This practice is performed via a neural network called ResNet-18 and allows you to capture important image information and use it in the similarity search phase. It is important to underline that even the images of the dataset that are being compared were all initially segmented to avoid the problem described above. If you want to know more about how this technique works you can read the article [ResNet-18 from Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385).


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Visual Similarity
ResNet-18 is a compelling choice for computing visual similarity between images, such as in our application for identifying similarities between pill images. Its effectiveness lies in its architecture and its capability for feature extraction. Here's a breakdown of why ResNet-18 is well-suited for this task: - **Deep Residual Learning:** ResNet-18, a variant of the Residual Network (ResNet) family, incorporates deep residual learning. In ResNet-18, there are 18 layers, including convolutional layers, batch normalization, ReLU activations, and fully connected layers. - **Feature Extraction:** One of the primary strengths of ResNet-18 is its feature extraction capability. In the context of pill images, ResNet-18 can learn and identify intricate patterns, shapes, and colors that are unique to different pills. During the forward pass, as the image goes through successive layers, the network learns hierarchically more complex and abstract features. Initial layers might detect edges or basic shapes, while deeper layers can identify more specific features relevant to different types of pills. - **Efficiency and Speed:** Despite being deep, ResNet-18 is relatively lightweight compared to other deeper models (like ResNet-50 or ResNet-101). This makes it a good choice for applications where computational resources or inference time might be a concern, without significantly compromising on the accuracy of feature extraction. - **Performance in Feature Embedding:** For tasks like visual similarity, it's essential to have a robust feature embedding, which is a compressed representation of the input image. ResNet-18, due to its deep structure, can create rich, discriminative embeddings. When we input two pill images, ResNet-18 processes them to produce feature vectors. The similarity between these vectors can then be computed using metrics like cosine similarity or Euclidean distance. The closer these vectors are in the feature space, the more similar the images are. This similarity is performed directly in Activeloop's Deep Lake Vector Stores, simply take the input image and pass it to Activeloop and it will return the `n` most similar images that were found. If you want to delve deeper into this topic you can find a guide on `Activeloop` at the [following link](https://docs.activeloop.ai/example-code/getting-started/vector-store/step-4-customizing-vector-stores). Going into a little more detail in this project we can see how visual similarity search is not the only one that has been used. Once the `n` most similar images have been returned they are split into two groups: - the `3` most similar images - the remaining `n - 3` images To talk about the second type of similarity we take an intermediate step and show what the user interface looks like.


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Text Extraction and Identification
In order to extract the text engraved in the pill, purely computer vision approaches were initially used and subsequently GPT-4 vision was chosen. This SAAS (software as a service) allows us to recover the text present in the pill which will then be compared with those present in the database. If a perfect match occurs, this pill will be identified as the input one, otherwise the closest image will be chosen.


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Gradio Interface
Gradio is an open-source Python library that provides an easy way to create customizable web interfaces for machine learning models. In this pill project, we have utilized Gradio to build a user-friendly interface that allows users to upload pill images and interact with the pipeline. The results are then displayed back to the user through the Gradio interface and are divided into two different columns: - in the first there are the `3 images` most similar to the input one - in the second there are `3 similar images` to which we must pay attention because they have a description of the pill as different as possible from the one inserted. It is necessary to specify that since the input image has no text but is just an image, the description taken is that of the image whose unique identification code is equal to one of those present in the dataset or, in case there is no exact match, that of the image which is absolutely most similar to the input one. The interface that the user will initially load is as follows: ![https://github.com/efenocchi/PillSearch-Activeloop/raw/main/gradio_ui.webp](https://github.com/efenocchi/PillSearch-Activeloop/raw/main/gradio_ui.webp)


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Activeloop Visualizer
To show the images returned after the search, the Activeloop rendering engine called Visualizer was used. This functionality allows us to view the data present in the Deep Lake by loading it in HTML format. It was then possible to embed the Activeloop visualization engine into our RAG applications. ![https://github.com/efenocchi/PillSearch-Activeloop/raw/main/gradio_ui_results.webp](https://github.com/efenocchi/PillSearch-Activeloop/raw/main/gradio_ui_results.webp) In this case for each cell we chose to return only the image we needed but if desired it is possible to view all the images in a single window and move between them using the cursor. If you want to delve deeper into this functionality and integrate it into your application via Python or Javascript code you can find the guide [here](https://docs.activeloop.ai/technical-details/visualizer-integration). Now we can move on to the last phase, the similarity search via description of the pill.


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Advanced Retrieval Strategies
A technique that is becoming increasingly popular in this period is the Retrieval-Augmented Generation (RAG) which enhances large language models (LLMs) by integrating external authoritative knowledge sources beyond their initial training datasets for response generation. LLMs, trained on extensive data and utilizing billions of parameters, excel in tasks such as question answering, language translation, and sentence completion. RAG builds upon these strengths, tailoring LLMs to particular domains or aligning them with an organization's internal knowledge, without necessitating model retraining. This method provides a cost-efficient solution to refine LLM outputs, ensuring their continued relevance, precision, and utility across diverse applications. There are five key stages within RAG, which in turn will be a part of any larger application you build. These are: - **Loading:** This involves transferring data from its original location, such as text files, PDFs, websites, databases, or APIs, into your processing pipeline. LlamaHub offers a wide range of connectors for this purpose. - **Indexing:** This process entails developing a data structure that supports data querying. In the context of LLMs, it typically involves creating vector embeddings, which are numerical representations of your data's meaning, along with various metadata strategies to facilitate the accurate retrieval of contextually relevant data. - **Storing:** After indexing, it's common practice to store the index and other metadata. This step prevents the need for re-indexing in the future. - **Querying:** Depending on the chosen indexing method, there are multiple ways to deploy LLMs and LlamaIndex structures for querying, including options like sub-queries, multi-step queries, and hybrid approaches. - **Evaluation:** This crucial phase in the pipeline assesses its effectiveness against other methods or following modifications. Evaluation provides objective metrics of the accuracy, reliability, and speed of your system's responses to queries. These processes can be easily and clearly represented by the following diagram in the LLamaIndex guide: [https://camo.githubusercontent.com/32c0e19ac812d4e4d1d6fb581e9fdbbd7a1e41b5f95ec00658e368e09b718062/68747470733a2f2f646f63732e6c6c616d61696e6465782e61692f656e2f6c61746573742f5f696d616765732f62617369635f7261672e706e67](https://camo.githubusercontent.com/32c0e19ac812d4e4d1d6fb581e9fdbbd7a1e41b5f95ec00658e368e09b718062/68747470733a2f2f646f63732e6c6c616d61696e6465782e61692f656e2f6c61746573742f5f696d616765732f62617369635f7261672e706e67) Since we have a description for each pill we used these descriptions as if they were documents in order to then be able to obtain the most similar ones (and therefore also the least similar ones) once the description of the input pill was passed to the model. After initially loading our data onto Activeloop's Deep Lake we can show how the data present in this space is used to do an Advanced Retrieval. ```python import DeepLakeVectorStore def create_upload_vectore_store( chunked_text: list, vector_store_path: Union[str, os.PathLike], filename: str, metadata: Optional[list[dict]] = None, ): vector_store = DeepLakeVectorStore( dataset_path=vector_store_path, runtime={"tensor_db": True}, overwrite=True, tensor_params=[ {"name": "text", "htype": "text"}, {"name": "embedding", "htype": "embedding"}, {"name": "filename", "htype": "text"}, {"name": "metadata", "htype": "json"}, ], ) vector_store = vector_store.vectorstore vector_store.add( text=chunked_text, embedding_function=embedding_function_text, filename=filename, embedding_data=chunked_text, rate_limiter={ "enabled": True, "bytes_per_minute": 1500000, "batch_byte_size": 10000, }, metadata=metadata if metadata else None, ) ``` For all subsequent steps we used LlamaIndex which is a data framework for LLM-based applications used to capture, structure and access private or domain-specific data. ### Indexing Phase Once we’ve ingested the data, LlamaIndex will help us index the data into a structure that’s easy to retrieve. This involves generating vector embeddings which are stored in a specialized database called a vector store, in our case we stored them into the Deep Lake Vector Store. Indexes can also store a variety of metadata about your data. Under the hood, Indexes store data in Node objects (which represent chunks of the original documents), and expose a Retriever interface that supports additional configuration and automation. This part is made up of two main blocks: - **Embedding**: we used OpenAI's `text-embedding-ada-002` as the embedding model - **Retriever**: we tried different approaches which will be described below ### Retriever Phase Retrievers are responsible for fetching the most relevant context given a


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Advanced Retrieval Strategies
user query (or chat message). It can be built on top of indexes, but can also be defined independently. It is used as a key building block in query engines (and Chat Engines) for retrieving relevant context. **Vector Store Index** A VectorStoreIndex is by far the most frequent type of Index you’ll encounter. The Vector Store Index takes your Documents and splits them up into Nodes. It then creates vector embeddings of the text of every node, ready to be queried by an LLM. The vector store index stores each Node and a corresponding embedding in a Vector Store. [https://camo.githubusercontent.com/18409f41016d52d6ad393091d52107485f8eaf669e3ada64206c5b9c9fae9440/68747470733a2f2f646f63732e6c6c616d61696e6465782e61692f656e2f6c61746573742f5f696d616765732f766563746f725f73746f72652e706e67](https://camo.githubusercontent.com/18409f41016d52d6ad393091d52107485f8eaf669e3ada64206c5b9c9fae9440/68747470733a2f2f646f63732e6c6c616d61696e6465782e61692f656e2f6c61746573742f5f696d616765732f766563746f725f73746f72652e706e67) *[Source Image](https://docs.llamaindex.ai/en/latest/_images/vector_store.png)* Querying a vector store index involves fetching the top-k most similar Nodes, and passing those into our Response Synthesis module. [https://camo.githubusercontent.com/155f03542ce748b38e2769f1bc15638afb035d9efd28be6faddf22ae6c49508c/68747470733a2f2f646f63732e6c6c616d61696e6465782e61692f656e2f6c61746573742f5f696d616765732f766563746f725f73746f72655f71756572792e706e67](https://camo.githubusercontent.com/155f03542ce748b38e2769f1bc15638afb035d9efd28be6faddf22ae6c49508c/68747470733a2f2f646f63732e6c6c616d61696e6465782e61692f656e2f6c61746573742f5f696d616765732f766563746f725f73746f72655f71756572792e706e67) *[Source Image](https://docs.llamaindex.ai/en/latest/_images/vector_store_query.png)* **BM25** BM25 is a popular ranking function used by search engines to estimate the relevance of documents to a given search query. It's based on probabilistic models and improves upon earlier models like TF-IDF (Term Frequency-Inverse Document Frequency). BM25 considers factors like term frequency and document length to provide a more nuanced approach to relevance scoring. It handles the issue of term saturation (where the importance of a term doesn't always increase linearly with frequency) and length normalization (adjusting scores based on document length to prevent bias toward longer documents). BM25's effectiveness in various search tasks has made it a standard in information retrieval. To use this retriever we need to take documents from Activeloop's Deep Lake and transform them into nodes, these nodes will then be returned in an orderly manner once a question is asked. This process, as already mentioned previously, exploits the similarity between the description of the pill (which will be passed as a query) and the description of the `n - 3` most similar pills. In the code below we used the name of the images to make a specific query that returned only the values of those images and by extrapolating their description we created the nodes and indexes. ```python def get_index_and_nodes_after_visual_similarity(filenames: list): vector_store = load_vector_store(vector_store_path=VECTOR_STORE_PATH_DESCRIPTION) conditions = " or ".join(f"filename == '{name}'" for name in filenames) tql_query = f"select * where {conditions}" filtered_elements = vector_store.vectorstore.search(query=tql_query) chunks = [] for el in filtered_elements["text"]: chunks.append(el) string_iterable_reader = download_loader("StringIterableReader") loader = string_iterable_reader() documents = loader.load_data(texts=chunks) node_parser = SimpleNodeParser.from_defaults(separator="\n") nodes = node_parser.get_nodes_from_documents(documents) # To ensure same id's per run, we manually set them. for idx, node in enumerate(nodes): node.id_ = f"node_{idx}" llm = OpenAI(model="gpt-4") service_context = ServiceContext.from_defaults(llm=llm) index = VectorStoreIndex(nodes=nodes) return index, nodes, service_context, filtered_elements ``` Since we have the `n` most similar images (obtained in the previous step through the visual similarity), we can extract the description of these `n` images and use them to generate the nodes. ### Why do we need to care about different retrieval methods and how are they different from each other? RAG (Retrieval-Augmented Generation) systems retrieve relevant information from a given knowledge base, thereby allowing it to generate factual, contextually relevant, and domain-specific information. However, RAG faces a lot of challenges when it comes to effectively retrieving relevant information and generating high-quality responses. Traditional search engines work by parsing documents into chunks and indexing these chunks. The algorithm then searches this index for relevant results based on a user’s query. Retrieval Augmented Generation is a new paradigm in machine learning that uses large language models (LLMs) to improve search and discovery. The LLMs, like the GPT-4, generate relevant content based on context. The advanced technique utilized in this project is the Hybrid Search. It is a technique that combines multiple search algorithms to improve the accuracy and relevance of search results. It uses the best


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Advanced Retrieval Strategies
features of both keyword-based search algorithms with vector search techniques. By leveraging the strengths of different algorithms, it provides a more effective search experience for users. ### Hybrid Fusion Retriever In advanced technique, we merge the vector store based retriever and the BM25 based retriever. This will enable us to capture both semantic relations and keywords in our input queries. Since both of these retrievers calculate a score, we can use the reciprocal rerank algorithm to re-sort our nodes without using additional models or excessive computation. We can see the scheme in the image below taken from the LlamaIndex guide: ![https://github.com/efenocchi/PillSearch-Activeloop/raw/main/hybrid_approach.webp](https://github.com/efenocchi/PillSearch-Activeloop/raw/main/hybrid_approach.webp) ### BM25 Retriever + Re-Ranking technique (classic approach with BM25) In this first case we use a classic retriever based only on the **BM25 retriever**, the nodes generated by the query will then be re-ranked and filtered. This allows us to keep the intermediate top-k values large and filter out unnecessary nodes. ```python _, nodes, service_context = get_index_and_nodes_from_activeloop( vector_store_path=VECTOR_STORE_PATH_BASELINE ) ``` These nodes will then be used by the bm25 retriever. It is useful to note that the index variable is not currently used because bm25 manages this entire part internally. ``` bm25_retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10) ``` Now that we have the nodes we need to obtain the similarity by passing the description of the input image as a query. ``` nodes_bm25_response = bm25_retriever.retrieve(description) ``` ClassicRetrieverBM25 is an object that we used to manage the creation of the BM25-based retriever in a more orderly way. ```python class ClassicRetrieverBM25(BaseRetriever): def __init__(self, bm25_retriever): self.bm25_retriever = bm25_retriever super().__init__() def _retrieve(self, query, **kwargs): bm25_nodes = self.bm25_retriever.retrieve(query, **kwargs) all_nodes = [] node_ids = set() for n in bm25_nodes: if n.node.node_id not in node_ids: all_nodes.append(n) node_ids.add(n.node.node_id) return all_nodes ``` The final part of this process is generated by the re-renker which takes care of ordering the nodes according to their score. ```python reranker = SentenceTransformerRerank(top_n=4, model="BAAI/bge-reranker-base") # nodes retrieved by the bm25 retriever with the reranker reranked_nodes_bm25 = reranker.postprocess_nodes( nodes_bm25_response, query_bundle=QueryBundle(QUERY), ) ``` ### Advanced - Hybrid Retriever + Re-Ranking technique with BM25 and the vector retriever Here we extend the base retriever class and create a custom retriever that always uses the **vector retriever** and **BM25 retreiver**. In this test, the previous approach was used but the vector store was added as a retriever which uses the index variable returned by the `get_index_and_nodes_after_visual_similarity` function to manage the nodes. At the beginning, we create the standard retriever: ```python index, nodes, _ = get_index_and_nodes_from_activeloop( vector_store_path=VECTOR_STORE_PATH_COMPLETE_SEQUENTIALLY ) self.vector_retriever = index.as_retriever(similarity_top_k=2) self.bm25_retriever = BM25Retriever.from_defaults( nodes=nodes, similarity_top_k=10 ) ``` The nodes will then be calculated both via the vector store and bm25 and once they are all put together they will be re-ranked and filtered. This process allows us to get the best of both models and filter out the `k best nodes`. ```python reranked_nodes_bm25 = self.reranker.postprocess_nodes( self.nodes_bm25_response, query_bundle=QueryBundle(QUERY), ) print("Reranked Nodes BM25\n\n") for el in reranked_nodes_bm25: print(f"{el.score}\n") reranked_nodes_vector = self.reranker.postprocess_nodes( self.nodes_vector_response, query_bundle=QueryBundle(QUERY), ) print("Reranked Nodes Vector\n\n") for el in reranked_nodes_vector: print(f"{el.score}\n") unique_nodes = keep_best_k_unique_nodes( reranked_nodes_bm25, reranked_nodes_vector ) print("Unique Nodes\n\n") for el in unique_nodes: print(f"{el.id} : {el.score}\n") ``` ### Advanced - Hybrid Retriever + Re-Ranking technique with BM25 and the vector retriever and QueryFusionRetriever In the last case we can see how through the QueryFusionRetriever object the entire process described previously can be represented with a single function. We fuse our index with a BM25 based retriever, this will enable us to capture both semantic relations and keywords in our input queries. Since both of these retrievers calculate a score, we can use the reciprocal rerank algorithm to re-sort our nodes without using an additional models or excessive computation.


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Advanced Retrieval Strategies
```python from llama_index.retrievers import BM25Retriever vector_retriever = index.as_retriever(similarity_top_k=2) bm25_retriever = BM25Retriever.from_defaults( docstore=index.docstore, similarity_top_k=2 ) ``` Here we can create our fusion retriever, which will return the top-2 most similar nodes from the 4 returned nodes from the retrievers: ```python from llama_index.retrievers import QueryFusionRetriever retriever = QueryFusionRetriever( [vector_retriever, bm25_retriever], similarity_top_k=2, num_queries=4, # set this to 1 to disable query generation mode="reciprocal_rerank", use_async=True, verbose=True, ) ``` Finally, we can perform the query search: ``` retriever.retrieve(description) ```


# Advanced Retrieval Augmented Generation (RAG) for Pill Searching
## Metrics and Conclusions
Advanced retrievers, like the ones we've been discussing, represent a significant leap in our ability to process and analyze vast amounts of data. These systems, armed with sophisticated algorithms like BM25, vector store, and the latest developments in embedding models from OpenAI, are not just tools; they are gateways to a new era of information accessibility and knowledge discovery. The power of these retrievers lies in their understanding of context, their ability to sift through massive data sets to find relevant information, and their efficiency in providing accurate results. They have become indispensable in sectors where precision and speed are essential. In the healthcare sector, for example, their application to identify and cross-reference medical information can represent a game changer, improving both the quality of care and patient safety. All tested models work very well for our use case and in favor of this thesis are the following metrics: **BM25 Retriever + Re-Ranking technique (classic approach with BM25)** ``` retrievers hit_rate mrr top-2 eval 0.964643 0.944501 ``` **Advanced - Hybrid Retriever + Re-Ranking technique with BM25 and the vector retriever** ``` retrievers hit_rate mrr top-2 eval 0.975101 0.954078 ``` **Advanced - Hybrid Retriever + Re-Ranking technique with BM25 and the vector retriever and QueryFusionRetriever** ``` retrievers hit_rate mrr top-2 eval 0.977138 0.954235 ``` Where the **hit_rate** and **MRR (Mean Reciprocal Rank)** are two metrics commonly used to evaluate the performance of information retrieval systems, search algorithms, and recommendation systems. **Hit Rate:** The hit rate is a measure of accuracy, specifically the proportion of times a system successfully retrieves relevant documents or items. It's often used in the context of recommendation systems to evaluate if any of the recommended items are relevant. A hit is usually defined by whether the relevant item appears in the top-N recommendations or retrieval results. For instance, if the system provides 10 recommendations, and at least one of them is relevant, it's considered a hit. The hit rate is the number of hits divided by the total number of recommendations or queries made. **Mean Reciprocal Rank (MRR):** MRR is a statistic that measures the average of the reciprocal ranks of results for a sample of queries. The reciprocal rank of a query response is the multiplicative inverse of the rank of the first correct answer. For example, if the first relevant document for a query is located at the third position in a list of ranked items, the reciprocal rank is 1/3. MRR is calculated by taking the average of the reciprocal ranks across all queries. It gives a higher score to systems where the relevant item appears earlier in the recommendation or search results list, therefore considering the ranking of results, unlike hit rate which is binary. Both metrics are critical for assessing how effectively a system presents the relevant information to users, with the hit rate focusing on presence in the results and MRR on the rank or position of the first relevant result.


Query Your DataFrames with Powerful Large Language Models using LangChain.

What technologies are we using? The vector database we will use is ChromaDB. It is possibly the most well-known option among open-source vector databases. For the models, we have chosen two in the Hugging Face library. The first one is dolly-v23b and the other flan-t5-large. It's worth noting that these are not only two different models but also trained for different functions. T5 is a text2text-generation model family, designed to generate text based on the datasets they were trained on. They can be used for text generation, but their responses may not be very creative. On the other hand, Dolly is a pure text-generation family. These models tend to produce more creative and extensive responses. The star library is LangChain, an open-source platform that enables the creation of natural language applications harnessing the power of large language models. It allows us to chain inputs and outputs between these models and other libraries or products, such as databases or various plugins.


Let's start the project with LangChain. The code is available in a notebook on Kaggle. This article and notebook are part of a course on creating applications with large language models, which is available on my GitHub profile. If you don't want to miss any lessons or updates to the existing content, it's best to follow the repository. I will be publishing new lessons in the public repository as I complete them.


Installing and loading the libraries. If you are working in your personal environment and have already been testing these technologies, you may not need to install anything. However, if you are using Kaggle or Colab, you will need to install the following libraries: langchain: The revolutionary library that enables the creation of applications with large language models.sentence_transformers: We will have to generate embeddings of the text we want to store in the vector database, for which we require this library.chromadb: The vector database to be used. Notably, ChromaDB stands out for its user-friendly interface. In addition to these libraries, we will also import the two most used Python libraries in data science: pandas and numpy.


Load the Datasets. As I mentioned earlier, the notebook has been prepared to work with two different datasets. These datasets are the same ones used in the previous example of RAG (Retrieval Augmented Generation). RAG means using your data with large language models, commonly referred to as 'Interrogating Your Documents'. Both datasets are tabular and contain news-related information: Topic Labeled News Dataset.MIT AI News Published till 2023. The content of both datasets is similar, but the column names and stored information differ. Personally, I believe that using multiple datasets can provide a good value, allowing you to validate and generalize the results with different data sources. It would be beneficial to explore a third different dataset and replicate the functionality with it upon completing the notebook. As we are working with limited resources on Kaggle, it's essential to be mindful of memory constraints. Therefore, we won't be working with the entire dataset to avoid exceeding Kaggle's memory limit, which is 30 GB, if you don't use GPUs. Working with a smaller subset of the dataset will still allow us to explore and demonstrate the functionality of LangChain effectively while staying within the resource limitations. Let's take a look at the first two records of the topic-labeled-news-dataset. In the first dataset, we will use the title column as our document. Although the texts may not be very lengthy, it serves us as a perfect example. We can use it to search through the database of articles and find those that discuss a specific topic. We have created the DataFrame subset_news containing a portion of the news from the dataset. To use one dataset or the other, it's as simple as uncommenting the lines corresponding to the dataset we want to use. In each case, we adjusted the name of the column to be used as data and the number of records that the subset will contain. This approach allows us to easily switch between datasets.


Generate the Document from the DataFrame. To create the document, we will use with LangChain, using one of its loaders. For our example, we will use the DataFrameLoader, but there are various loaders available for a wide range of sources, such as CSVs, text files, HTML, JSON, PDFs, and even loaders for products like Confluence. Once we have the library loaded, we need to create the loader. To do this, we specify the DataFrame and the name of the column we want to use as the document's content. This information will be passed to the vector database, ChromaDB, where it will be stored and used by the language model when generating its responses. To create the document, we simply need to call the load function of the Loader. Let's take a look at the content of the document: As we can see, it has created a document where each page corresponds to the content of a record from the specified column. Additionally, we find the other data in the 'metadata' field, labeled with the column name. I encourage you to try with the other dataset and see how the data looks.


Creating the Embeddings. First, in case it's necessary, let's understand what Embedding is. It's simply a numerical representation of any data. In our specific case, it will be the numerical representation of the text to be stored. This numerical representation takes the form of vectors. A vector is simply a representation of a point in a multidimensional space. In other words, we don't have to visualize the point on a two-dimensional or three-dimensional plane, as we are used to. The vector can represent the point in any number of dimensions. For us, it may seem complicated or hard to imagine, but mathematically, there isn't much difference between calculating the distance between two points whether they are in two dimensions, three, or any number of dimensions. These vectors allow us to calculate the differences or similarities between them, making it possible to search for similar information very efficiently. The trick lies in determining which vectors we assign to each word, as we want words with similar meanings to be closer in distance than those with more different meanings. Hugging Face libraries take care of this aspect, so we don't have to worry too much. We just need to ensure consistent conversion for all the data to be stored and the queries to be performed. Let's import a couple of libraries: CharacterTextSplitter: We will use this library to group the information into blocks.HuggingFaceEmbeddings or SentenceTransformerEmbedding: In the notebook, I have used both, and I haven't found any difference between them. These libraries are responsible for retrieving the model that will execute the embedding of the data. There is no 100% correct way to divide the documents into blocks. The key consideration is that larger blocks will provide the model with more context. However, using larger blocks will also increase the size of our Vector Store, which can be memory-intensive. It's essential to find a balance between context size and memory usage to optimize the performance of our application. I have decided to use a block size of 250 characters with an overlap of 10. This means that the last 10 characters of one block will be the first 10 characters of the next block. It is a relatively small block size, but it is more than sufficient for the type of information we are working with. Now, we can create embeddings with the text. As you can see, I used SentenceTransformerEmbeddings instead of HuggingFaceEmbeddings. You can easily change it by modifying the commented line. With both libraries, you can call the same pre-trained model for generating embeddings. I'm using all-MiniLM-L6-v2 for both. Therefore, while there might be slight differences between the embeddings generated by each library, they will be minimal and won't affect the performance significantly. Initially, SentenceTransformerEmbeddings is specialized in transforming sentences, whereas HuggingFaceEmbeddings is more general, capable of generating embeddings for paragraphs or entire documents. Indeed, given the nature of our documents, it is expected that there would be no difference when using either library. With the generated embeddings, we can create the index This index is what we will use to ask questions, and it is specially designed to be highly efficient! After all this effort, the last thing we would want is for it to be slow and imprecise :-).


Let's start using LangChain! Now comes the fun part: chaining the actions with LangChain to create our first small application using a large language model! The application is very simple, consisting of only two steps and two components. The first step will involve a retriever. This component is used to retrieve information from documents or the text we provide as the document. In our case, it will perform a similarity-based search using embeddings to retrieve relevant information for the user's query from what we have stored in ChromaDB. The second and final step will involve our language model, which will receive the information returned by the retriever. Therefore, we need to import the libraries to create the retriever and the pipeline. Now, we can create the retriever, using the embeddings index we created earlier. We have completed the first step of our chain, or pipeline. Now, let's move on to the second one: the language model. In the notebook, I have used two different models available in Hugging Face. The first model is dolly-v23b, the smallest one in the Dolly family. I personally like this model a lot. While it might not be as popular as some others, the responses it generates are significantly better than those from GPT-2, reaching a level similar to what we could achieve with OpenAI's GPT-3.5. With 3 billion parameters, it is close to the memory limit we can load on Kaggle. This model is trained for text generation, resulting in well-crafted responses. The second model is from the T5 family. Pay attention because this model is specifically designed for text-2-text generation, resulting in much more concise and short responses. Make sure to try both models at least to see how they perform. Perfect! We have everything we need to create the pipeline! Now let's do it! Let's see what each of the parameters means: model_id: The identifier of the model in Hugging Face. You can obtain it from Hugging Face, and it usually consists of the model name followed by the version.task: Here, we specify the task for which we want to use the model. Some models are trained for multiple tasks. You can find the supported tasks for a specific model in the model's Hugging Face documentation.model_kwargs: This parameter allows us to specify additional arguments specific to the model. In this case, I'm providing the temperature (how creative we want the model to be) and the maximum length of the response. Now, it's time to configure the pipeline using the model and the retriever. In the variable chain_type, we indicate how the chain should function, and we have four options: stuff: The simplest option, it just takes the documents it deems appropriate and uses them in the prompt to pass to the model.refine: It makes multiple calls to the model with different documents, trying to obtain a more refined response each time. It may execute a high number of calls to the model, so it should be used with caution.map reduce: It tries to reduce all the documents into one, possibly through several iterations. It can compress and collapse the documents to fit into the prompt sent to the model.map re-rank: It calls the model for each document and ranks them, finally returns the best one. Similar to `refine`, it can be risky depending on the number of calls expected. Now we can use the newly created chain to ask our questions, and they will be answered considering the data from our DataFrame, which is now part of the Vector Database. So, the answer obtained will depend, as it is evident, on the Dataset used and also on the Model. For the question of whether we can buy a Toshiba laptop, we get two very different responses depending on the model: Dolly: "No, Toshiba officially shuts down their laptops in 2023. The Legendary Toshiba is Officially Done With Making Laptops. Toshiba shuts the lid on laptops after 35 years. Toshiba officially shut down their laptops in 2023." T5: "No." As you can see, each model adds its own personality to the response!


Conclusions and Continuing Learning! It's actually been a lot simpler than one might think. It's much easier now than it was before the explosion of large language models and the emergence of tools like LangChain. We have used a Vector Database to store the data that we previously loaded into a DataFrame. Although, we could have used any other data source. We used them as input for a couple of language models available in Hugging Face and observed how the models provided a response considering the information from the DataFrame. But don't stop here, make your own modifications to the notebook and solve any issues that may arise. Some ideas include: Use both datasets, and preferably search for a third one. Even better, do you think you can adapt it to read your resume? I'm sure it can be achieved with some minor adjustments.Try using a third Hugging Face model.Change the data source. It could be a text file, an Excel file, or even a document tool like Confluence.


Modern NLP: A Detailed Overview. Part 3: BERT

Prerequisite Before we dive into understanding BERT, we need to understand in order to create the model, the authors have used or referenced several concepts and improvements from several other preceding works. We need to have a very good understanding of those concepts to understand the concepts used in BERT finally. The dependencies are Transformers, Semi-supervised Sequence Learning, ULMFit, OpenAI GPT, and ELMo. So, I have discussed all the required topics other than ELMo, I will suggest going through those articles and then coming back to continue the journey for a better understanding, given, the concepts are not known. As for ELMo, no worries, we will start by understanding this concept.


Deep contextualized word representations This paper was released by Allen-AI in the year 2018. Till then, pre-trained word embeddings like Bag of Words, Word2Vec, and Glove have already gained popularity and have been proven to improve the performance of training models significantly. But, considering the success the only drawback was, the word embeddings obtained from these modelings are based on a generalized approach, not context-specific ones. In simpler words, a word can be used in different contexts in a sentence, the meaning shifts based on the sentence, but the representation or the embedding remains the same. For example, consider these two sentences, The word "deal" is used in both sentences, but the meanings are different depending on usage, hence both should not be represented by the same embeddings. This is where ELMo comes in. The authors of ELMo suggest that word embedding should express two things: (i) The complex characteristics of the word usage in semantic sense and (ii) how the words are used in varying linguistic contexts. ELMo addresses both these issues, and produces embeddings or representations of a word, as a function of the entire input sentence. The authors claim that the proposed method can be easily adapted in other models as well and have done so, improvements could be noted. Architecture: The authors have used a two-layered Bidirectional LSTM to demo the concept. It has been observed that the bi-directionality of the model, i.e, its ability to discover the words both preceding and succeeding it, gives the model an upper hand over the basic single directional LSTM, in a way to understand the context of the full sentence, thus improving the quality of the embedding.


BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding BERT was introduced in 2019, by Jacob Devlin and his colleagues from Google. Before we jump into the thorough discussion of BERT architecture and working, let's make some things clear, so that we don't land into the same confusion I did, while I was exploring these topics. In the NLP world, there are usually two types of models or tasks broadly, auto-regressive models and auto-encoding models. Auto-regressive models are the ones, that, on feeding a sentence, or a starting word token, generate words related to the input, for example, GPTs. They are best for tasks like Text generation, translation, and summarization. The other type of model is the auto-encoding model, which, is responsible for the tasks that involve the understanding of the entire sentence context in order to achieve some tasks better like, classifications, question-answers, entailments, and Name Entity Recognitions, that is, finding a particular entity in a text. In our current Google searches, we see this application from time to time, where, on web pages, the lines matching our search query get highlighted. Now, don't get me wrong, I am not saying, regressive models can't be used for encoding tasks and vice versa, I am just saying the models are more suitable in their own domain. BERT is designed as an Auto-Encoding model, so whereas models like GPT use the decoder portion of the transformer, the BERT model consists of only the Encoder portion of the transformer. The question is, WHY? If we see the architecture, the Decoder takes in the output, or on the first step, takes the start token, applies self-attention, and produces the probabilities of the next word, so, it only sees the previous tokens or words or has knowledge of only previous contexts, so it is best for regressive models, as they are basically trained to produce words. But the encoder is the part that sees the entire sentence together, and hence has the entire context and can produce better inferences, which is suitable for auto-encoding tasks. Inclusion of ELMo to support feature-based embedding: The authors of the BERT paper argued that, in the case of certain tasks, only a left-to-right learning approach is a very high price to pay, as the knowledge of the context after the word is lost. BERT introduced bi-directional learning, that is it will learn the word from left-to-right and right-to-left as well using self attention. The authors also focused on the two types of learning approaches devised so far, Feature Based learning approach like ELMo and Fine-tuning based learning like OpenAI's GPT. BERT brought the best of both worlds together. As we already learned, Bi-LSTM-based ELMo is easy to integrate into a model. So, taking advantage of the encoder, to receive the entire sentence as input at once, BERT added ELMo, to generate high-quality context-dependent Embedding for the words, which enriched the performance. Pre-training and GPT: The authors also realized that pre-training contributes majorly to the performance of the model and is highly necessary. So, it was decided to use GPT's pre-training and fine-tuning technique at the end to achieve completely what fine-tuning-based approaches had to offer. The issue was, GPT training is suited only for left-to-right or forward predicting models, but BERT had bi-directional data. There was another issue that, if a normal language modeling task is used to pre-train, the model might try to bypass the learning and leak data from the right-to-left model to left-to-right and vice versa. To eradicate the issues, the authors introduced an MLM or Masked Language Modelling task the primary pretraining task, which basically means randomly a word is masked in the text, and the model has to predict the masked word. Architecture and Training: BERT uses an encoder-based architecture of the transformer as given by Vaswani et al., without much change. BERT has released two models BERT-Base and BERT-Large. BERT-Base contains 12 encoder layers, a hidden layer vector of size 768, and 12 self-attention heads in each block, resulting in approximately 110 million parameters, whereas BERT-Large has 24 encoder layers, each with 16 attention heads handling 1024 sized vector resulting in a total of 340 million parameters. BERT is also trained in a 2 step approach: Pre-training followed by Fine Tuning. In BERT's case, Pre-training is done using two tasks: Masked Language Modeling: In this case, randomly, 15% of words in the text are masked and the model has to use the context of the sentence to predict the word.Next Sentence Prediction: As BERT targets to be used for tasks like question answers and entailments, it is important for BERT to learn to analyze the context for longer sequences. So, in this case, the model is passed sentence and told to predict the next sentence. In order to train, the authors have considered 50% of cases where the sentence appearing next is the real next sentence (IsNextSequence), and in 50% of others, it's not (IsNotNextSequence). A SoftMax layer is added to the top of the model to provide a probability between these two cases. For the input, if it's a single sentence, a [CLS] token is added in the front, which denotes the classification task as the authors explain. For tasks like sentence similarity and question answering, the authors have put a separator [SEP] token in the middle to differentiate the sentences, then we will put the context embedding vectors of the sentence, to the side they belong to. For the pre-training corpus authors used the BooksCorpus (800M words) and English Wikipedia (2,500M words). For Wikipedia, we extract only the text passages and ignore lists, tables, and headers. Fine-Tuning BERT to a specific target task happens very easily, as we have seen in the case of the other Fine-tuning based learning approaches. For tasks like question answers and similarity, sentences are concatenated using a separator. The token embeddings for the sentences are added with segmentation embeddings to determine which sentence the word belongs to which sentence and a positional embedding, as shown in the original transformer paper, to indicate the positions of the word in the sentence. The above image shows the creation of the final token embeddings for learning. Paper link: https://arxiv.org/pdf/1810.04805.pdf This is all about how a BERT model works. Now, we will briefly see two works that have been done based on BERT, which has been immensely used and successful.


RoBERTa: A Robustly Optimized BERT Pretraining Approach In 2019, a team led by Yinhan Liu from Meta AI research published this work on improving the performance of BERT using a robustly optimized training approach. They used BERT-Large to test the assumptions. The authors made the following changes: The BERT, while training, added a segmentation encoding to the token embedding to understand which sentence the words belonged to. RoBERTa removed the encoding and instead added a tokenizer. Separator character to separate the sentences.BERT was initially trained on 16GB data, while RoBERTa introduced its own novel dataset combining the dataset from (1) BERT (16GB),(2) CC-NEWS, which we collected from the English portion of the CommonCrawl News database (76 GB), (3) OPENWEBTEXT, an open-source recreation of WebText where the text is web content extracted from URLs shared on Reddit with at least three upvotes. (38 GB)(4) STORIES, a dataset containing a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas. (31GB)Resulting in a total size of 160GB. The authors claim that a bigger dataset helped the model get a more compact and comprehensive pre-trained weight.BERT uses an MLM or masked language modeling technique, but the masking is done while pre-processing, so basically, there is a pattern of the masking, and it becomes static, which the learning model can detect, decaying the generalized performance. Instead, RoBERTa introduced dynamic masking done at runtime. So, the input sequence is trained for 40 epochs with 10 different masking patterns, so each of the masking patterns appears 4 times, bringing variation and preventing the model from learning the masking pattern.The authors of RoBERTa found that the Next Sentence Prediction objective of BERT in pre-training slightly hurts the performance of the model in downstream tasks, as it affects the model's ability to learn long-range dependencies. So, that pre-training target is removed.From previous works, it was established that speed and performance optimization can be increased if the learning rate and mini-batch sizes are increased properly. Originally BERT trained for 1 million steps using a batch of 256 sequences. Keeping the amount of memory requirements fixed, authors of RoBERTa, discovered the model can train on either train for 125k steps on a 2K sequence batch or 31k steps on an 8K batch. After some ablations, the authors decided on an 8K batch size.RoBERTa introduced a modification in the input representation as well. BERT originally used a character-level BPE or byte-pair encoding of 30K vocabulary, while RoBERTa increased it to a vocabulary of 50K byte-level word units. We have listed all the points of improvement of RoBERTa over BERT, which has been used extensively by later works to improve the performance of BERT. Paper link: https://arxiv.org/pdf/1907.11692.pdf Next, we will see one last and so far the most used variant of BERT: DistilBERT.


DistilBERT, a distilled version of BERT: smaller, faster, cheaper, and lighter This work was published by hugging face in 2020 with the aim of reducing the BERT architecture and time consumed to train the model. The authors observed that the training data and the model size were exponentially increasing with each work, as a requisite for state-of-the-art performance, which made it extremely difficult to train and use such models under memory constraints. To solve this issue, DistilBERT was introduced which, through the method of knowledge distillation training, claimed to reduce, the size of the BERT model by 40%, and its training time by 60% while retaining 97% of its performance. Knowledge Distillation is a method where 2 different models are used, one is known as the Teacher model, and another is called the Student model. So, DistilBERT becomes the student model and BERT becomes the teacher. As we know, in the case of supervised learning, a well-trained model predicts if an instance belongs to a class with a very high probability and if not with a very low value. Sometimes the values are neither too high nor too low. Such cases are used by the student to optimize, as the easy cases are already out leaving distilled samples to learn from. DistilBERT model is very similar to BERT, only the number of encoder layers is factored by 2. So, DistilBERT has 6 encoder layers. The student model is initialized with BERT's pre-trained weights only as they are very similar in architecture. For each two-layer in BERT, the authors have selected one layer's weight for each layer of DistilBERT. DistilBERT trains on the same corpus as the original BERT model: a concatenation of English Wikipedia and Toronto Book Corpus. The loss function is a summation of the Distill Loss, a Cosine Embedding Loss function that tries to reduce the distance between the hidden vectors produced by the teacher and student, and the actual Masked Language modeling loss, which was used in the original BERT paper. Paper link: https://arxiv.org/pdf/1910.01108.pdf


Conclusion In this article, we have tried to learn all about BERT and its improvements. In the later parts of this series, we will see more developments in this domain. Till then, Happy Reading!


LangChain 101: Part 1. Building Simple Q&A App

Introduction LangChain is a powerful framework for creating applications that generate text, answer questions, translate languages, and many more text-related things. I've been working with LangChain since the beginning of the year and am quite impressed by its capabilities. This article is the start of my LangChain 101 course. I'll start sharing concepts, practices, and experience by showing you how to build your own LangChain applications. Today, we will discuss the following topics: What exactly is LangChain?LangChain's fundamental concepts and componentsHow to build a basic LangChain application


What is LangChain? Lang stands for language, which is the primary focus of LangChain, and chain - the connotation of connecting things - refers to the chain component used in LangChain. Chains are sequences of instructions that the framework executes to perform a task. This simplifies the use of Large Language Models for specific tasks and enables you to combine the power of LLMs (Large Language Models) with other programming techniques. I've been asked how LangChain differs from ChatGPT or LLM. To answer this question, I'm attaching a table that highlights the differences:


Basic components of LangChain There are six basic components of Langchain:- Models- Prompts- Chains- Memory- Indexes- Agents and Tools Let's briefly talk about all components.


Models Models in LangChain are large language models (LLMs) trained on enormous amounts of massive datasets of text and code. Models are used in LangChain to generate text, answer questions, translate languages, and much more. They are also used to store information that the framework can access later.Examples: GPT-x, Bloom, Flan T5, Alpaca, LLama, Dolly, FastChat-T5, etc.


Prompts Prompts are pieces of text that guide the LLM to generate the desired output. Prompts can be simple or complex and can be used for text generation, translating languages, answering questions, and more. In LangChain, prompts play a vital role in controlling the output of the LLM. You can influence the LLM to generate the desired output by carefully crafting the prompt. Here are some examples of how prompts can be used: Specify the desired output format: You can, for example, use a prompt to instruct the LLM to generate text, translate languages, or answer questions.Example: Translate the input to ArabicProvide context: A prompt can provide context for the LLM, such as information about the output topic or examples of the desired output.Example: Explain the answer step-by-step like a school teacherConstrain the output: You can use a prompt to limit the LLM's output by specifying a maximum length or selecting a list of keywords to include in the output.Example: Generate a tweet post using less than 140 words


Indexes Indexes are unique data structures to store information about the data content. This information can include terms found in each document, document location in the dataset, relationships between documents, etc. Vectorstores are data structures storing vector representations of a dataset's terms. A retriever is an interface that returns documents in response to an unstructured query. It is broader in scope than a vector store. Understanding indexes, vectorstores, and retrievers is key to building an app on your specific data. Consider another example of a chain that includes all three components: A chain is formed to answer financial questions.The chain uses an index to find all documents that contain the word "finance."The chain uses a vectorstore to find other terms that are most similar to the word "finance" ("money," "investments," etc.).The chain uses a retriever to retrieve the documents that are ranked highest for the query "What are ways to invest?".


Memory Memory in LangChain is a method of storing data that the LLM can later access. This information can include previous chain results, the context of the current chain, and any other information that the LLM requires. It also enables the application to keep track of the current conversation's context. Memory plays a vital role in LangChain, allowing the LLM to learn from previous interactions and build up a knowledge base. This knowledge base can then be used to improve the performance of future chains. Consider the creation of a chain designed to answer real estate-related questions. The chain may use memory to store the results of previous chains that have responded to real estate questions. When the chain asked a new question about a similar topic, this information could be used to improve its performance.


Chains Chains are sequences of instructions the LangChain framework executes to perform a task. Chains may connect other LangChain components based on the application requirements. They allow the framework to perform a wide variety of tasks. Assume we're creating an app to assist us in interview preparation: Prompt: "I'm getting ready for an interview for a position as a software engineer. Can you ask me some common interview questions that I may expect?"Function A: This function would access the LLM's knowledge of the software engineering field, such as its knowledge of common interview questions for software engineers. It can also look for appropriate data in the vectorstore.Function B: This function would manipulate data, such as generating a list of common interview questions for software engineers or a list of resources to help the student prepare for the interview. It will select a question and ask it.Memory: some follow-up questions might be asked to help better understand the knowledge of the chosen topic. Memory implementation will allow the chain to keep the context of the conversation.


Agents and Tools Agents and tools are two important concepts in LangChain. Agents are reusable components that can perform specific tasks such as text generation, language translation, and question-answering. Tools are function libraries that can be used to aid in developing various agents. Examples: NewsGenerator agent - for generating news articles or headlines.DataManipulator tool - for manipulating data (cleaning, transforming, or extracting features).


Application. Building a Q&A system. Colab and Github links. Let's build a Q&A system to answer our questions regarding official holidays in the United Arab Emirates. Based on the components we've learned, we can draw a schema similar to the following one below: First, we install all dependencies. Some of the versions might be updated, but installing a specific version will resolve most, if not all, inner dependencies. The data we'll be using can be found at the following website, as shown in the table below: After we store the data in memory or dataframe, we can import the necessary packages: In this tutorial, we'll use OpenAI API key. We can put all the modules in place now. We'll talk in more detail about all the components. For the moment, we've chosen gpt-3.5-turbo from OpenAI, a custom prompt, out of the box VectorstoreIndexCreator(), set up the ConversationBufferMemory to keep track of chat_history and chained everything into a RetrievalQA chain.


Holidays in March/December Correct response. What about December? Did you notice how we used the memory here? If it wasn't for it, the response would've sounded as: It is worth noticing that despite the error in counting two official holidays at the beginning, the response contains all three holidays in the list. A prompt upgrade may solve the issue.


Multichain Ramadan example Now let's ask a trickier question. So now, the chain will need to find the information about the name of the holiday that marks the end of Ramadan and only after that retrieve the required information from the table. This is quite interesting. The chain correctly identified Eid al-Fitr as the end-of-Ramadan holiday. The issue is that the data is dirty, and the model cannot recognize that it is a four-day weekend. Of course, the easy solution here would be to either clean the data, possibly through tools or modify the prompt.


What is the closest upcoming holiday? Last for today is retrieving what official holiday is next in line. As one can see, the nearest holiday is detected correctly. I'm leaving it to you to think of the possible chain that was built to answer the question.


Conclusion In this lecture, we have learned about the basics of LangChain, including models, chains, prompts, agents, memory, and indexes. We have also seen examples of how LangChain can perform different tasks and build an app for answering questions.


Upcoming lectures The LangChain 101 course is currently in development. Later in the course, we'll learn: how to we utilize models on our computershow to use own datahow to use chains to execute more complex taskshow to use memory to store informationand much more. We will continue building real-world apps with LangChain. Consider following me on Medium or LinkedIn to learn more about LangChain and get updates, including new lectures, features, tutorials, and examples. Thank you for your time!


This AI newsletter is all you need #57

What happened this week in AI by Louie In the AI world this week, LLM model performance evaluations were a topic of focus. In particular, there was a lively debate around a recent study conducted by students from Stanford and Berkeley. The research presents evidence suggesting that the GPT-4 models might be experiencing a decline in performance, colloquially referred to as getting "dumber" over time. The paper offered a range of evaluations like identifying prime numbers which decreased from 97.6% in March to 2.4% in June, and solving coding questions experienced a significant drop from 52% to 10% accuracy. This sparked many discussions including whether OpenAI was prioritizing inference speed and cost over model performance. There are also ongoing discussions concerning the accuracy of these claims. Some findings have surfaced indicating that later models exhibit significantly improved performance simply by altering the output's formatting. Furthermore, another evaluation demonstrated that neither version of the API outperformed pure chance when it came to the prime number classification problem. In response to the new findings, the authors of the paper clarified their intentions, stating that the aim of the experiment was not to demonstrate the degradation of quality in the OpenAI APIs. Instead, they sought to shed light on the issue of instability and raise awareness about the potential for applications to crash when the underlying model's responses change. Furthermore, OpenAI addressed these claims in a blog post and assured users that they are taking steps to enhance API stability. They provide the capability for developers to specify and utilize a particular model version, thus providing more control and predictability in their applications. On a more positive note for GPT-4 evaluations this week, a separate study conducted on clinical case exams compared GPT-4 to medical students, and the results showed that GPT-4 outperformed the first and second-year Stanford students. The research specified that they did not use any special prompting technique. One of the authors took to Twitter to share insights about the findings, sparking discussions about rethinking the future of student evaluations. We found both these papers interesting and the possible degradation of GPT-4 performance sparked an important debate. One thing is clear to us - in the world of commercialized products using LLM APIs - the changing and unstable performance of LLMs on specific tasks and for specific prompts as models are updated is a new challenge for developers and companies to work with. And this will be all the more important for potential medical applications! We also believe we need more work on LLM evaluation standards more broadly, including accurately and consistently benchmarking the performance of new open-source LLMs like Llama 2. - Louie Peters - Towards AI Co-founder and CEO This issue is brought to you by: Join us at the industry's leading artificial intelligence conference, Ai4 2023, in Las Vegas on August 79 at the MGM Grand. This is your last chance to join 2200+ AI leaders, 240 speakers, and 100 cutting-edge AI exhibits. Apply for a complimentary pass or register now to save 12% off final prices.


This AI newsletter is all you need #57

Hottest News Meta Releases Llama 2 Meta has released Llama-2, an open-source model with a commercial license, which demonstrates similar performance to ChatGPT. Trained on 2T tokens with varying parameter sizes, Llama-2 underwent further fine-tuning and improvement through a combination of instruction and reinforcement learning, surpassing other open-source models like Falcon and MPT in performance. 2. Announcing LangSmith: A Unified Platform for LLM applications LangChain has developed LangSmith, a platform designed to help developers close the gap between prototype and production. By providing essential debugging, testing, evaluation, and monitoring features, LangSmith assists AI professionals in identifying and addressing issues such as unexpected results, errors, and latency. 3. Apple Is Testing a ChatGPT-Like AI Chatbot Apple is developing its chatbot, named "Apple GPT," to challenge OpenAI and Google. Despite initial security concerns, the chatbot is now more widely accessible to Apple employees for prototyping purposes, with restricted usage and no customer-bound features allowed. 4. Cerebras Systems Signs a $100 Million AI Supercomputer Deal with UAE's G42 Cerebras Systems has announced a $100 million deal with G42, signaling the debut of AI supercomputers that could potentially challenge Nvidia's market position. To expedite the rollout, Cerebras will construct three Condor Galaxy systems in the United States, with the first supercomputer set to go online this year. 5. Custom Instructions for ChatGPT OpenAI is introducing personalized custom instructions for ChatGPT, enabling users to have a more tailored and adaptable experience. This feature emphasizes the significance of customization in catering to diverse needs. Custom Instructions will be gradually rolled out to all users, with beta access initially offered to Plus plan subscribers.


This AI newsletter is all you need #57

Five 5-minute reads/videos to keep you learning LLaMA 2 - Every Resource You Need This blog post includes all the relevant resources to help you get started with LLaMa 2. It compiles topics such as "What is LLaMa 2," where you can test the model, the research behind the model, how good the model is, how to correctly prompt the chat model, and more. 2. Hallucinations in AI Hallucinations in AI propel hyperbolic narratives around foundation models and open-source. It's hard to know what to believe and whom to trust. This insightful read from John Luttig delves into some of the narratives and trends in AI that are easily misconstrued or just plain wrong. 3. Building an AI WebTV The AI WebTV project showcases the potential of text-to-video models like Zeroscope and MusicGen in generating entertaining videos. Created using Hugging Face services, it utilizes a combination of ChatGPT, Zeroscope V2, and FILM to create high-quality video clips with accompanying music. 4. Fearing The Wrong Thing Mike Loukides argues that the only thing to fear is failing to make the transition to AI-assisted programming. He has been talking and writing about the end of programming, but what does this mean in practice? In this article, Mike shares why and how the use of AI will change the discipline as a whole. 5. How To Ensure Consistency in AI Visuals This tutorial focuses on generating consistency in AI visuals. It offers basic to advanced techniques for achieving consistency control in Stable Diffusion, Midjourney, and InsightFace.


This AI newsletter is all you need #57

Papers & Repositories FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Stanford University has introduced FlashAttention-2, an algorithm that accelerates attention and reduces memory usage in language models. The updated version is 2x faster than the original and achieves improved performance through better parallelism and work partitioning techniques. 2. Lost in the Middle: How Language Models Use Long Contexts This study investigates the performance of language models in utilizing extended contexts for tasks such as question answering and retrieval. While models excel in finding relevant information at the start or end of input, their performance declines when accessing middle sections of long contexts. The study highlights the challenges of effectively utilizing long contexts and emphasizes the necessity for future improvements in this area. 3. Towards A Unified Agent with Foundation Models Researchers have found that incorporating language models and vision language models in reinforcement learning agents can address significant challenges in the field. By harnessing the knowledge stored in these models, agents can effectively explore sparse-reward environments, reuse data for learning, schedule skills for novel tasks, and learn from expert observations. 4. Learning to Retrieve In-Context Examples for Large Language Models Researchers have developed a framework that utilizes dense retrievers to automatically select high-quality examples for in-context learning of LLMs. Experimental results demonstrate its effectiveness in improving LLM performance by retrieving similar and contextually relevant examples. 5. How Is ChatGPT's Behavior Changing Over Time? A research study examined the performance of GPT-3.5 and GPT-4 on various tasks over time. It found some significant variations in their behavior, with GPT-4's accuracy in identifying prime numbers dropping from March to June 2023. Additionally, both models displayed an increase in formatting mistakes during code generation. Enjoy these papers and news summaries? Get a daily recap in your inbox!


This AI newsletter is all you need #57

Meme of the week! Meme shared by rucha8062


This AI newsletter is all you need #57

Featured Community post from the Discord Louvivien has developed an open-source AI trading app that seamlessly connects to Alpaca, enabling users to access positions, orders, and conduct stock transactions. With this app, users can import collaborative trading strategies and efficiently manage AI trading funds. You can explore this project on GitHub and support a fellow community member. For those interested in AI trading, you can join this open-source project by connecting on the thread here.


This AI newsletter is all you need #57

AI poll of the week! Join the discussion on Discord.


This AI newsletter is all you need #57

Article of the week Fixing SimCLR's Biggest Problem - BYOL Paper Explained by Boris Meinardus SimCLR successfully implemented the idea of Contrastive Learning and, at that time, achieved a new state-of-the-art performance. However, the idea has fundamental weaknesses, such as its sensitivity to specific augmentations and the requirement for very large batch sizes. A new approach to self-supervised learning called Bootstrap Your Own Latent (BYOL), developed by researchers at DeepMind, implements a completely fresh approach to training self-supervised models.


This AI newsletter is all you need #57

Our must-read articles Harness the Power of Vector Databases: Influencing Language Models with Personalized Information by Pere Martra Machine Learning in a Non-Euclidean Space by Mastafa Foufa Top Computer Vision Papers During Week From 10/7 To 16/7 by Youssef Hosni Data Science Accelerated: ChatGPT Code Interpreter as Your AI Assistant by Esmaeil Alizadeh If you are interested in publishing with Towards AI, check our guidelines and sign up. We will publish your work to our network if it meets our editorial policies and standards.


This AI newsletter is all you need #57

Job offers Head of Content + Developer Relations @ngrok Inc. (Remote) Senior Backend Engineer @Remote (Remote) Senior Infrastructure Software Engineer @ClickHouse (Remote) Product Developer @Shiru (Alameda, CA, USA) Senior Software Test Engineer @Clari (Bangalore, India) Product Engineer @Encord (London, UK) Interested in sharing a job opportunity here? Contact sponsors@towardsai.net. If you are preparing your next machine learning interview, don't hesitate to check out our leading interview preparation website, confetti!


Machine Unlearning in 2023: Where We Are and Where It's Headed

Table of Contents Introduction to Machine UnlearningApplications and MotivationsAlgorithms and EvaluationCurrent Challenges


Introduction to Machine Unlearning What is machine unlearning and how does it differ from machine learning? Why do we need machine unlearning and what value can it unlock? What is the intuition underlying existing methods and paradigms for machine unlearning? Machine unlearning is a relatively new field that essentially teaches a trained model to "forget" a certain subset of the input dataset that it was trained on, without significantly affecting performance. There's a variety of motivations: some personal information is no longer available per updates in data privacy policies; some data is proven to be outdated or inaccurate and is counterproductive for the model; or a model is inadvertently trained on data that perpetuates biases and we want to remove the specific bias-inducing subset. You may be wondering: this sounds like a simple task of dropping the problematic data subset and retraining a model using the updated dataset. Why complicate it? As it turns out, many models, such as Large Language Models like ChatGPT, often require vast computational resources to train in the first place. The forget set often constitutes only a small fraction of the total training set; thus the "retrain from scratch" approach becomes an excessively expensive and repetitive solution, necessitating a better algorithm that removes the forget set while preserving the performance of the original model.


Applications and Motivations What are some use cases motivating the development of machine unlearning? How does machine unlearning uniquely contribute to these fields and tasks? What potential future applications are being explored, and what can we expect? We've mentioned the idea that due to scaling computational resources, the advantage that machine unlearning holds over a full retrain-from-scratch is much more significant on larger datasets, often in real-life applications. Let's look at a few of the top application areas necessitating machine unlearning: Fair and ethical models (bias removal): When trained on massive real-world data, sometimes models pick up on the biases inherent to the dataset. As the model scales, the algorithm can sometimes amplify the bias. In some cases the consequence of model bias can be especially harmful, such as in models used for predictive policing in preventing crimes or hiring from a candidate pool. When these biases are identified, they should be immediately removed, and machine unlearning offers a potential way to do this and mitigate harm.Data updates (lifelong learning): Data for a model can become outdated after corrections or updates are made to the dataset for a variety of reasons. Through unlearning and forgetting these counterproductive data points, models can adapt to these changes and produce more accurate predictions. This is especially useful in dynamic learning models such as lifelong learning; offloading outdated or incorrect data improves both the accuracy and efficiency of computation.Data privacy (the right to be forgotten): This is perhaps what machine unlearning has been most commonly associated with. In a rapidly changing digital world, the laws and regulations governing digital practices are constantly updated. Forgetting certain attributes of personal data helps models stay compliant with regulations such as GDPR³ by safeguarding against data leaks or anonymization concerns.


Algorithms and Evaluation Metrics What are the main algorithmic paradigms in machine unlearning and how are they implemented? What are the strengths and weaknesses of each? What metrics have been proposed to assess machine unlearning performance?


Methods Essentially the logic behind removing data without completely retraining is to decrease the correlation between certain features and the output. Let's look at a main technique, known as "data poisoning": This is where we intentionally add inaccurate data to "confuse" the model about a certain feature or attribute. Here's an example: a model trained on a dataset of crime reports observes a trend where people between 5060 years old are highly likely to commit crimes. As a result, it unfairly targets the 5060 age group in future predictive crime detection. To mitigate this bias, we may generate a synthetic dataset of crime statistics among other age groups, matching that of the 5060 age group without interfering with other features. This weakens the strength of the correlation between conviction and the 5060 age group.


Evaluation metrics Since this is a relatively new field, many evaluation standards are still being proposed every day to better assess machine unlearning performance. Below, we highlight a few of the proposed evaluation metrics to measure the quality and effectiveness of machine unlearning methods: Forgetting rate: measures the proportion of data samples that originally belonged to a dataset but are no longer recognized after the unlearning process.Membership Inference Attack: measures the degree to which an adversary can correctly infer whether a data sample is part of the training set for the model. This metric is especially relevant in security contexts.


Current Challenges What are the major challenges facing the Machine Unlearning field as a whole? What are the challenges that emerge in specific contexts or applications? Machine unlearning can be quite an intricate task for a number of reasons. In fact, in July 2023 Google launches a public competition with the aim of unifying and advancing effective machine unlearning methods. Lack of standardized evaluation. Although a number of publications have addressed the notion of a unified assessment metric by using well-known datasets such as CIFAR10 and CIFAR100 as benchmarks, there lacks a universal evaluation standard that can unify the results produced by different researchers in the field. In fact, this is one of the aims of Google's recent machine unlearning challenge - to provide a platform to compare different methods "apple-to-apple"².Susceptibility to adversarial attacks. Research has found that models that have been taught to unlearn some data subsets can fall prey to camouflaged attacks because of the poisoning mechanism involved in unlearning.


Future directions These current obstacles provide potential directions for interesting future research questions: How do we train models to defend against camouflaged poisoning attacks?How can we leverage computational resources to optimize unlearning efficiency?How can we effectively clean and generate data to overcome the lack of training data?How can we unify a set of standards to assess unlearning algorithms to ensure consistent evaluation?


Conclusion Thanks for sticking with me this far! In this article, we walked through the field of machine unlearning, including its main motivations and application areas, existing methods and paradigms, and current challenges. Later in this series, we will take a deeper dive into the technical aspects of existing unlearning algorithms and Python implementations with real statistical and image datasets. Happy coding!


Bibliography [1] Shaik, T., Tao, X., Xie, H., Li, L., Zhu, X., & Li, Q. (2023). Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy. https://doi.org/10.48550/ARXIV.2305.06360 [2] Announcing the first Machine Unlearning Challenge. (2023, June 29). Ai.googleblog.com. https://ai.googleblog.com/2023/06/announcing-first-machine-unlearning.html [3] European Union Agency for Fundamental Rights. (2019). The General Data Protection Regulation: One year on civil society: awareness, opportunities, and challenges. Publications Office. https://data.europa.eu/doi/10.2811/538633


Modern NLP: A Detailed Overview. Part 2: GPTs

Evolution of NLP domain after Transformers Before we start, let's take a look at the timeline of the works which brought great advancement in the NLP domain. This will help better understand how the advancement of the previous works impacted the upcoming developments and to better address the interdependencies among them.


Generative Pre-trained Transformer (GPT) In 2018, OpenAI introduced GPT, which has shown, with the implementation of pre-training, transfer learning, and proper fine-tuning, transformers can achieve state-of-the-art performance. Ever since, OpenAI has been working on increasing the power and parameters of the model, and improving the training data, which has given rise to modern-day tools like ChatGPT, InstructGPT, and AutoGPT. But, the question is, how did all these concepts come together? Let's see it step by step.


Semi-Supervised Sequence Learning As we all know, supervised learning has a drawback, as it requires a huge labeled dataset to train. In the NLP domain, it has been a challenge to procure large amounts of data, to train a model, in order for the model to get proper context and embeddings of words. In 2015, Andrew M. Dai from Google published a paper named Semi-Supervised Sequence Learning, in which the authors proposed an unsupervised approach to pre-train a model on different documents like DBPedia extracts, IMDB, Rotten-tomato reviews and unpublished manuscripts. It was observed that this pre-training allowed the model to have a greater understanding of the words, thus, better embeddings. Finally, they fine-tuned the model to specific tasks, which was supervised task, to find that the learning was much stabilized and converged quicker. So, on the whole, a semi-supervised process worked wonders. The authors have used a basic LSTM and trained using the clipping gradient method, which is proven to work better than the normal RNNs for predictions. Next, the authors have used 2 methods for pretraining. The model is passed a sequence and trained to predict the next word. This has been termed the LM-LSTM, or language-modeling model, which is the name of this task.The model is passed a sequence, and then it creates a single vector which is then used by the model's decoder to reconstruct the whole sentence. This has been termed the SA-LSTM, or sequence-autoencoder model, accordingly due to the encoding creation task. Later these pre-trained models were used after adding fine-tuning for tasks like the sentimental analysis and were found to surpass the state-of-the-art performance. Other findings are: The pre-training increased the stability of learning and the generalization power of the model.The addition of more data further increased the refinements.The SA-LSTM is found to be performing better, owing to the fact that they had to look for a longer context of the entire sentence, compared to the LM-LSTMs, which predicted only the next word, so they paid attention to a shorter context. Paper link: https://arxiv.org/pdf/1511.01432.pdf Important Conclusion: Pre-training boosts performance.


Generating Wikipedia By Summarizing Long Sequences This work was published by Peter J Liu at Google in 2019. The authors proposed that, by using only the decoder part of the original transformer architecture, instead of using the entire thing, state-of-the-art performance can be achieved for generative tasks like summarizations. The architecture is an auto-regressive architecture, i.e., the model produces one word at a time and then takes in the sequence attached with the predicted word, to predict the next word. Basically, it predicts a word with the context of the previous word. The authors have used multiple Wikipedia documents as the source document for the task and used strong RNN and Transformer encoder-decoder baseline and models to validate the proposed architecture's performance. Having used multiple source documents, there have been duplicates and resulted in a huge set, which is impossible to train a model on, due to lack of processing power. So, the authors have used a two-stage process, following what we do as humans. Step 1: We highlight the points from different articles to make a set of useful information. To perform this, extractive summarization methods like tf-idf, and text-rank algorithms have been used. The authors have 5 extractive methods and used the best performer Step 2: The final step is to actually summarize the points picked in step 1 by the extractive methods using an abstractive summarization method or deep-learning method, in this case, the decoder of the transformer. The architecture is the same as what we saw in the real transformer, with a Masked Attention block followed by a self-attention block and finally a fully connected layer, only the encoder block is dropped. The sentences and their results are concatenated in a single vector and passed into the model to train. The decoder layers create a hidden vector, which is passed into a softmax layer, for the word prediction. Paper link: https://arxiv.org/pdf/1801.10198.pdf Important Conclusion: Decoder is Enough for generative tasks.


Universal Language Model Fine-tuning for Text Classification: UlmFit This work was published in 2018 by Jeremy Howard of Fast AI, which became a prime development in this world, which is empowering all the NLP models we see today. The authors introduced the idea of transfer learning in the natural language processing, understanding, and inference world. People who are familiar with the world of computer vision will have an idea of the concept called transfer learning, so we need to shift a bit to the computer vision world to understand the importance and urge to find something equivalent. As supervised learning required huge datasets for image processing to train super-deep models, the scarcity of data became an issue. At this point, datasets like ImageNet and MS-COCO came with huge data, enough to train these models. Once the models were trained, it was found that the models had a good general idea of object boundary detection, color, and gradient detection. So, though these models were trained on a different source task, we can use these trained models and change the final classification layer, and fine-tune them for other tasks easily. These generalized models helped to achieve great results for tasks where data was less. Now, coming back to the NLP world, the models were trained mostly very task-specific, so, if there was less data, there was no way out. A generalization requirement emerged. It was not like, the concept of transfer learning was never tried before, but they never succeeded, then came the savior. The authors found a solution for so-called Downstream Tasks, which are the ones that have less data availability and need a pre-trained model to be fine-tuned accordingly. The authors mentioned the reason for the failure, as the lack of knowledge of how to fine-tune the model while acknowledging the good impact of initializing the model with pre-trained weights found using objectives like language modeling. The authors have mentioned that previous works have required too much-labeled data to fine-tune, which has affected usability severely. When the researchers have tried to train using smaller datasets, due to lack of data, it has caused overfitting on the dataset, and as a consequence, a forgetting character in the model or loss of generalization has been introduced, finally diverging. The authors have proposed a three-step method to train and fine-tune a pre-trained model to perform best. In order to prove the claim, the authors have used a normal 3-layered LSTM model, and fine-tuning has been done using a short dataset of 100 labeled IMDB reviews to prove its sample efficiency.