Step 2: Retrieve Data

In this step, we're setting up the environment to retrieve data from DeepLake using the CohereEmbeddings for transforming numerical data back to text. We'll then use ContextualCompressionRetriever & CohereRerank to search, rank and retrieve the relevant data.Add this code to your home page in DatabuttonFirst we set the COHERE_API_KEY and ACTIVELOOP_TOKEN environment variables, using db.secrets.get, allowing us to access the Cohere and ActiveLoop services. DeepLake() retrieve data CohereEmbeddings() Following this, we create a DeepLake object, passing in the dataset path to the DeepLake instance, setting it to read-only mode and passing in the embedding function.Next, we define a data_lake function. Inside this function, we instantiate a CohereEmbeddings object with a specific model, embed-english-v2.0. ContextualCompressionRetriever() & CohereRerank() Reranking (cohere.com) We then instantiate a CohereRerank object with a specific model and number of top items to consider (top_n), and finally create a ContextualCompressionRetriever object, passing in the compressor and retriever objects. The data_lake function returns the DeepLake object, the compression retriever, and the retriever.The data retrieval process is set up by calling the data_lake function and unpacking its return values into dbs, compression_retriever, and retriever.The Rerank endpoint acts as the last stage reranker of a search flow.


Step 3: Use ConversationBufferWindowMemory to Build Conversation Chain with Memory

In this step, we will build a memory system for our chatbot using the ConversationBufferWindowMemory.The memory function instantiates a ConversationBufferWindowMemory object with a specific buffer size (k), a key for storing chat history, and parameters for returning messages and output key. The function returns the instantiated memory object.We then instantiate the memory by calling the memory function.1@st.cache_resource() 2def memory(): 3 memory=ConversationBufferWindowMemory( 4 k=3, 5 memory_key='chat_history', 6 return_messages=True, 7 output_key='answer' 8 ) 9 return memory 10 11memory=memory() 12CopyThe chatbot uses the AzureChatOpenAI() function to initiate our LLM Chat model. You can very easily swap this out with other chat models listed here.1from langchain.chat_models import AzureChatOpenAI 2 3BASE_URL = '<URL>' 4API_KEY = db.secrets.get('AZURE_OPENAI_KEY') 5DEPLOYMENT_NAME = '<deployment_name>' 6llm = AzureChatOpenAI( 7 openai_api_base=BASE_URL, 8 openai_api_version='2023-03-15-preview', 9 deployment_name=DEPLOYMENT_NAME, 10 openai_api_key=API_KEY, 11 openai_api_type='azure', 12 streaming=True, 13 verbose=True, 14 temperature=0, 15 max_tokens=1500, 16 top_p=0.95 17) 18CopyNext, we build the conversation chain using the ConversationalRetrievalChain. We use the from_llm class method, passing in the llm, retriever, memory, and several additional parameters. The resulting chain object is stored in the qa variable.qa = ConversationalRetrievalChain.from_llm( llm=llm, retriever=compression_retriever, memory=memory, verbose=True, chain_type='stuff', return_source_documents=True )


Verbose Display Code for Streamlit

1import databutton as db 2import streamlit as st 3import io 4import re 5import sys 6from typing import Any, Callable 7 8def capture_and_display_output(func: Callable[..., Any], args, **kwargs) -> Any: 9 # Capture the standard output 10 original_stdout = sys.stdout 11 sys.stdout = output_catcher = io.StringIO() 12 13 # Run the given function and capture its output 14 response = func(args, **kwargs) 15 16 # Reset the standard output to its original value 17 sys.stdout = original_stdout 18 19 # Clean the captured output 20 output_text = output_catcher.getvalue() 21 clean_text = re.sub(r'\x1b[.?[@-~]', '', output_text) 22 23 # Custom CSS for the response box 24 st.markdown(''' 25 <style> 26 .response-value { 27 border: 2px solid #6c757d; 28 border-radius: 5px; 29 padding: 20px; 30 background-color: #f8f9fa; 31 color: #3d3d3d; 32 font-size: 20px; # Change this value to adjust the text size 33 font-family: monospace; 34 } 35 </style> 36 ''', unsafe_allow_html=True) 37 38 # Create an expander titled 'See Verbose' 39 with st.expander('See Langchain Thought Process'): 40 # Display the cleaned text in Streamlit as code 41 st.code(clean_text) 42 43 return response 44CopyThat is all you need to start building your own RAG Chatbot on your own data! I can't wait to see what you build and how you develop this idea forward.


What is Retrieval Augmented Generation (RAG)?

Retrieval Augmented Generation, or RAG, is a machine learning technique combining the best aspects of retrieval-based and generative language models. This method cleverly integrates the strength of retrieving relevant documents from a large set of data and the creative ability of generative models to construct coherent and diverse responses. Moreover, RAG allows the internal knowledge of the model to be updated efficiently without retraining the entire model.


Conclusion: Retrieval Augmented Generation to Power Chatbots & Economy

In conclusion, Retrieval Augmented Generation (RAG) is not just an emerging AI technology but a transformative force reshaping how businesses operate. With its proven potential to boost productivity, catalyze AI-first businesses, and increase GDP, it's clear that adopting RAG and Large Language Models is crucial for maintaining a competitive edge in today's rapidly-evolving digital landscape. The potential of applications like the Educational Chatbot demonstrates how these AI tools can streamline tasks, making operations more efficient and user-friendly. Businesses, developers, and technology enthusiasts need to understand and leverage these advancements. The ongoing development of AI tools like Knowledge Assistants emphasizes the importance of keeping pace with these technological evolutions. As we stand at the brink of the third revolution in human-computer interfaces, we are reminded of the immense value and opportunities RAG and LLMs hold, estimated to unlock $1 trillion in the U.S. economy alone. The future is here, and it's AI-driven.


How does Retrieval Augmented Generation work?

RAG operates in two distinct stages. The first stage involves retrieving relevant documents from a vast vector database like Deep Lake using 'dense retrieval.' This process leverages vector representations of the query and documents to identify the most relevant document matches. The second stage is the generation phase, where a sequence-to-sequence model is utilized to create a response, considering not just the input query but also the retrieved documents. The model learns to generate responses based on the context of these retrieved documents.


What are the benefits of using Retrieval Augmented Generation

RAG offers several advantages over traditional LLMs: RAG can easily acquire knowledge from external sources, improving the performance of LLMs in domain-specific tasks. RAG reduces hallucination and improves the accuracy of generated content. It requires minimal training, only needing to index your knowledge base. RAG can utilize multiple sources of knowledge, allowing it to outperform other models. It has strong scalability and can handle complex queries. It can overcome the context-window limit of LLMs by incorporating data from larger document collections. RAG provides explainability by surfacing the sources used to generate text.


How to implement Retrieval Augmented Generation?

Implementation of RAG involves three key components: a knowledge-base index like Deep Lake, a retriever that fetches indexed documents, and an LLM to generate the answers. Libraries like Deep Lake and LangChain have made it easier to implement these complex architectures.


What is the historical Ccntext of Retrieval Augmented Generation?

Retrieval Augmented Generation, as a concept, has its roots in foundational principles of Information Retrieval (IR) and Natural Language Processing (NLP). Retrieving relevant information before generating a response is common in IR. With the rise of neural network-based models in NLP, these approaches started merging, leading to the development of RAG.


What are the Complexities Involved in RAG?

The main challenge with RAG lies in its dual nature - retrieval and generation. The retrieval phase requires an efficient system to sift through vast data. On the other hand, the generation phase needs a model capable of constructing high-quality responses. Both phases require significant computational resources and advanced machine-learning expertise. Using libraries like Deep Lake for efficient data storage and retrieval helps streamline using RAG.


What are the Current Challenges with Retrieval Augmented Generation?

Current challenges with RAG include: Handling complex queries that require deep understanding. Managing computational resources efficiently. Ensuring response relevance and quality. Improving these aspects would make RAG even more effective in tasks like chatbots, question-answering systems, or dialogue generation.


Where is Retrieval Augmented Generation used?

RAG is useful for complex, knowledge-intensive tasks, such as question-answering and fact verification. It has been used to improve the performance of large language models (LLMs) like GPT-4 or LLama-v2, fine-tuning their performance to be more factual, domain-specific, and diverse.


What are Retrieval Augmented Generation advantages?

RAG combines the benefits of both retrieval-based and generative models. This means it gains from the specificity and factual correctness typical of retrieval-based methods while leveraging the flexibility and creativity inherent in generative models. This combination often results in more accurate, detailed, and contextually appropriate responses.


Introduction to Radiology in Machine Learning & Multi-Image Segmentation with TransUNet

Deep learning-based image segmentation is becoming essential in computer vision. Domains such as biomedical ML, autonomous vehicles & robotics, machine recognition, face detection, etc. rely on image segmentation to identify regions of interest in a given image to make decisions. In radiology, image segmentation seems promising as it helps radiologists to quickly identify affected areas in CT scans or MRIs. Radiologists can use such tools to diagnose patients precisely, providing them with relevant treatments.Deep learning-based image segmentation tools are potent as they can identify tiny anomalies that human eyes can miss. As deep learning algorithms develop, novel SOTA models are being released, laying the foundation for next-gen radiology.U-Net, the most used architecture for image segmentation in biomedicine, has paved its way as the primary architecture. This architecture uses convolutional neural networks as its main component. But with the introduction of transformers, convolutional neural networks are either being hybridized or removed for better/more accurate results.In this article, we will discuss how UNet, hybridized with CNN and transformers, can yield better results. Apart from that, we will also explore the following: What is image segmentation? How is image segmentation used in image processing? How is image segmentation used in radiology machine learning? What is U-Net? What are Transformers? Finally, we will learn how to develop our multi-segment U-Net model by hybridizing transformers and convolution neural networks.


What is Image Segmentation?

Image segmentation is computer vision task separating different objects in the image at a pixel level. Essentially, the pixels of all the individual objects in the image are clustered and grouped as a set of pixels, separating one object from another. These clustered pixels are then assigned a particular color known as a mask and a label for identification. This makes image analysis more straightforward, and meaningful information can be derived efficiently.Source: Deep Lake


How is Segmentation Used in Radiology?

In radiology, image segmentation is used to detect different parts of an object in a medical image (CT, X-rays, & MRI scans) by putting a layer of segmentation masks (or painting a set of pixels relating to the object) and assigning labels to the same to detect anomalies such as malignant cells or find defects such as a none fracture bones for diagnosis and screening purposes.


How many types of image segmentation exist?

As such, there are three types of image segmentation:Semantic Segmentation: It is a segmentation task where different objects of the same class are segmented together. For instance, all the objects belonging to humans will be segmented with the same segmentation mask. Likewise, all the objects representing trees will be segmented with the same segmentation mask.Instance Segmentation: Here, all objects, regardless of whether they belong to the same class, are uniquely segmented with different segmentation masks. Instance segmentation, as the name suggests, creates segmentation masks based on the number of instances.Panoptic Segmentation: A combo of semantic and instance segmentation, utilizing the strengths of both approaches to create accurate masks for both well-defined and less distinct objects, resulting in improved overall performance.Source: Deep LakeThe image above shows the difference between the different types of segmentations.


U-Net: Convolutional Networks for Biomedical Image Segmentation

In effect, this allows for good extraction of the features through the downsampling blocks, followed by sound localization of the extracted features by the upsampling blocks at each layer.Here is the code for a basic U-net:1 2class Decoder(nn.Module): 3 def __init__(self, in_channels, middle_channels, out_channels): 4 super(Decoder, self).__init__() 5 self.up = nn.ConvTranspose2d(in_channels, out_channels, kernel_size=2, stride=2) 6 self.conv_relu = nn.Sequential( 7 nn.Conv2d(middle_channels, out_channels, kernel_size=3, padding=1), 8 nn.ReLU(inplace=True) 9 ) 10 def forward(self, x1, x2): 11 x1 = self.up(x1) 12 x1 = torch.cat((x1, x2), dim=1) 13 x1 = self.conv_relu(x1) 14 return x1 15Copy1class UNet(nn.Module): 2 def __init__(self, n_class): 3 super().__init__() 4 5 self.base_model = torchvision.models.resnet18(True) 6 self.base_layers = list(self.base_model.children()) 7 self.layer1 = nn.Sequential( 8 nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False), 9 self.base_layers[1], 10 self.base_layers[2]) 11 self.layer2 = nn.Sequential(*self.base_layers[3:5]) 12 self.layer3 = self.base_layers[5] 13 self.layer4 = self.base_layers[6] 14 self.layer5 = self.base_layers[7] 15 self.decode4 = Decoder(512, 256+256, 256) 16 self.decode3 = Decoder(256, 256+128, 256) 17 self.decode2 = Decoder(256, 128+64, 128) 18 self.decode1 = Decoder(128, 64+64, 64) 19 self.decode0 = nn.Sequential( 20 nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True), 21 nn.Conv2d(64, 32, kernel_size=3, padding=1, bias=False), 22 nn.Conv2d(32, 64, kernel_size=3, padding=1, bias=False) 23 ) 24 self.conv_last = nn.Conv2d(64, n_class, 1) 25 26 def forward(self, input): 27 e1 = self.layer1(input) # 64,128,128 28 e2 = self.layer2(e1) # 64,64,64 29 e3 = self.layer3(e2) # 128,32,32 30 e4 = self.layer4(e3) # 256,16,16 31 f = self.layer5(e4) # 512,8,8 32 d4 = self.decode4(f, e4) # 256,16,16 33 d3 = self.decode3(d4, e3) # 256,32,32 34 d2 = self.decode2(d3, e2) # 128,64,64 35 d1 = self.decode1(d2, e1) # 64,128,128 36 d0 = self.decode0(d1) # 64,256,256 37 out = self.conv_last(d0) # 1,256,256 38 return out 39CopyWe can consider UNet a symmetric encoder-decoder network that leverages skip connections to extract vital information from the image and enhance detail retention .  Since UNet leverages convolutional operation to extract features, it fails to model long-range dependency explicitly.  This issue is because of the intrinsic locality of convolution operations.  To tackle this issue, we use transformers.


Self-Attention

Hence the self-attention mechanism does a great job of finding meaningful feature representations in the given distribution compared to any other architecture.Here is the code for the multi-head self-attention mechanism written in Pytorch:1class Attention(nn.Module): 2 def __init__(self, num_attention_heads, hidden_size, attention_dropout_rate): 3 super(Attention, self).__init__() 4 self.num_attention_heads = num_attention_heads 5 self.attention_head_size = int(hidden_size / self.num_attention_heads) 6 self.all_head_size = self.num_attention_heads * self.attention_head_size 7 8 self.query = Linear(hidden_size, self.all_head_size) 9 self.key = Linear(hidden_size, self.all_head_size) 10 self.value = Linear(hidden_size, self.all_head_size) 11 12 self.out = Linear(hidden_size, self.all_head_size) 13 self.attn_dropout = Dropout(attention_dropout_rate) 14 self.proj_dropout = Dropout(attention_dropout_rate) 15 16 self.softmax = torch.nn.Softmax(dim=-1) 17 18 def transpose_for_scores(self, x): 19 new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size) 20 #multihead attention 21 x = x.view(*new_x_shape) 22 return x.permute(0, 2, 1, 3) 23 24 def forward(self, hidden_states): 25 #finding representations 26 mixed_query_layer = self.query(hidden_states) 27 mixed_key_layer = self.key(hidden_states) 28 mixed_value_layer = self.value(hidden_states) 29 30 #multi-head view 31 query_layer = self.transpose_for_scores(mixed_query_layer) 32 key_layer = self.transpose_for_scores(mixed_key_layer) 33 value_layer = self.transpose_for_scores(mixed_value_layer) 34 35 #calculating attention 36 attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2)) 37 attention_scores = attention_scores / math.sqrt(self.attention_head_size) 38 attention_probs = self.softmax(attention_scores) 39 weights = attention_probs 40 attention_probs = self.attn_dropout(attention_probs) 41 42 context_layer = torch.matmul(attention_probs, value_layer) 43 context_layer = context_layer.permute(0, 2, 1, 3).contiguous() 44 new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,) 45 context_layer = context_layer.view(*new_context_layer_shape) 46 attention_output = self.out(context_layer) 47 attention_output = self.proj_dropout(attention_output) 48 49 return attention_output, weights 50CopyThe multi-head self-attention module is accompanied by various other modules, such as Norm and a linear layer .  This whole set of mechanisms can be known as an attention block.  See the image below.Source: Attention Is All You NeedWe can enclose the above diagram in the following code:1class Block(nn.Module): 2 def __init__( 3 self, 4 num_attention_heads, 5 hidden_size, 6 linear_dim, 7 dropout_rate, 8 attention_dropout_rate, 9 eps, 10 std_norm, 11 ): 12 13 super(Block, self).__init__() 14 self.hidden_size = hidden_size 15 self.attention_norm = LayerNorm(hidden_size, eps=eps) 16 self.ffn_norm = LayerNorm(hidden_size, eps=eps) 17 self.ffn = Mlp( 18 hidden_size=hidden_size, 19 linear_dim=linear_dim, 20 dropout_rate=dropout_rate, 21 std_norm=std_norm, 22 ) 23 self.attn = Attention( 24 num_attention_heads=num_attention_heads, 25 hidden_size=hidden_size, 26 attention_dropout_rate=attention_dropout_rate, 27 ) 28 29 def forward(self, x): 30 h = x 31 x = self.attention_norm(x) 32 x, weights = self.attn(x) 33 x = x + h 34 h = x 35 x = self.ffn_norm(x) 36 x = self.ffn(x) 37 x = x + h 38 return x, weights 39Copy


The U-Net architecture stands and remains the state-of-the-art architecture for image segmentation.  It was introduced in 2015 by Ronneberger et al.  The paper describes a CNN with a U-Net design.Source: U-Net: Convolutional Networks for Biomedical Image SegmentationThe U-Net design is specifically used to extract features and construct precise segmentation masks.  It consists of two main components: a downsampling block and an upsampling block connected at each layer.The downsampling block is made up of a CNN that extracts features of the given image and yields two sets of outputs.  One set is fed into the max-pooling layer, where the image is reduced, while the other is fed to the adjacent upsampling block, where localization is performed.  See the image below.Source: U-Net: Convolutional Networks for Biomedical Image SegmentationThe red arrow represents max-pooling, i.e., the reduction of the image, while the gray arrow represents the passing of the image to the upsampling block.The upsampling blocks are made of transpose-CNN that expands the size of the image.  Essentially, it takes two inputs.  The first input comes from the adjacent downsampling block.  The second input comes from the lower upsampling block.  The network adds padding via transpose convolution operation for the latter, which has a reduced dimension.  The upsampling block's job is to concatenate the two inputs and generate a localized output of the original image.  See the image below.Source:U-Net: Convolutional Networks for Biomedical Image SegmentationThe green arrow represents upsampling of the feature where it can be concatenated to the larger features.  Once the image is concatenated, the CNN performs extraction and localization.We must bear in mind that each downsampling block is connected to a consecutive upsampling block


Encoder

In a transformer, the encoder consists of three major components: Embedding layer: It creates n x n number of patches of the image, which is then sequentially fed into the Multi-head self-attention block.  Multi-head self-attention: block yields global representations or features extracted from the given input.  Multilayer perceptron: the MLP consists of two linear layers with Gaussian Error Linear Unit (GELU).  Essentially, it is a classification head that is local and translationally equivariant.  Source: An Image Is Worth 16x16 Words: Transformers For Image Recognition At ScaleThe figure above represents a schematic diagram of the Vision Transformer.  It is worth noting that the Vision Transformer uses only the encoder part of the entire Transformer architecture and not the decoder.  A classification head extends the encoder block of the ViT.To better understand the architecture, we will explore each module separately.  We will explore the encoder module, the transformer module, and the entire ViT module:Encoder module:The core idea of the encoder module is to create a list of attention blocks and then pass the input one at a time through the attention block, followed by normalizing them.1class Encoder(nn.Module): 2 def __init__( 3 self, 4 num_layers, 5 hidden_size, 6 num_attention_heads, 7 linear_dim, 8 dropout_rate, 9 attention_dropout_rate, 10 eps, 11 std_norm, 12 ): 13 super(Encoder, self).__init__() 14 self.layer = nn.ModuleList() 15 self.encoder_norm = LayerNorm(hidden_size, eps=eps) 16 for _ in range(num_layers): 17 layer = Block( 18 num_attention_heads, 19 hidden_size, 20 linear_dim, 21 dropout_rate, 22 attention_dropout_rate, 23 eps, 24 std_norm, 25 ) 26 self.layer.append(copy.deepcopy(layer)) 27 28 def forward(self, hidden_states): 29 attn_weights = [] 30 for layer_block in self.layer: 31 hidden_states, weights = layer_block(hidden_states) 32 attn_weights.append(weights) 33 encoded = self.encoder_norm(hidden_states) 34 return encoded, attn_weights 35CopyTransformer module:This module sequentially arranges the embedding module and the encoder module.1class Transformer(nn.Module): 2 def __init__( 3 self, 4 img_size, 5 hidden_size, 6 in_channels, 7 num_layers, 8 num_attention_heads, 9 linear_dim, 10 dropout_rate, 11 attention_dropout_rate, 12 eps, 13 std_norm, 14 ): 15 super(Transformer, self).__init__() 16 .  17 .  18 .  19 20 def forward(self, input_ids): 21 embedding_output = self.embeddings(input_ids) 22 encoded, attn_weights = self.encoder(embedding_output) 23 return encoded, attn_weights 24CopyVision Transformer:The ViT module, like the transformers module, combines just two modules: the transformer module and a linear layer that acts as a classification head


Self-Attention

Self Attention is an attention mechanism that operates the different positions of a sequence, namely a key-value pair and query, to yield good representations.  The attention mechanism maps a given sequence to an output with a Scaled Dot-Product operation (as shown in the figure above).Source: Attention Is All You NeedQ, K, and V are Query, Key, and Value in the formula above.  dk is the scaling factor.  d is usually the size of attention heads.  Essentially, d can be calculated by the following formula:d = hidden feature/number of attention head Source: Attention Is All You NeedNotably, the capabilities of producing rich representation with self-attention are significantly increased by using the multi-head attention mechanism.  According to the authors, it 'allows the model to jointly attend to information from different representation subspaces at different positions'.This operation allows the model to yield a global representation of the given input.  In NLP, self-attention enables the model to learn the long-term dependencies, i.e., the correlation between the current output and the previous outputs, so that the contextual information is preserved over time.In computer vision, the self-attention mechanism can enable the model to focus on the critical part of the image.  It can neglect or discern the objects, such as the background from the foreground, and yield results based on the same.  It can also enable models to preserve essential features extracted in the early stages of the convolution operation.Source: Show, Attend and Tell: Neural Image Caption Generation with Visual AttentionIn the image classification task, the model can concentrate or pay attention to the objects that yield good attention scores.  The image above shows that the model focuses on the two humans more than the background and the frisbee


Hybridizing Encoder and UNet

So far, we have discussed CNN-based UNet and Attention-based Transformers. Further on, we will see how we can combine both to create a hybrid architecture.We want to create a hybridized UNet because both CNN-based UNet and Transformers have advantages and disadvantages. Combining both can help us leverage both models' advantages in a single model.We have learned that CNN-based UNet has a significant limitation because of the convolutional operations that make them unable to model long-term dependencies. Transformers, on the other hand, can model long-term dependencies. So why not use transformers instead of CNN?Although transformers can model long-term dependencies, they cannot model local features, which CNN is best at. Hence, combining both models can help us to leverage the power of both the CNN and self-attention mechanism to yield better results.How can we combine them both?In the released paper, 'TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation, ' the authors proposed using CNN to create image patches or image embedding that can be fed into the transformer's encoder. The transformer can encode image patches from a CNN feature map as the input sequence to extract global features. The decoder, made up of transpose CNN can then upsamples the encoded features. Like the original UNet at each layer, the upsample features will be combined with the high-resolution CNN feature maps to yield high-quality and precise localization. For a better understanding, look at the image below. Source: TransUNet | Transformers Make Strong Encoders for Medical Image SegmentationAs you can see, the CNN layer extracts local features, and the transformer extracts global features. The local features are combined or concatenated with the upsampling block at each layer for precise segmentation masks. To comprehend the whole architecture, we will break it into different modules and explore the working of what each module does.


Encoder:

We will explore and understand the code released by the authors in their paper.To start with, let us explore the CNN module.  The CNN module leverages ResNet.  The ResNet defined in the paper is ResNet-50.  This network is very efficient in extracting features from the given input.  Although it is a big network, it tackles vanishing gradients using skip connections.Here is the pseudo-code for the ResNet:1class ResNetV2(nn.Module): 2 def __init__(self, block_units, width_factor): 3 .  4 .  5 self.root = '''it consists of conv layers, group norm, and an activation''' 6 self.body = '''it consists of pre-activation bottleneck block ''' 7 8 def forward(self, x): 9 features = [] 10 b, c, in_size, _ = x.size() 11 x = self.root(x) 12 features.append(x) 13 x = nn.MaxPool2d(kernel_size=3, stride=2, padding=0)(x) 14 for i in range(len(self.body)-1): 15 x = self.body[i][18] 16 right_size = int(in_size / 4 / (i+1)) 17 if x.size()[2] != right_size: 18 pad = right_size - x.size()[2] 19 assert pad < 3 and pad > 0, 'x {} should {}'.format(x.size(), right_size) 20 feat = torch.zeros((b, x.size()[1], right_size, right_size), device=x.device) 21 feat[:, :, 0:x.size()[2], 0:x.size()[3]] = x[:] 22 else: 23 feat = x 24 features.append(feat) 25 x = self.body[-1][18] 26 return x, features[::-1] 27CopyResNet defined consists of two essential modules:Root block:The idea of the root block is to standardize the given input image.1nn.Sequential(OrderedDict([ 2 ('conv', StdConv2d(3, width, kernel_size=7, stride=2, bias=False, padding=3)), 3 ('gn', nn.GroupNorm(32, width, eps=1e-6)), 4 ('relu', nn.ReLU(inplace=True)), 5 # ('pool', nn.MaxPool2d(kernel_size=3, stride=2, padding=0)) 6 ])) 7CopyBody block:The body block contains a sequential layer of preactivation CNN module, a bottleneck block extracting vital information from the image.  The preactivation block itself is a ResNet block.1nn.Sequential(OrderedDict([ 2 ('block1', nn.Sequential(OrderedDict( 3 [('unit1', PreActBottleneck(cin=width, cout=width*4, cmid=width))] + 4 [(f'unit{i:d}', PreActBottleneck(cin=width*4, cout=width*4, cmid=width)) for i in range(2, block_units[0] + 1)], 5 ))) 6CopyThe ResNetV2 is then integrated into the Embedding module, which crops the image and feeds it into the transformers.  See the figure below.Source: TransUNet | Transformers Make Strong Encoders for Medical Image SegmentationResNetV2 provides an edge to the feature extraction process.  This hybrid embedding module returns two outputs: the embedded features are passed into the transformers, while the features extracted from the ResNet are passed into the upsampling block.In the forward function of the ResNetV2 module, you will find that the output from each CNN layer, which is located in the body block, is stored in a 'features' list.  This list is reversed once the output from all the CNN layers is stored, enabling the concatenation of global features for upsampling.  Source: TransUNet | Transformers Make Strong Encoders for Medical Image Segmentation1class Embeddings(nn.Module): 2 '''Construct the embeddings from the patch, position embeddings


Encoder

Because it is a classification head, the final output has to pass through the cross-entropy loss.1class VisionTransformer(nn.Module): 2 def __init__( 3 self, 4 img_size, 5 num_classes, 6 hidden_size, 7 in_channels, 8 num_layers, 9 num_attention_heads, 10 linear_dim, 11 dropout_rate, 12 attention_dropout_rate, 13 eps, 14 std_norm, 15 ): 16 super(VisionTransformer, self).__init__() 17 self.classifier = 'token' 18 19 self.transformer = Transformer( 20 img_size, 21 hidden_size, 22 in_channels, 23 num_layers, 24 num_attention_heads, 25 linear_dim, 26 dropout_rate, 27 attention_dropout_rate, 28 eps, 29 std_norm, 30 ) 31 self.head = Linear(hidden_size, num_classes) 32 33 def forward(self, x, labels=None): 34 x, attn_weights = self.transformer(x) 35 logits = self.head(x[:, 0]) 36 37 if labels is not None: 38 loss_fct = CrossEntropyLoss() 39 loss = loss_fct(logits.view(-1, 400), labels.view(-1)) 40 return loss 41 else: 42 return logits, attn_weights 43Copy


Encoder:

3 ''' 4 def __init__(self, config, img_size, in_channels=3): 5 super(Embeddings, self).__init__() 6 self.hybrid = None 7 self.config = config 8 img_size = _pair(img_size) 9 10 if config.patches.get('grid') is not None: # ResNet 11 grid_size = config.patches['grid'] 12 patch_size = (img_size[0] // 16 // grid_size[0], img_size[1] // 16 // grid_size[1]) 13 patch_size_real = (patch_size[0] * 16, patch_size[1] * 16) 14 n_patches = (img_size[0] // patch_size_real[0]) * (img_size[1] // patch_size_real[1]) 15 self.hybrid = True 16 else: 17 patch_size = _pair(config.patches['size']) 18 n_patches = (img_size[0] // patch_size[0]) * (img_size[1] // patch_size[1]) 19 self.hybrid = False 20 21 if self.hybrid: 22 self.hybrid_model = ResNetV2(block_units=config.resnet.num_layers, width_factor=config.resnet.width_factor) 23 in_channels = self.hybrid_model.width * 16 24 self.patch_embeddings = Conv2d(in_channels=in_channels, 25 out_channels=config.hidden_size, 26 kernel_size=patch_size, 27 stride=patch_size) 28 self.position_embeddings = nn.Parameter(torch.zeros(1, n_patches, config.hidden_size)) 29 30 self.dropout = Dropout(config.transformer['dropout_rate']) 31 32 def forward(self, x): 33 if self.hybrid: 34 x, features = self.hybrid_model(x) 35 else: 36 features = None 37 x = self.patch_embeddings(x) # (B, hidden .  n_patches^(1/2), n_patches^(1/2)) 38 x = x.flatten(2) 39 x = x.transpose(-1, -2) # (B, n_patches, hidden) 40 41 embeddings = x + self.position_embeddings 42 embeddings = self.dropout(embeddings) 43 return embeddings, features 44CopyNow we will understand how the two outputs from the embedding module, i.e., embeddings and features, can be processed to acquire the segmentation masks.  The process of acquiring a segmentation mask will be carried out in the Vision Transformer.  But it has to be built in the form of a UNet, which means that we have to add two additional components to the existing transformer architecture: the encoder module.  The additional two components are a decoder and a segmentation head.Let's build each component step by step for better understanding:Transformer or encoder module: This module will receive only the embedded output and then feed it to the attention block followed by the multilayer perceptron.1class Transformer(nn.Module): 2 def __init__(self, config, img_size, vis): 3 super(Transformer, self).__init__() 4 self.embeddings = Embeddings(config, img_size=img_size) 5 self.encoder = Encoder(config, vis) 6 7 def forward(self, input_ids): 8 embedding_output, features = self.embeddings(input_ids) 9 encoded, attn_weights = self.encoder(embedding_output) # (B, n_patch, hidden) 10 return encoded, attn_weights, features 11CopyAs illustrated, since we have the ResNetV2 in the embedding, the transformers yield three outputs instead of two, as seen in earlier encoded features, attention weights, and convolutional features.


TransUnet

Now comes the main component, ViT or TransUnet. This module consists of three critical parts: Transformer, DecoderCup, and Segmentation head. We already saw the transformer architecture. Here we will explore the decoder cup and the segmentation head.a) DecoderCupThe idea of the decoder cup is to upsample the given input.1class DecoderCup(nn.Module): 2 def __init__(self, config): 3 super().__init__() 4 self.config = config 5 head_channels = 512 6 self.conv_more = Conv2dReLU( 7 config.hidden_size, 8 head_channels, 9 kernel_size=3, 10 padding=1, 11 use_batchnorm=True, 12 ) 13 decoder_channels = config.decoder_channels 14 in_channels = [head_channels] + list(decoder_channels[:-1]) 15 out_channels = decoder_channels 16 17 if self.config.n_skip != 0: 18 skip_channels = self.config.skip_channels 19 for i in range(4-self.config.n_skip): # re-select the skip channels according to n_skip 20 skip_channels[3-i]=0 21 22 else: 23 skip_channels=[0,0,0,0] 24 25 blocks = [ 26 DecoderBlock(in_ch, out_ch, sk_ch) for in_ch, out_ch, sk_ch in zip(in_channels, out_channels, skip_channels) 27 ] 28 self.blocks = nn.ModuleList(blocks) 29 30 def forward(self, hidden_states, features=None): 31 B, n_patch, hidden = hidden_states.size() # reshape from (B, n_patch, hidden) to (B, h, w, hidden) 32 h, w = int(np.sqrt(n_patch)), int(np.sqrt(n_patch)) 33 x = hidden_states.permute(0, 2, 1) 34 x = x.contiguous().view(B, hidden, h, w) 35 x = self.conv_more(x) 36 for i, decoder_block in enumerate(self.blocks): 37 if features is not None: 38 skip = features[i] if (i < self.config.n_skip) else None 39 else: 40 skip = None 41 x = decoder_block(x, skip=skip) 42 return x 43Copyb) Segmentation headThis module inherits the nn.Sequential class, which by default has a forward() function. The function of this module is to upsample the final output yielded by the last layer of decodercup to create a segmentation mask.1class SegmentationHead(nn.Sequential): 2 3 def __init__(self, in_channels, out_channels, kernel_size=3, upsampling=1): 4 conv2d = nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, padding=kernel_size // 2) 5 upsampling = nn.UpsamplingBilinear2d(scale_factor=upsampling) if upsampling > 1 else nn.Identity() 6 super().__init__(conv2d, upsampling) 7 8Copyc) Vision TransformerAs usual, this module sequentially arranges all the modules together to create a seamless pipeline.1class VisionTransformer(nn.Module): 2 def __init__(self, config, img_size=224, num_classes=21843, zero_head=False, vis=False): 3 super(VisionTransformer, self).__init__() 4 self.num_classes = num_classes 5 self.zero_head = zero_head 6 self.classifier = config.classifier 7 self.transformer = Transformer(config, img_size, vis) 8 self.decoder = DecoderCup(config) 9 self.segmentation_head = SegmentationHead( 10 in_channels=config['decoder_channels'][-1], 11 out_channels=config['n_classes'], 12 kernel_size=3, 13 ) 14 self.config = config 15 16 def forward(self, x): 17 if x.size()[1] == 1: 18 x = x.repeat(1,3,1,1) 19 x, attn_weights, features = self.transformer(x) # (B, n_patch, hidden) 20 x = self.decoder(x, features) 21 logits = self.segmentation_head(x) 22 return logits 23Copy


How to visualize the UWMGI Dataset?

Once you have downloaded the data, you can explore it by running this script: print(ds.tensors.keys()) dict_keys([‘images', 'labels', ‘masks']) You can use the above keys to get the required output according to the task. Since we are working with segmentation, we want to explore ‘images' and ‘masks'. We can then use these keys to visualize the image.1plt.imshow(ds.tensors['images'][1].numpy()) 2Copy Similarly, we can visualize the masks using the same command:1plt.imshow(ds.tensors['masks'][0].numpy()[:, :, 0]) 2Copy Final image after combining image and its respective segmentation.


What is Transformer in ML?

The Transformer architecture was introduced by Vaswani et al. in 2016. It was designed for sequential modeling, language modeling, or NLP. One of the reasons why transformers are popular is because of the self-attention mechanism, which yields global feature extraction compared to local feature extraction, as we see in CNN. Due to its global feature extraction capabilities, it serves as a decent tool for image segmentation.Source: Attention Is All You Need


Querying the Data

Data querying is an essential tool for ML practitioners as it can enable them to filter out necessary information according to the requirement to train the model. For instance, datasets such as the CoCo dataset have 91 classes, of which only 80 classes contain segmented masks. If we are building a DL model that has to be trained to classify only humans, then we can filter out images that only contain classes pertaining to humans.Deep Lakes enables its users to filter out images in two ways, i.e., using UI and CLI. Let's explore both of them in detail.


How to download UWMGI Dataset?

To download the data, you can follow the steps given below: Install the Python package: pip3 install deeplake Visit https://app.activeloop.ai/ Select the data: You can search for the dataset you want to work within the search bar. In our case, it will be 'UWMGI' Once you find the dataset copy, the link provided. Source: Deep Lake Access the data: Now, let's download the data from the Activeloop server by following the code below:1import deeplake 2import torch 3from torchvision import transforms, models 4 5ds = deeplake.load('hub://perceptronai/UWMGI| training') 6Copy hub://perceptronai/UWMGI-1 loaded successfully. This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/perceptronai/UWMGI-1 1tform_i = transforms.Compose([ 2 transforms.ToTensor(), 3 transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]), 4]) 5 6tform_m = transforms.Compose([ 7 transforms.ToTensor()]) 8 9deeplake_loader = ds.pytorch(batch_size=1, transform={ 'images': tform_i, 'masks': tform_m}) 10CopyThe ds.pytorch class works exactly like the data loader class from Pytorch. You can use the same methodology to get the images and masks. For instance,1imgs, msks = next(iter(deeplake_loader)) 2CopyDeep Lake also provides a way to visualize the images. You can use the simple command ds.visualize() to visualize the images.Here are some additional practices that can help you to understand and discover once you download the data.


How to query datasets using Python?

Querying data using CLI is quite simple. Activeloop provides filtering with user-defined functions which allow you to filter data based on your requirements. Here are the steps which can enable filtering data.a) Load the data.1ds = deeplake.load('hub://perceptronai/UWMGI| training') 2Copyb) Create a list of objects that you want to filter.1labels_list = ['stomach', 'large bowel'] 2Copyc) Create a function for filtering data using the deeplake.compute decorator.1labels_list = ['stomach'] 2 3@deeplake.compute 4def filter_labels(sample_in, labels_list): 5 # print(sample_in.labels.data()['text'][0]) 6 return sample_in.labels.data()['text'][0] in labels_list 7Copyd) We can start filtering the data using the ds.filter function, where we will pass the filter_labels function.1ds_view = ds.filter(filter_labels(labels_list)) ``` 2 3 Another way to filter data is by using the ds.query function. 4 5```python 6print(ds.query('SELECT * WHERE CONTAINS(labels, 'stomach')')) 7Copy Dataset(path='hub://perceptronai/UWMGI| training', index=Index([(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99)]), tensors=['images', 'labels', ‘masks']) Once the data is filtered, you can check the length of the data and even plot an image and its corresponding segmentation mask.1print(len(ds_view)) 2Copy 100 1Image.fromarray(ds_view.images[1].numpy()) 2Copy Once the data is filtered, you can use it to train your DL model.


Training and Testing Loop

Once the dataset is downloaded, we can then start defining the model.  We will be finetuning the model since the authors have already uploaded the weights and biases in their repository.  To do that, we will clone the repository.  You can find the complete training code here.1!git clone https://github.com/Beckschen/TransUNet 2CopyOnce we have cloned the repository, we can then import the model.1from TransUNet.networks.vit_seg_modeling import VisionTransformer as ViT_seg 2from TransUNet.networks.vit_seg_modeling import CONFIGS as CONFIGS_ViT_seg 3CopyWe will then configure the model according to our requirements.  We can define a class that will download the pre-trained weights and biases.  We can also use the same class to define the model.  The following code is an example of how we can download and configure the model:1class TransUnet(nn.Module): 2 def __init__(self, 3 model_name=CFG.MODEL_NAME, 4 pretrain_path='./R50+ViT-B_16.npz', 5 n_classes=3, 6 n_skip=3, 7 dropout_rate=0.2, 8 mlp_dim=3072, 9 num_heads=12, 10 num_layers=8, 11 img_size=CFG.img_size[0]): 12 13 super(TransUnet, self).__init__() 14 15 #config 16 config_vit = CONFIGS_ViT_seg[model_name] 17 config_vit.pretrained_path = pretrain_path 18 config_vit.n_classes = n_classes 19 config_vit.n_skip = n_skip 20 config_vit.transformer.dropout_rate = dropout_rate 21 config_vit.transformer.mlp_dim = mlp_dim 22 config_vit.transformer.num_heads = num_heads 23 config_vit.transformer.num_layers = num_layers 24 25 #model 26 self.model = ViT_seg(config_vit, img_size=img_size, num_classes=n_classes) 27 self.model.load_from(weights=np.load(config_vit.pretrained_path)) 28 29 def forward(self, x): 30 img_segs = self.model(x) 31 32 return img_segs 33 34model = TransUnet() 35CopyLet us define the other essential modules, including the loss function, optimizer, and scheduler.  If your task requires you to predict binary segmentation masks, then you can use LovaszSoftmax, Hausdorff, FocalLoss, DiceLoss, or DiceBCELoss


Training and Testing Loop

These loss functions can be downloaded from the following git repo:1!git clone https://github.com/JunMa11/SegLoss.git 2CopyBut since our task requires us to predict multilabel segmentation, we will use CrossEntropyLoss.1def Loss(): 2 if CFG.criterion == 'Multiclass': 3 criterion = nn.CrossEntropyLoss() 4 elif CFG.criterion == 'DiceBCELoss': 5 criterion = DiceBCELoss() 6 elif CFG.criterion == 'DiceLoss': 7 criterion = DiceLoss() 8 elif CFG.criterion == 'FocalLoss': 9 criterion = FocalDLoss() 10 elif CFG.criterion == 'Hausdorff': 11 criterion = Hausdorff_loss() 12 elif CFG.criterion == 'Lovasz': 13 criterion = Lovasz_loss() 14 return criterion 15CopyWe will use an Adam optimizer with a learning rate of 2e-3 and a scaler for the backward propagation and uniform scaling of gradients, respectively.1optimizer = Adam(model.parameters(), lr=CFG.lr) 2scaler = torch.cuda.amp.GradScaler() 3CopyNow we can write our training loop and start the finetuning process.1def train_engine(model, train_loader, device=CFG.device): 2 epoch_loss = [] 3 loss_list = [] 4 acc = [] 5 num_correct = 0 6 num_pixels = 0 7 8 model = model.to(device) 9 model.train() 10 for epoch in range(20): 11 loop = tqdm(enumerate(train_loader),total=len(train_loader), desc='training') 12 for batch_idx, (X, y) in loop: 13 X = X.to(device) 14 y = y.to(device) 15 # forward 16 with torch.cuda.amp.autocast(): 17 logits = model(X) 18 loss = Loss()(logits, y) 19 20 # backward 21 optimizer.zero_grad() 22 scaler.scale(loss).backward() 23 scaler.step(optimizer) 24 scaler.update() 25 26 #loss and accuracy 27 softmax = nn.Softmax(dim=1) 28 preds = softmax(logits) 29 preds = torch.argmax(preds,axis=1) 30 num_correct += (preds == y).sum() 31 num_pixels += torch.numel(preds) 32 loss_list.append(loss.item()) 33 accuracy=num_correct.item()/num_pixels*100 34 acc.append(accuracy) 35 # update tqdm loop 36 loop.set_postfix(loss=loss.item(), accuracy=accuracy) 37 epoch_loss.append(loss.item()) 38 return loss_list, epoch_loss, acc 39 40loss, epoch_loss, accuracy = train_engine(model, train_loader) 41Copy training: 79%|███████▊ | 23606/30000 [58:13<15:46, 6.76it/s, accuracy=67.8, loss=0.23] The result after the first epoch: One of the advantages of using a pre-trained model is that the model learns faster than training the model from scratch .  As you can see from the image above, the model is starting to learn anomalies.When we train the model for a more extended period, the model will start predicting accurate segmentation masks.  Please check the following section for the final results.


Concluding remarks: UNet vs TransUNet

This article showed how to finetune & build a deep-learning model for machine learning in radiology.UNet remains one of the primary architectures for the task of image segmentation, no matter what the field is. Especially in radiology, it can also impact lives. Accuracy score is an essential metric for developing UNet for medical purposes. Lee et al. introduced a variant of UNet predicting affinities between the nearest neighbors. The network can leverage this information from the derived affinities to create a segmentation mask by greedily clustering the mean affinity. This implies that UNet can achieve superhuman accuracy in predictive tasks.Likewise, many methods, such as error detection and correction tasks (both important in image segmentation), may enhance the model's accuracy.TransUNet can leverage both of these techniques. We've already seen how TransUNet uses the multi-head self-attention mechanism to preserve long-range dependencies. Similarly, we can enhance the preserving power of TransUNet by predicting the affinity of the neighbor, clustering likable data points, and creating segmentation masks. This, combined with error detection, increases the accuracy of the model.


References

Deep Lake: a Lakehouse for Deep Learning U-Net: Convolutional Networks for Biomedical Image Segmentation Attention Is All You Need An Image Is Worth 16x16 Words: Transformers For Image Recognition At Scale TransUNet | Transformers Make Strong Encoders for Medical Image Segmentation https://github.com/Beckschen/TransUNet https://github.com/JunMa11/SegLoss.git : Loss functions for image segmentation https://www.kaggle.com/stainsby/fast-tested-rle https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/data https://www.kaggle.com/code/awsaf49/uwmgi-unet-train-pytorch Activeloop Superhuman Accuracy on the SNEMI3D Connectomics Challenge An Error Detection and Correction Framework for Connectomics Show, Attend and Tell: Neural Image Caption Generation with Visual Attention


How to query datasets using Deep Lake UI?

To filter data using Activeloop's UI, you only need to click on the 'Run query' button, as shown in the image below. Source: Deep LakeOnce you click on the button, you will get the SQL command line just below the button. You can explore the queries by clicking on the 'Example Queries' button on the command line.We will use the second command to filter out images that contain only humans. We can copy-paste the command into the SQL command line and press shift+enter to execute it. Source: Deep LakeYou can then use the 'Save query result' button to save the query. Source: Deep LakeAfter saving the query result, you can click on 'Query history' to get the saved query. Source: Deep LakeAfter which, you can use the query id, which will serve as a Python API enabling you to train the DL model on the filtered dataset.


Key Takeaways

Here are some of the key takeaways from this article: Pre-trained TransUNet can be used in radiology machine learning projects. The TransUnet architecture is SOTA regarding image segmentation, as it can model hard-to-find anomalies. Because the network uses both CNN and a self-attention mechanism, it can find local features and preserve them longer, thus enabling it to find features that would typically be out-of-reach. As seen below, the network can predict correct segmentation masks even for images that aren't clear, showing the reliability of TransUnet. Deep Lake provides an efficient way to load, query, visualize, & stream the data for training and testing purposes. The queried dataset can be saved and later materialized, which enables reproducibility.


What is UWMGI Dataset?

The UWMGI dataset is the competition dataset released in Kaggle. The dataset represents gastrointestinal tract images, consisting of roughly 50 cases. The images in each case are a 16-bit grayscale in a PNG format. The segmentation masks representing three classes, stomach, small and large bowels, are provided as Run Length encoded or RLE-encoded masks.You can check out this notebook that explains how you can download the data from Kaggle and upload it to Deep Lake after preprocessing it to the required Numpy format. Source: Deep Lake


The New Way to Understand Code Repositories

The new way is just four steps that take less than an hour to build: Index the codebase Store embeddings and the code in Deep Lake Use Conversational Retriever Chain from LangChain Ask any questions you'd like! Now, this doesn't mean you don't need to take the steps outlined above in the previous section, but we do hope this new approach aids the learning speed along the way. We will delve deeper into this process below, but let's review the basics first.


LangChain basics

Before moving on to the process and architecture behind code comprehension, let's first understand the basics.


What is Deep Lake as a Vector Store in LangChain?

In the LangChain ecosystem, Deep Lake is a serverless, open-source, and multi-modal vector store. Deep Lake not only stores embeddings but also the original data with automatic version control. For these reasons, Deep Lake can be considered one of the best Vector Stores for LangChain (if you ask us, haha!). Deep Lake goes beyond a simple vector store, but we'll dive into it in another blog post.


What is LangChain?

In essence, LangChain is a wrapper for utilizing Large Language Models like GPT-4 and connecting them to many tools (such as vector stores, calculators, or even Zapier). LangChain is especially appealing to developers because it offers a novel way to construct user interfaces. Instead of relying on dragging and dropping or coding, users can state their desired outcome. Broadly speaking, LangChain is enticing to devs as it augments already robust LLMs with memory and context (which comes in handy in tasks such as code understanding). By artificially incorporating 'reasoning,' we can tackle more sophisticated tasks with greater precision.If you want to learn more about LangChain, read the ultimate guide on LangChain. In this example, we build ChatGPT to answer questions about your financial data. If you were to use an LLM about the top-performing quarter of all time for Amazon (maybe after feeding it a copy-paste text from a pdf), It would likely produce a plausible SQL query to retrieve the result using fabricated yet real-sounding column names. However, using LangChain, you can compose a workflow that would iteratively go through the process and arrive at a definitive answer, such as 'Q4 2022 was the strongest quarter for Amazon all-time'. You can read more about analyzing your financial data with LangChain.


How to Build Code Understanding App with LangChain, GPT-4, & Conversational Retriever Chain?

Index the Codebase: Duplicate the target repository, load all contained files, divide the files, and initiate the indexing procedure. Alternatively, you can bypass this step and use a pre-indexed dataset. Store Embeddings and the Code: Code segments are embedded using a code-aware embedding model and saved in the Deep Lake VectorStore. Assemble the Retriever: Conversational Retriever Chain searches the VectorStore to find a specific query's most relevant code segments. Build the Conversational Chain: Customize retriever settings and define any user-defined filters as necessary. Pose Questions: Create a list of questions about the codebase, then use the Conversational Retrieval Chain to produce context-sensitive responses. The LLM (GPT-4, in this case) should now generate detailed, context-aware answers based on the retrieved code segments and conversation history.


What is LangChain Conversational Retriever Chain?

A conversational Retriever Chain is a retrieval-centric system interacting with data stored in a VectorStore like Deep Lake. It extracts the most applicable code snippets and details for a specific user request using advanced methods like context-sensitive filtering and ranking. The conversational Retriever Chain is designed to provide high-quality, relevant outcomes while considering conversation history and context.


The Legacy Way to Understand Code

Acquire a broad comprehension of the codebase's role within the project. Read the codebase documentation. Develop a dependency map for the codebase to comprehend its organization and interconnections. Examine the primary function to grasp the code's structure. Ask a colleague, 'wtf is the main function doing?'. For test-driven development, execute test cases and use breakpoints to decipher the code. If test cases exist but are outside test-driven development, review them to comprehend the specifications. Shed a few tears. Employ a debugger to step through the code if test cases are absent. Examining Git history can reveal the codebase's evolution and areas more susceptible to modifications. Alter the code and introduce personal test cases to assess the consequences. Investigate previous alterations to identify potential impact areas and confirm your assumptions. Continually monitor changes made by teammates to remain informed on current advancements.


Step 2: Indexing the Twitter Algorithm Code Base (Optional)

You can skip this part and jump right into using an already indexed dataset (just like the one in this example). To index the code base, first clone the repository, parse the code, break it into chunks, and apply OpenAI indexing:1!git clone https://github.com/twitter/the-algorithm # replace any repository of your choice 2CopyNext, load all files inside the repository.1import os 2from langchain.document_loaders import TextLoader 3 4root_dir = './the-algorithm' 5docs = [] 6for dirpath, dirnames, filenames in os.walk(root_dir): 7 for file in filenames: 8 try: 9 loader = TextLoader(os.path.join(dirpath, file), encoding='utf-8') 10 docs.extend(loader.load_and_split()) 11 except Exception as e: 12 pass 13CopySubsequently, divide the loaded files into chunks:1from langchain.text_splitter import CharacterTextSplitter 2 3text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) 4texts = text_splitter.split_documents(docs) 5 6CopyPerform the indexing process. This takes roughly 4 minutes to calculate embeddings and upload them to Activeloop. Afterward, you can publish the dataset publicly:1username = 'davitbun' # replace with your username from app.activeloop.ai 2db = DeepLake(dataset_path=f'hub://{username}/twitter-algorithm', embedding_function=embeddings) 3db.add_documents(texts) 4CopyIf the dataset has been already created, you can load it later without recomputing embeddings as seen below.


Step 3: Conversational Retriever Chain

First, load the dataset, establish the retriever, and create the Conversational Chain:1db = DeepLake(dataset_path='hub://davitbun/twitter-algorithm', read_only=True, embedding_function=embeddings) 2CopyA preview of the dataset would look something like this:1Dataset(path='hub://davitbun/twitter-algorithm', read_only=True, tensors=['embedding', 'ids', 'metadata', 'text']) 2 3 tensor htype shape dtype compression 4 ------- ------- ------- ------- ------- 5 embedding generic (23152, 1536) float32 None 6 ids text (23152, 1) str None 7 metadata json (23152, 1) str None 8 text text (23152, 1) str None 9Copy1retriever = db.as_retriever() 2retriever.search_kwargs['distance_metric'] = 'cos' 3retriever.search_kwargs['fetch_k'] = 100 4retriever.search_kwargs['maximal_marginal_relevance'] = True 5retriever.search_kwargs['k'] = 10 6CopyYou can also define custom filtering functions using Deep Lake filters:1def filter(x): 2 if 'com.google' in x['text'].data()['value']: 3 return False 4 metadata = x['metadata'].data()['value'] 5 return 'scala' in metadata['source'] or 'py' in metadata['source'] 6 7# Uncomment the following line to apply custom filtering 8# retriever.search_kwargs['filter'] = filter 9CopyConnect to GPT-4 for question answering.1from langchain.chat_models import ChatOpenAI 2from langchain.chains import ConversationalRetrievalChain 3 4model = ChatOpenAI(model='gpt-3.5-turbo') # switch to 'gpt-4' 5qa = ConversationalRetrievalChain.from_llm(model,retriever=retriever) 6Copy


Step 1: Installing required libraries and authenticating with Deep Lake and Open AI

First, we will install everything we'll need.1!python3 -m pip install --upgrade langchain deeplake openai tiktoken 2 3CopyNext, let's import the necessary packages and make sure the Activeloop and OpenAI keys are in the environmental variables ACTIVELOOP_TOKEN, OPENAI_API_KEY and define the OpenAI embeddings. For full documentation of Deep Lake please the Deep Lake LangChain docs page and the Deep Lake API reference.You'd need to authenticate into Deep Lake if you want to create your own dataset and publish it. You can get an API key from the Deep Lake platform here1import os 2import getpass 3 4from langchain.embeddings.openai import OpenAIEmbeddings 5from langchain.vectorstores import DeepLake 6 7os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:') 8os.environ['ACTIVELOOP_TOKEN'] = getpass.getpass('Activeloop Token:') 9embeddings = OpenAIEmbeddings() 10Copy


The Twitter Recommendation Algorithm

Ironically, we will use some words to describe the Twitter Algorithm for the general audience. Still, you can skip right to the code part (that will answer even more questions on how the Twitter algorithm works in 2023).With approximately 500 million Tweets daily, the Twitter recommendation algorithm is instrumental in selecting top Tweets for your the 'For You' feed. The Twitter trending algorithm employs intertwined services and jobs to recommend content across different app sections, such as Search, Explore, and Ads. However, we will focus on the home timeline's 'For You' feed.


Twitter Recommendation Pipeline

Twitter's open-sourced recommendation algorithm works in three main steps: Candidate Sourcing (fancy speak for data aggregation): the algorithm collects data about your followers, your tweets, and you. The 'For You' timeline typically comprises 50% In-Network (people you follow) and 50% Out-of-Network (people you don't follow) Tweets. Feature Formation & Ranking: Turns the data into key feature buckets: Embedding Space (SimClusters and TwHIN), In Network (RealGraph and Trust & Safety), and Social Graph (Follower Graph, Engagements); look for our practical example to discover what each of those is. Later, a neural network trained on Tweet interactions to optimize for positive engagement is used to obtain the final ranking. Mixing: Finally, in the mixing step, the algorithm groups all features into candidate sources and uses a model called Heavy Ranker to predict user actions, applying heuristics and filtering.


Code Understanding with LangChain: Asking about Functions or Algorithms

Question: How does Heavy ranker work. what are it's main inputs? Answer: The Heavy Ranker, a vital machine learning model, ranks and scores candidates within the recommendation algorithm, mainly predicting user engagement with tweets or connections on the platform.Key Heavy Ranker inputs include: Static Features: Features calculated directly from a tweet as it's created, such as URL presence, cards, quotes, etc. The Index Ingester generates these features and stores them in the index. Real-time Features: Per-tweet features that change post-indexing, including social engagements (retweet, favorite, reply counts) and spam signals. The Signal Ingester, part of a Heron topology, processes event streams to gather and compute these features. User Table Features: Per-user features obtained from the User Table Updater, processing a stream written by the user service. This input stores sparse real-time user data, later linked to the tweet being scored via author lookup. Search Context Features: Features representing the searcher's context, like UI language, content consumption, and current time (implied). These features are combined with Tweet Data for scoring purposes. The Heavy Ranker processes these inputs to score and rank candidates based on relevance and user engagement likelihood. Question: What are some unexpected fingerprints for spam factors? Answer: Within the given context, a peculiar spam sign is a tweet containing a non-media, non-news link. If a tweet has a link without an image, video, or news URL, it's seen as a potential spam source, setting a user reputation threshold (tweepCredThreshold) to MIN_TWEEPCRED_WITH_LINK.Note that this rule may not contain all atypical spam indicators but is based on the particular codebase and logic provided in the context.


Code Understanding with LangChain: Asking about Variables

Question: What is favCountParams? Is it just Likes and Bookmarks? Answer: From the provided code, it is not clear if the favorite count metric is determined by the sum of likes and bookmarks. The favorite count is mentioned in the code, but there is no explicit reference to how it is calculated in terms of likes and bookmarks. Question: What are the major negative modifiers that lower your linear ranking parameters? Answer: In this code repository, the major negative modifiers lowering the linear ranking parameters are: scoringData.querySpecificScore: This score adjustment is based on the query-specific information. If its value is negative, it will lower the linear ranking parameters. scoringData.authorSpecificScore: This score adjustment is based on the author-specific information. If its value is negative, it will also lower the linear ranking parameters. If the author has been blocked, muted, reported for abuse or spam, or has been unfollowed (over the last 50 day rolling period) - the algorithm will adjust the score negatively.


Step 4: Ask Questions to the Codebase in Natural Language

Define all the juicy questions you want to be answered:1questions = [ 2 'What does favCountParams do?', 3 'is it Likes + Bookmarks, or not clear from the code?', 4 'What are the major negative modifiers that lower your linear ranking parameters?', 5 'How do you get assigned to SimClusters?', 6 'What is needed to migrate from one SimClusters to another SimClusters?', 7 'How much do I get boosted within my cluster?', 8 'How does Heavy ranker work. what are it's main inputs?', 9 'How can one influence Heavy ranker?', 10 'why threads and long tweets do so well on the platform?', 11 'Are thread and long tweet creators building a following that reacts to only threads?', 12 'Do you need to follow different strategies to get most followers vs to get most likes and bookmarks per tweet?', 13 'Content meta data and how it impacts virality (e.g. ALT in images).', 14 'What are some unexpected fingerprints for spam factors?', 15 'Is there any difference between company verified checkmarks and blue verified individual checkmarks?', 16] 17chat_history = [] 18 19for question in questions: 20 result = qa({'question': question, 'chat_history': chat_history}) 21 chat_history.append((question, result['answer'])) 22 print(f'-> **Question**: {question} \n') 23 print(f'**Answer**: {result['answer']} \n') 24CopyFinally, configure the conversational model and chain:The output would be a series of questions and answers. We've redacted the answers of the model for brevity. You're welcome to read through the re-run of the notebook here)


Concluding Remarks: Analyzing Codebase with LangChain and Deep Lake

In conclusion, the powerful combination of LangChain, Deep Lake, and GPT-4 revolutionizes code comprehension, making it faster and more efficient. Developers can quickly grasp complex codebases like Twitter's recommendation algorithm using four key steps: indexing the codebase, storing embeddings and code in Deep Lake, using LangChain's Conversational Retriever Chain, and asking questions in natural language.Hopefully, this powerful combination of tools enables developers to quickly gain insights into the inner workings of any code repository, eliminating the need for tedious, time-consuming methods. Since the release of this blogpost, we've seen some great usage of our code to build exciting projects, just like this one:


Code Understanding with LangChain: Asking about Suggestions Based on the Code

You can also brainstorm with the source code in mind on how to accomplish a certain task. Some of the output will be more general (and generic), but it can be finetuned to heavily base its answers on what the model detects in the source code. Question: Do you need to follow different strategies to get most followers vs to get most likes and bookmarks per tweet? Growing followers: The primary objective is to expand your audience. Tactics include:____ Sharing top-notch, niche-specific content regularly to gain autority within SimCluster. Interacting with users through replies, retweets, and mentions. Engage with your commenters: recap.engagement.is_replied_reply_engaged_by_author in Heavy Ranker gives you 75x boost. Utilizing pertinent hashtags and joining popular discussions. Partnering with influencers and users with sizable followings. Publishing content when your target audience is most active. Boosting likes and bookmarks per tweet: The goal is to produce content that connects with your existing followers and promotes engagement. Tactics include: Creating tweets on topics that are trending. tweetHasTrendBoost gives you a 1.1x boost for your tweet to be shown to people. Incorporating eye-catching visuals like images, or videos. twieetHasImageUrlBoost or twieetHasVideoUrlBoost will get you a 2x boost for having a video or image in your tweet. Posing questions, expressing opinions, or starting conversations for enhanced user engagement.


Top 7 Twitter Recommendation Algorithm Tips: How to Trend on Twitter

Here's some other interesting facts we've found from our exploration of the Twitter code base. Perhaps they'll help you to gain a larger Twitter following and even trend on Twitter!To be more visible on Twitter, you should: Aim for more likes and bookmarks as they give your tweet a significant boost. Encourage retweets as they give your tweet a 20x boost. Include videos or images in your tweets for a 2x boost. Avoid posting links or using unrecognized languages to prevent deboosts. Create content that appeals to users in your SimClusters to increase relevance. Engage in conversations by replying to others and encouraging replies to your tweets. Maintain a good reputation by avoiding being classified as a misinformation spreader, blocked, muted, reported for abuse, or unfollowed.


Behind the Scenes at Activeloop

In 2016, before starting the company, I started my Ph.D. research at the Connectomics lab at Princeton Neuroscience Institute. I have witnessed the transition from a gigabyte to a terabyte, then to petabyte-scale datasets to achieve super-human accuracy in reconstructing neural connections inside a brain in just several years. Our problem was to figure out how to optimize and cut the cost 4-5x by rethinking how the data is stored, streamed from storage to the compute, which models to use, and how to compile them and run them at scale. While the industry moved slowly, we have observed how similar patterns repeat themselves on a much larger scale.We started Activeloop (formerly known as Snark AI) as part of the Y Combinator Summer 2018 batch to enable organizations to be more efficient at deploying deep learning solutions. We helped build a large language model for patents for a legal tech startup and streamable data pipelines for the petabyte-scale machine learning use case in AgriTech. Through trial and error and talking to hundreds of companies, we learned that all the awesome databases, data warehouses, and data lakes (joined by lakehouses) are great at analytical workloads but not as much for deep learning applications. The demand for storing unstructured data such as audio, video, and images has exploded over the years (more than 90% of the data is now generated in unstructured form). We knew that building the database for AI, the solution to store it, was the proper challenge for us.In 2020, we open-sourced the dataset format called 'Hub', which enabled storing images, videos, and audio as chunked arrays on objects storages and connecting to deep learning frameworks such as PyTorch or Tensorflow. We have collaborated with teams from Google AI, Waymo, Oxford University, Yale University, and other deep learning groups to figure out the nuts and bolts of a solid data infrastructure for deep learning applications.In 2021, the open-source project trended #1 in Python & #2 across all GitHub repositories and was even named as one of the top 10 python ML packages. As of writing this post, the project has 4.8K stars, 75+ contributors, and +1K community members. It is in production both at research institutions, startups, and public companies.We also released the managed version of Activeloop that lets you visualize datasets, version-control or query them, and stream to deep learning frameworks. Apart from providing access to 125+ machine learning datasets, it enables sharing private datasets and collaboration on building and maintaining datasets across organizations. Of course, I couldn't be more proud of our small and under-resourced team for achieving results in such a short time, but the industry has been innovating at a staggering speed.


Executive Summary

One of three ML projects fails due to the lack of a solid data foundation. Projects suffer from low-quality data, under-utilized compute resources, and significant labor overhead required to build and maintain large amounts of data. Traditional data lakes break down data silos for analytical workloads, enable data-driven decision-making, improve operational efficiency, and reduce organizational costs. However, most of these benefits are unavailable for deep learning workloads such as natural language processing (NLP), audio processing, computer vision, agriculture, healthcare, multimedia, and robotics/automotive, and safety & security verticals. Hence repeatedly, organizations opt-in to develop in-house systems. Deep Lake maintains the benefits of a vanilla data lake and enables you to iterate on your deep learning models 2x faster without teams spending time building complex data infrastructure. Deep Lake stores complex data, such as images, videos, annotations, embeddings, and tabular data, in the form of tensors and rapidly streams the data over the network to Tensor Query Language, in-browser visualization engine, and deep learning frameworks without sacrificing GPU utilization. As deep learning rapidly takes over traditional computational pipelines, storing datasets in a Deep Lake will become the new norm


Machine Learning Loop with Deep Lake

These are five fundamental pillars of Deep Lake. Version Control: Git for data Visualize: In-browser visualization engine Query: Rapid queries with Tensor Query language Materialize: Format native to deep learning Stream: Streaming Data Loaders We discuss those features in-depth in the Deep Lake White Paper and shed light on how it works in the Academic Paper.


What is the Deep Lake?

Deep Lake is a vanilla data lake for deep learning, but with one key difference. Deep Lake stores complex data, such as images, audio, videos, annotations, embeddings, and tabular data, in the form of tensors and rapidly streams the data over the network to Tensor Query Language, an in-browser visualization engine, or deep learning frameworks without sacrificing GPU utilization.Deep Lake provides key features that make it the optimal data storage platform for deep learning applications, including: A scalable and efficient data storage system that can handle large amounts of complex data in a columnar fashion Querying and visualization engine to fully support multimodal data types Native integration with deep learning frameworks and efficient streaming of data to models and back Seamless connection with MLOps tools.


Deep Lake and the Data Loader Landscape

Data loaders are one of the more significant bottlenecks of machine learning pipelines (Mohan et al., 2020), and we've built Deep Lake to specifically resolve the data to compute handoff bottleneck.We are thankful to Ofeidis, Kiedanski, & Tassiulas from Yale Institute For Network Science, who have spent a lot of time producing an independent, extensive survey, & benchmarking of open-source data loaders. The research concluded that the third major iteration of our product, Deep Lake, is not only 2x faster than the previous version but is superior to other data loaders in various scenarios.*Comparing the performance of Activeloop Hub, Deep Lake, and Webdataset when loading data from different locations: Local, AWS, and MinIO. (Ofedis et al. 2022) *Speed as a function of the number of workers for RAN- DOM on a single GPU. (Ofedis et al. 2022)


Large Foundational Models Taking Over by Storm

Deep learning achieved super-human accuracy in applications across industries in a few years. Cancer detection from X-Ray images, anatomical reconstruction of human neural cells, playing highly complex games such as Dota or Go, driving cars, unfolding proteins, having human-like conversations, generating code, and even realistic images that took the internet by storm (it took about 40 words to create the perfect prompt, but AI generated the stunning title image of this post). Three factors enable this speed: (1) novel architectures such as Transformers, (2) massive compute capabilities using GPUs or TPUs, and the large volume of datasets such as ImageNet, CommonCrawl, and LAION-400M.At Activeloop, we firmly believe that connecting deep learning models to the value chain in the next five years will produce a foundational shift in the global economy. While innovators primarily focused on models and computing hardware, maintaining or streamlining the complex data infrastructure has been an afterthought. In the build versus buy dilemma, organizations (for the lack of a 'buy' option) repeatably build hard-to-manage in-house solutions. All this led us to decide on the next chapter for the company - Deep Lake.


Where does Deep Lake fit in the MLOps?

As numerous MLOps tools get into the market, it becomes hard for buyers to understand the landscape. We collaborated with the AI Infrastructure Alliance to craft the new MLOps blueprint that provides a clear overview across tools. The blueprint goes bottom-up from infrastructure to human interface and left-to-right from ingestion to development. In the blueprint, Deep Lake takes on the role of a solid data foundation.


Why we renamed Hub to Deep Lake?

Originally, Hub was a chunked array format that naturally evolved with version control, streaming engine, and query capabilities. Our broad open-source community - users from companies, startups, and academia were instrumental in iterating on the product. Increasingly, we found the name too generic of a descriptor (or, as one of our team members put it, 'everyone has a Hub nowadays'). Often, it would cause confusion with dataset hubs. Internally, we were calling it a 'deep lake' (or named it after the deepest lakes in the world). We were delighted to see people like A. Pinhassi also think in the same direction. Overnight, calling the tool, we're building 'deeplake' instead of 'hub', which felt just right (although our marketing department wasn't too thrilled on account of freshly-ordered swag with the Hub branding).pip3 install deeplake


The reasoning behind some of Deep Lake's architectural decisions

Naturally, it took a lot of thinking and iteration cycles to arrive at the way Deep Lake is architected - and here are a few of considerations we've had.


Is there a Deep Lakehouse, and where does it come into place?

The format includes versioning, and lineage is fully open-source. Query, streaming, and visualization engines are built in C++ and are closed source for the time being. Nonetheless, they are accessible via Python interface for all users. Being committed to open-source principles, we plan to open-source high-performance engines as they commoditize.


What's next for Deep Lake?

As Deep Lake evolves, we will continuously optimize the performance, add a custom data sampler, sub-tile queries for constructing complex datasets for the 3.1.0 release, performant TensorFlow support, and ACID transactions scheduled for the 3.2.0 release (watch our GitHub repo to stay tuned).We believe that the next step for AI research is to capture text, audio, images, and videos by large multi-modal foundational models. Just think about how many days it took to get to Dall-E and how many took from that milestone to Stable Diffusion or Make-A-Video by Meta AI. Having a solid data infrastructure is going to be a necessary condition for delivering those models into consumers' hands. As deep learning rapidly takes over traditional computational pipelines, storing datasets in a Deep Lake is becoming the new norm.You can dive right into Deep Lake (yes, we will be making endless water puns) by trying out this Getting Started with Deep Lake Colab, and checkout our new C++ dataloader and query engine (Alpha) in this Colab. Join our slack community or book an introductory call with us if you want to start the onboarding immediately.Citations The Future of Deep Learning with Deep Lake. Activeloop Hambardzumyan, Sasun, et al. 'Deep Lake: a Lakehouse for Deep Learning.' arXiv preprint arXiv:2209.10785 (2022). Deep Lake — an architectural blueprint for managing Deep Learning data at scale — part I. Pihnasi, A. Ofeidis, Kiedanski, & Tassiulas 'An overview of the data-loader landscape: comparative analysis.' arXic preprint arXiv:2209.13705 (2022) Mohan, Jayashree, et al. 'Analyzing and mitigating data stalls in dnn training.' arXiv preprint arXiv:2007.06775 (2020).


Does Deep Lake connect to the Modern Data Stack and MLOps tools?

The Deep Lake Airbyte destination allows ingesting a dataset from vast amounts of data sources. On the MLOps side, we have been collaborating with W&Bs, Heartex LabelStudio, Sama, CleanLab, AimStack, and Anyscale Ray to provide seamless integrations, which we are going to release in subsequent blog posts.


What is image augmentation?

Image augmentation is a technique in computer vision to supplement the dataset with artificial variations of existing images. The goal of image augmentation is to prevent overfitting, increase the diversity of the data, and make the model more robust to different types of data. Augmentation can be conducted with transformations such as rotation, scaling, flipping, cropping, or adding noise to the images. The newly synthesized images are used to train the machine learning models, leading to better generalization and improved performance on unseen data.For example, consider a deep learning model trained to recognize handwritten digits. Suppose, like in the MNIST dataset, the training data consists of only 28x28 pixel grayscale images of numbers centered in the picture. In that case, the model may only be able to recognize digits that appear in a similar fashion. The training data can be augmented by rotating, scaling, skewing the images, adding random noise, or blurring to simulate different lighting conditions to reduce the likelihood of this happening. This helps the ML model be more robust and recognize handwritten digits in various formats. That is why, specifically for the MNIST dataset, a handful of enhanced versions of MNIST alternatives like EMNIST or NOT MNIST were introduced.


Examples of image augmentation include:

Flipping the image Random cropping Reducing the contrast or brightness Shearing the image Image augmentation is also widely applied when the amount of training data is not enough.


What is image enhancement?

Image enhancement encompasses a wide range of techniques aimed at improving the quality and visual appeal of an image. It involves the manipulation of either the entire image or specific attributes within it. For example, adjusting the contrast and brightness can bring out details and make the image more vibrant, while fine-tuning the range of the RGB color pattern can enhance the color accuracy and overall tonality.The nature and extent of these image modifications can vary significantly depending on the specific goal of the enhancement. Image enhancement is, to some extent, a subjective process as the desired outcome heavily relies on the intended purpose of the image. For instance, in the realm of medical imaging, the objective may be to highlight and emphasize specific structures or anomalies within the body. In contrast, when it comes to headshot photography, the primary focus might be on accentuating facial features while fine-tuning brightness and contrast levels to achieve a pleasing aesthetic.Traditionally, image enhancement has been accomplished through the use of specialized image editing software like Photoshop and Affinity. However, advancements in generative AI technology have opened up new possibilities. Innovative products such as Midjourney, LAION's Stable Diffusion model, DreamStudio, and Adobe's Firefly have emerged, enabling image enhancement through the assistance of large language models. While these AI-based approaches are gaining traction, the predominant methods of image enhancement currently involve programmatically manipulating images using libraries and tools such as Pillow, OpenCV, and machine learning algorithms.Moreover, image enhancement technology has proven to be invaluable in the field of medical imaging. By applying enhancement techniques to radiographs, CT scans, and other medical images, the aim is to facilitate the interpretation of results by medical professionals. These enhancements can improve the visibility of critical structures, highlight anomalies or abnormalities, and enhance the overall diagnostic accuracy, ultimately aiding in better patient care.


What is the difference between image enhancement and image augmentation?

Yes, DataChad supports chatting with many files at the same time. You can chat with PDFs, text documents, Word documents or CSV files all at the same time.While image enhancement is mainly focused on changing the visual appeal of a single image, image augmentation aims to generate additional image files to expose a deep learning model to more aspects of the same image.


What are some examples of image enhancement?

Histogram Equalization: Histogram equalization is a method of image enhancement that stretches the contrast of an image. In code, this is most often achieved using the OpenCV or Pillow libraries with the following functions: cv2.equalizeHist() or PIL.ImageOps.equalize(). Gamma Correction: Gamma correction is used to brighten or darken an image. In code, this can be accomplished easily using the OpenCV or Pillow libraries with the following function calls: cv2.LUT() or PIL.ImageEnhance.Brightness(). Contrast Stretching: Contrast stretching is a technique that stretches the contrast of an image in order to increase its visibility. In code, contrast stretching is most often done using the OpenCV or Pillow libraries with the following functions: cv2.resize() or PIL.ImageOps.autocontrast(). Sharpening: Sharpening is a technique used to enhance certain features in an image. In code, this can be achieved using the OpenCV or Pillow libraries with the following functions: cv2.filter2D() or PIL.ImageEnhance.Sharpness(). Noise Reduction: Noise reduction is an image enhancement technique used to reduce the amount of noise in an image. In code, this can be accomplished using the OpenCV or Pillow libraries with the following functions: cv2.blur() or PIL.ImageFilter.GaussianBlur(). Image Dehazing: is a common image enhancement technique used to reduce the amount of haze in an image. In code, this can be accomplished using the OpenCV or Pillow libraries with the following functions: cv2.bilateralFilter() or PIL.ImageTweak.Dehaze().


Joint image filtering

Image filtering with guidance signals is known as joint or guided filtering. Joint image filtering involves transferring critical structural details from the guidance image to the target image. It aims at enhancing the target image while not passing along structures that did not exist in the image. It is typically applied in various computer vision tasks such as: Structure-texture separation. Joint upsampling Cross-modality noise reduction Depth map enhancement The Joint Image Filtering with Deep Convolutional Networks method proposes a CNN network with three-sub networks and skip connections. The first two sub-networks are responsible for extracting informative features from the guidance and input image. These features are concatenated and used as features for the third sub-network. Skip connections are used so that the network learns to predict residuals between the target image and the ground truth output. The parameters of the three sub-networks are updated simultaneously during training.


Changing colors/recoloring

Colorization is commonly applied in changing black and white images to colored images. This can be achieved using photo editing tools or neural networks. It is applicable in photography, where one wants to color old images and videos.Coloring images manually is a cumbersome and expensive process that takes long. Fortunately, image recoloring can be automated using deep learning. This is done using a Convolutional Neural Network that takes in an input image and predicts the colors to fill the image.Image recoloring can also be applied to colored images where you want to change the colors in an image. Recoloring images can be used to achieve various artistic goals. Recoloring is also applied in photography to ensure an image looks professional. The example below shows recoloring a flower to purple, green, red, and orange using Generative Adversarial Networks (GANs).


Image shifting

Image shifting involves shifting the pixels in an image to new locations. For example, you can shift an image vertically or horizontally. When shifting an image, all the image's pixels are moved to a new position while the dimensions of the image are maintained.


Image denoising

Denoising convolutional neural networks (DnCNNs) is a technique for reducing image noise using feedforward neural networks. It uses residual learning and batch normalization to speed up training and improve denoising performance. The method tackles denoising with unknown levels of Gaussian noise –– blind Gaussian denoising. The first image below shows a noisy image, while the second one shows the result generated by DnCNNs.


Advanced examples of image enhancement

Image quality enhancement techniques such as Super Resolution and Low Light Image Enhancement are used to recover as much detail as possible from low-resolution images. Accurate Image Super-Resolution Using Very Deep Convolutional Networks and the Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement (Zero-DCE) utilize both traditional methods and deep learning approaches such as AI upscaling and Zero-DCE respectively to reduce noise, increase contrast and brightness, allowing them to be used in applications such as security cameras and object detection. LIME: A Method for Low-Light Image Enhancement also solves this problem by creating an illumination map from the R, G, and B channels and does not require paired or unpaired data, making it less prone to overfitting and to be used in various applications such as security cameras, object detection, and scene understanding.


Image demosaicing

Color image demosaicing involves interpolating missing color values using nearby pixels in raw images captured by cameras. The images are typically captured with 12 bits per pixel, and Color Filter Arrays (CFA) are placed in front of the camera's image sensor array. CFAs allow different wavelengths of light to pass through the camera, and one common type is the Bayer pattern, which captures information at red, green, and blue wavelengths.The output of the Bayer pattern is a mosaiced image, which needs to be converted to a standard RGB image through the process of color image demosaicing. During image capture, a camera runs a demosaicing algorithm to generate the full RGB image. Bilinear interpolation is a commonly used method in demosaicing, where missing pixels are assumed to be similar to adjacent ones, and the missing values are replaced by the average of neighboring pixels. However, demosaicing itself does not remove noise from the image, so additional denoising may be required, either through a separate denoising algorithm or by using an algorithm that combines denoising and demosaicing.Traditional demosaicking methods include bilinear interpolation and Malvar interpolation. In addition to these conventional techniques, machine learning methods can also be applied to demosaicing. Demosaicing is similar to super resolution in that both involve filling in missing pixel information, so super-resolution techniques can be used in demosaicing as well. Some machine learning methods suitable for image demosaicing are k-Nearest Neighbors, Support Vector Regression, and Super-Resolution Convolutional Neural Network.Nearest neighbor Bayer image demosaicing can be performed in the following steps: Split the bayer-pattern image into three color channels. Fill in missing pixel information for each color channel using bilinear interpolation. Merge the interpolation result with the detailed information predicted by the k-Nearest Neighbors model.


Watermark Removal

Adding watermarks to images serves various purposes, such as preventing unauthorized copying or reproduction. However, watermarks can sometimes obscure details in the image. In such cases, it becomes necessary to remove the watermark to reveal those hidden details.


Defading

Defading is a process used to recover information from faint or degraded images. For instance, the ink on old documents can wear off over time, making the text difficult to read. Fading can also occur due to overexposure during the digitization of documents. Defading methods aim to restore a more visible and legible version of the image by mitigating the effects of fading. UnblurUnblur is the technique employed to remove blur from an image, enhancing its clarity. Blur in images can be caused by subject movement during capture, camera focus issues, or camera shake. Various methods can be used to deblur images, including prior-based methods that estimate blur kernels and parameters, as well as learning-based methods that utilize deep learning algorithms to learn deblurring models.


Contrast-Limited Adaptive Histogram Equalization (CLAHE)

CLAHE is a method for contrast enhancement that works on small regions or tiles of an image. It improves local contrast, which is particularly useful in areas such as microscopic imaging, X-ray imaging, medical image analysis, and high-definition television (HDTV). CLAHE utilizes parameters like the number of tiles and clip limit to control noise amplification and enhance visibility in various applications.


Histogram Matching

Histogram matching, also known as histogram specification, is an image processing technique used to generate an image based on a specified histogram. It involves obtaining histograms for both a reference image and a target image, computing cumulative distribution functions, and applying a histogram matching function to each pixel of the reference image. Histogram matching is employed to normalize images taken under different conditions and can be performed using libraries like skimage, OpenCV, or deep learning techniques.


Wiener Filter

Wiener filter is an image processing technique that removes additive noise and inversely deblurs an image. It outperforms the inverse filter by accounting for both degradation functions and the statistical properties of noise, resulting in a better restoration of the original image. Wiener filtering minimizes mean squared error in noise smoothing and inverse filtering, providing a linear estimation of the original image.


Median Filter

Median filter is a non-linear image processing technique used to remove noise while preserving edges. It replaces each pixel in an image with the median value of neighboring pixels within a window or neighborhood. The median filter is particularly effective in removing salt and pepper noise, producing better results than the mean filter due to its robustness and preservation of edge information.


Binarization

Binarization is a process that separates an image into foreground and background, resulting in a binary image with black-and-white regions. It is commonly used to eliminate degradations like noise and is crucial for tasks such as optical character recognition and document layout analysis. Binarization can be performed using different approaches, including global thresholding and local thresholding based on neighboring pixels.


Linear Contrast Enhancement

Linear contrast enhancement, also known as contrast stretching, is a method for stretching the pixel values in an image to a new distribution. It can be achieved through different approaches such as Min-Max Linear Contrast Stretch, Percentage Linear Contrast Stretch, or Piecewise Linear Contrast Stretch. Linear contrast enhancement improves the visibility of attributes in an image by expanding the range of contrast.


Using Deeplake + Pillow for image enhancement

Finding suitable datasets for image enhancement tasks can be challenging. However, several datasets are available for specific purposes, such as the LOL (Low-Light) dataset for low-light image enhancement, ARID dataset for activity recognition in videos, Bickley diary dataset containing images of a diary, and NoisyOffice dataset with noisy images of an office. These datasets can be used for training machine learning models or evaluating image enhancement algorithms.The LOL (Low-Light) dataset can be easily downloaded using deeplake from Activeloop.1import deeplake 2ds = deeplake.load('hub://activeloop/lowlight-train') 3CopyNext, let's look at a summary of the data.ds.summary()Activeloop also makes it easy to visualize the data using the visualize function.ds.visualize()With some datasets at hand and having learned various image enhancement techniques, let's now look at how to implement them in Python using Pillow and OpenCV.First, copy the dataset from Activeloop to gain write access. This is done using the deepcopy function, which expects: The data source The dataset destination, that is, your personal Activeloop account Your API token The function signature is shown below:1ds = deeplake.deepcopy('hub://activeloop/lowlight-train', dest='hub://mwitiderrick/lowlight-data', token='YOUR_TOKEN')` 2 3CopyNext, load your version of the low-light dataset. Passing your API token and read_only=False gives you write access to the dataset.1 2ds = deeplake.load('hub://mwitiderrick/lowlight-data', token='YOUR_TOKEN',read_only=False)` 3Copy


Make images sharper with Pillow

Let's start by looking at how to make all the images sharper—checkout to a different branch. You will then commit the changes you make to this branch and later merge it to main if you want to.1from PIL import Image, ImageEnhance, ImageFilter 2import numpy as np 3from PIL import Image 4import numpy as np 5 6ds.checkout('sharp_image', create = True) 7CopyNext, loop through the images, make them sharp using Pillow, and commit the changes.1with ds: 2 for i, sample in enumerate(ds): 3 image = ds['highlight_images'][i].numpy() 4 im = Image.fromarray(image).convert('RGB') 5 enhancer = ImageEnhance.Sharpness(im) 6 ds.highlight_images[i] = np.asarray(enhancer.enhance(10.0)) 7 sharpen_commit_id = ds.commit('Sharpen the images') 8Copy Head back to your Activeloop account to see the new images. Click Version and switch to the sharp_image branch.