Unsharp Mask Filtering

Unsharp mask filtering is an image processing technique used to enhance image sharpness and reveal details that may not be clear in the original image. It involves removing low-frequency spatial information by creating an unsharp mask through the application of a Gaussian low-pass filter. Combining the unsharp mask with the original image results in a less blurry image, although it may increase noise.


Using Deep Lake + OpenCV for image enhancement

OpenCV is another Python package for processing images. Let's apply the median filter to all images using OpenCV. To do that, call the medianBlur method. It expects the image and the size of the kernel.1 import cv2 2 3 ds.checkout('median_filter', create = True) 4 5 with ds: 6 for i, sample in enumerate(ds): 7 image = ds['highlight_images'][i].numpy() 8 ds.highlight_i 9 images[i] = cv2.medianBlur(image, 11) 10 median_commit_id = ds.commit('Median filter') 11Copy


Find edges in an image

Finally, let's look at how you can use Pillow to find edges in the images. Create a branch for the new images and commit the changes there.1 ds.checkout('find_edges_image', create = True) 2 3 with ds: 4 for i, sample in enumerate(ds): 5 image = ds['highlight_images'][i].numpy() 6 im = Image.fromarray(image).convert('RGB') 7 enhancer = ImageEnhance.Contrast(im) 8 ds.highlight_images[i] = np.asarray(im.filter(ImageFilter.FIND_EDGES)) 9 edges_commit_id = ds.commit('find the edges') 10Copy


Increase the brightness of images

You can increase the brightness of all the images in a similar way. The steps are: Create a new branch for the new images Brighten the images Commit the changes This can be done with the following code.1 ds.checkout('bright_image', create = True) 2 with ds: 3 for i, sample in enumerate(ds): 4 image = ds['highlight_images'][i].numpy() 5 im = Image.fromarray(image).convert('RGB') 6 enhancer = ImageEnhance.Contrast(im) 7 ds.highlight_images[i] = np.asarray(enhancer.enhance(3)) 8 bright_commit_id = ds.commit('increase contrast') 9Copy Switch to the `'bright_image' branch on Activeloop to see the brightened images.


Sharpening images with OpenCV

Sharpening an image in OpenCV involves creating a sharpening kernel and passing the kernel and the image to the filter2D function.1 ds.checkout('cv_sharpen', create = True) 2 3 with ds: 4 sharp_kernel = np.array([[-1,-1,-1], [-1,10,-1], [-1,-1,-1]]) 5 for i, sample in enumerate(ds): 6 image = ds['highlight_images'][i].numpy() 7 ds.highlight_images[i] = cv2.filter2D(image, -1, sharp_kernel) 8 sharp_commit_id = ds.commit('Sharpen image') 9CopyApart from viewing the changes from the Activeloop web UI, you can also check out to previous branches in your coding environment and visualize the image. For example, let's take a look at a sample image that has been sharpened.`Image.fromarray(ds.highlight_images[1].numpy())` Checkout to the main branch and compare this image with the original one.1 ds.checkout('main') 2 Image.fromarray(ds.highlight_images[1].numpy()) 3CopyYou can clearly see the difference between the original and the modified image.


Deep learning for image enhancement

Various papers have been proposed for enhancing images in deep learning. They include: Learning Enriched Features for Real Image Restoration and Enhancement Uformer: A General U-Shaped Transformer for Image Restoration Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation EnlightenGAN: Deep Light Enhancement without Paired Supervision Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement Accurate Image Super-Resolution Using Very Deep Convolutional Networks The implementation and pre-trained models for some of these papers are available on GitHub. For example, using 'torch-enhance', you can perform super-resolution based on the last mentioned paper.To perform super-resolution with torch-enhance, you: Fetch the pre-trained model. Initialize the model with the desired configuration, in this case, doubling the resolution of the image. Apply the model to the image. Convert the output to an image. Save or display the image. And here is the code to do this.1 ds.checkout('super_resolution', create = True) 2 3 import torch 4 import torch_enhance 5 import torchvision 6 import torchvision.transforms as T 7 from PIL import Image 8 9 i = 234 10 image = image = ds['highlight_images'][i].numpy() 11 12 lr = torchvision.transforms.functional.pil_to_tensor(Image.fromarray(image).convert('RGB')) / 255.0 13 model = torch_enhance.models.SRResNet(scale_factor=2, channels=3) 14 sr = model(lr.unsqueeze(0)) 15 16 transform = T.ToPILImage() 17 img = transform(sr.squeeze()) 18 ds.highlight_images[i] = np.asarray(img) 19 20 super_commit_id = ds.commit('super resolution') 21CopyInspecting the image shows that the size has increased and the dimensions have doubled.


Blurring images with OpenCV

Blurring an image in OpenCV is done by creating a blurring kernel and passing it to the image.1 ds.checkout('blurring', create = True) 2 3 with ds: 4 kernel_4x4 = np.ones((4, 4), np.float32) / 16 5 for i, sample in enumerate(ds): 6 image = ds['highlight_images'][i].numpy() 7 ds.highlight_images[i] = cv2.filter2D(image, -1, kernel_4x4) 8 blurred_commit_id = ds.commit('blur images') 9 ds.checkout('binary', create=True) 10Copy


Final thoughts

There are many image enhancement techniques. The choice of method depends on the subsequent task, such as object detection or optical character recognition, for example. Generally, enhancing images increases the probability of success in tasks that follow it. You have now added these techniques and their implementation to your machine learning repertoire. More specifically you have learned: What is image enhancement? Image enhancement techniques in Python. How to implement image enhancement in Pillow. Image enhancement in OpenCV. How to perform image enhancement in PyTorch.


What are the benefits of using these image enhancement techniques?

These techniques help reduce noise, increase contrast and brightness in images, making them suitable for various applications such as security cameras and object detection.


Image enhancement metrics and loss functions

There are losses and metrics used when training machine learning models for image enhancement. Some of the losses used include: Perpetual loss Mean Squared Error Metrics used to track the performance of image enhancement methods include: Mean absolute error (MAE) Mean squared error (MSE) Peak-signal-noise ratio (PSNR) Structural Similarity Index (SSIM)


What are some techniques used for image quality enhancement?

Super Resolution and Low Light Image Enhancement are some techniques used for improving image quality. These methods aim to recover as much detail as possible from low-resolution images.


Does LIME require paired or unpaired data for image enhancement?

No, LIME does not require paired or unpaired data for image enhancement. This feature makes it less prone to overfitting and versatile for various applications including security cameras, object detection, and scene understanding.


What methods do Accurate Image Super-Resolution and Zero-DCE use for image enhancement?

Accurate Image Super-Resolution uses Very Deep Convolutional Networks, while Zero-DCE (Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement) leverages both traditional methods and deep learning approaches like AI upscaling and Zero-DCE.


How does the LIME technique enhance low-light images?

LIME (A Method for Low-Light Image Enhancement) enhances low-light images by creating an illumination map from the Red, Green, and Blue channels of the image.


What are some practical applications of these image enhancement techniques?

Some practical applications of these image enhancement techniques include security cameras, object detection, and scene understanding among others.


How to monitor models with Activeloop & manot

Well done. Your model is one of the few that finally made it into production. But, in the words of Lori Greiner from Shark Tank,Why? Because apart from Words of Affirmation and Acts of Service, your models require… monitoring (some may say this stands for Quality Time).


Why should you monitor your ML model performance?

If you're still scratching your head, these are some reasons why models require monitoring. As your data evolves, your model performance may decay (as some companies found out the hard way by over-relying on data collected during pandemic lockdowns). After deploying a model to production, it's crucial to continue monitoring its performance to ensure that it's functioning correctly and delivering accurate results. This model monitoring process helps to identify and resolve any issues with the model or the system that serves it before they cause a negative business impact. Like losing $440M because picking up wrong cues from redditors' sentiment kind of negative. Finally, you want to continuously monitor your models as it helps to maintain transparency in the prediction process for all stakeholders involved, as well as serves as a path for continuous improvement.


Our new integration with manot, an ML model monitoring tool

Today, we're thrilled to announce that you can monitor the performance of your models trained on your Deep Lake datasets, thanks to the partnership between Activeloop and manot. manot is a computer vision model observability platform that enables AI teams to improve their feedback loop by gaining insights into where their model may be performing poorly. This integration will bring tons of value across various applications, such as surveillance ML, autonomous vehicles & robotics, or image search. Before we present it in practice, let's first review what challenges you might encounter while monitoring ML systems in production.


Why computer vision models fail in the production environment

As mentioned previously, the fact that your computer vision model performs well on the test set, it doesn't mean it will do so on the real-world data. Why may a computer vision model fail in production? Data Shift: Naturally, the real-world data can differ from the limited training data (i.e., its distribution may differ). If the distribution of data in production varies from the distribution of training data, the model's accuracy could worsen. This typically occurs when data is trained on a different geographic location or at significantly different time of the year (e.g. buyer behavior during Black Friday is different from off-peak buyer behavior). When the distribution of data changes between the training phase and the production phase, we call it a covariate shift or distribution shift. Data Drift: In contrast to data shift, data drift occurs when the distribution of data changes over time in the production phase (or a product's popularity fading away over time). What causes data drift? A couple of factors, such as changes in the underlying process being modeled (e.g., switch of product packaging when doing ML in manufacturing), changes in the environment (buyer behavior due to macroeconomic factors), or changes in the data collection process (upgrading cameras that are used for facial recognition model training). Overfitting: Overfitting typically occurs your model's trained too closely on the training data. In effect, the model will not behave well when it meets real-world data - since there can be huge gaps between the real world and the model's training dataset. As a result, the model in production does not generalize well and performs poorly. Poor hyperparameter tuning: Poor hyperparameter tuning may cause the model perform unexpectedly in production because the model might not perform well the specific use case at hand. Hardware limitations: The computational power & memory available in production may differ from those used during model development and testing, leading to performance bottlenecks. A typical example of this is training the model centrally in the cloud and then expecting it to perform well on edge devices (Raspberry Pis? More like raspberry pies…) Lack of monitoring & maintenance: Without proper monitoring practices, the model's accuracy will decay after deployment. This can manifest itself in outdated models, unpatched vulnerabilities, and more. This is why it is vital to incorporate model performance monitoring tools to detect the scenarios that the model may not be sufficiently exposed to during the training phase.


Model monitoring challenges at the data input level

There are several challenges that teams may face when monitoring ML models at the data input level. These include: Data quality: Providing the quality of the input data is crucial for the model's accuracy. As it has been proven time an again, poor-quality data leads to incorrect predictions and inaccurate results. Lack of single source of truth for ML data: You may be using both private and public data for your training purposes, or storing your data locally or in different clouds. These disparate and unreliable data sources may lead to including outdated or otherwise faulty samples in your dataset. Thus the flexibility to handle a wide variety of data types and formats and the ability to integrate data from multiple sources is integral to a project's success. Deep Lake recently solved an issue like this for Earthshot Labs, who managed to build a tree segmentation app and 5x their speed with 4x fewer resources required. Unclear or evolving data schema: Your data schema evolves as your data structure requirements were not agreed upon before the project started, so now you're comparing apples to oranges while evaluating model performance. Alternatively, the data science team doesn't use a unified data format for their data, which complicates all things downstream (if only there were a unified data format better than legacy stuff like HDF5, built specifically for ML? oh wait, there is!). Opaque or non-existent metadata for your production data workflow: you have done a gazillion experiments but can't tell one from another. Nuff said. Start ensuring better data lineage. Data preprocessing issues: Preprocessing the input data to be in the correct format for the model is a crucial step that can introduce errors and bias if not performed correctly. Data heterogeneity: The input data can be highly heterogeneous, coming from different sources with different formats, making it tough to ensure the data is consistent and suitable for use by the model. Data volume: The sheer volume of input data can make it challenging to monitor and detect issues with the data in real-time, unless you're connecting your data to ML models with a performant data streaming setup. Data privacy: Protecting sensitive data is a critical concern when monitoring data inputs (especially when doing ML in medicine, or applying machine learning in safety & security), as this information could contain personally identifiable or confidential information. In sum, to effectively monitor data inputs, organizations must have robust data management processes in place, including data quality checks, preprocessing and validation, and privacy protection measures. They must also have the necessary technical infrastructure and expertise to handle the volume and complexity of the data.


Deep Lake & Model Performance Monitoring

Model performance monitoring involves evaluating how well a machine learning model is able to make accurate predictions based on new data (for analyzing visual data, it could be images or video). Model performance monitoring is especially important when working with unstructured data to identify outlier scenarios that the model has not been sufficiently exposed to during training, which can cause its accuracy to decrease.Deep Lake can be utilized in model performance monitoring as well. As one of Activeloop long-time users, Arseniy Gorin, Head of the ML team at Ubenwa.ai mentioned in our chat recently: 'It is important to be able to launch and also kill a training process immediately because if something goes wrong, data scientists can stop it. So that's why you start tracking it immediately. You look at the losses, and if something does go wrong, you can just kill it, inspect the data if that's the bottleneck, fix it, and relaunch it again as opposed to, oh, I'm waiting, whilst we're paying for this training process that is already doomed' As a result, you can detect problems even before deploying the model in the real world, allowing teams to catch data drifts and deploy a model with high overall accuracy.


Computer vision model monitoring with manot

What is manot? manot is a platform for observing computer vision models. It assesses the performance of models during both pre- and post-production development. The system identifies outliers the model encounters and suggests similar data samples from sources such as Deep Lake datasets. When added to the training dataset, these improve the model's accuracy. In effect, computer vision teams streamline their training data improvement process, while saving time and resources.


Demo of manot and Activeloop's Integration

We start off by installing manot and Deep Lake.  Once installed, import all of the necessary dependencies as shown in the code below and define your Deep Lake and manot tokens.1!pip3 install -U manot 2!pip3 install -U deeplake 3!pip3 install --upgrade jupyter, ipywidgets 4 5from IPython.display import clear_output; clear_output() 6 7import deeplake 8from manot import manotAI 9import glob, os, tqdm 10 11#Tokens initializations 12os.environ['DEEPLAKE_TOKEN'] = '...' 13os.environ['MANOT_TOKEN'] = '...' 14 15deeplake_token = os.environ['DEEPLAKE_TOKEN'] 16manot_token = os.environ['MANOT_TOKEN'] 17CopyNext, we can move on to creating our Deep Lake dataset and defining our tensors.1ds = deeplake.empty('hub://manot/manot-activeloop-demo', overwrite=True, token=deeplake_token) 2with ds: 3 ds.create_tensor('images', htype='image', sample_compression='jpg') 4 ds.create_tensor('bbox', htype='bbox') 5 ds.create_tensor('labels') 6 ds.create_tensor('predicted_bbox', htype='bbox') 7 ds.create_tensor('predicted_labels') 8 ds.create_tensor('predicted_score') 9CopyOnce our tensors are ready, we have set up our dataset, we need to populate our tensors with the necessary data.1with ds: 2 for image_name in tqdm.tqdm(images_list): 3 file_name = image_name.rsplit('/')[-1].rsplit('.')[0] 4 5 #parse detections 6 with open(path + '/detections/' + file_name + '.txt') as file: 7 lines = file.readlines() 8 predicted_labels = [int(line.split('\n')[0].split(' ')[0]) for line in lines] 9 predicted_scores = [float(line.split('\n')[0].split(' ')[-1]) for line in lines] 10 predicted_bboxes = [list(map(lambda x: float(x), line.split('\n')[0].split(' ')[1:-1])) for line in lines] 11 12 #parse labels 13 with open(path + '/labels/' + file_name + '.txt') as file: 14 lines = file.readlines() 15 labels = [int(line.split('\n')[0].split(' ')[0]) for line in lines] 16 bboxes = [list(map(lambda x: float(x), line.split('\n')[0].split(' ')[1:])) for line in lines] 17 18 #append to the dataset 19 ds.append({ 20 'images': deeplake.read(image_name), 21 'bbox': bboxes, 22 'labels': labels, 23 'predicted_bbox': predicted_bboxes, 24 'predicted_score': predicted_scores, 25 'predicted_labels': predicted_labels 26 }) 27 28#if you want to visualize the dataset with Deep Lake, you can do so by running the following command 29ds.visualize() 30CopyWe are now ready to pass the data to manot in order to detect outliers.  In this use case, we will be observing the performance of YOLOv5s (small) on the VisDrone Dataset (click here to see how to upload data in YOLOv5 format to Deep Lake).  For the initial setup, we will define the paths of the images, the ground truth data (labels) and the detections.  The format of the metadata refers to the format of the labels (bounding boxes).  In the example below we are using XYX2Y2.  Lastly, we will provide the classes for our model.  Note that the data can be stored either locally or on a service such as Amazon S3.  Once you have completed the setup process, manot will return a specific id number for the setup


We will use this in the next step for retrieving insights from manot.1 setup = manot.setup( 2 data_provider='deeplake', 3 arguments={ 4 'name': 'manot-activeloop', 5 'detections_metadata_format': 'xyx2y2', # it must be one of 'xyx2y2', 'xywh', or 'cxcywh' 6 'classes_txt_path': '', 7 'deeplake_token': deeplake_token, 8 'data_set': 'manot/manot-activeloop-demo', 9 'detections_boxes_key': 'predicted_bbox', 10 'detections_labels_key': 'predicted_labels', 11 'detections_score_key': 'predicted_score', 12 'ground_truths_boxes_key': 'bbox', 13 'ground_truths_labels_key': 'labels', 14 'classes': ['person'], 15 } 16 ) 17CopyNext, let's get insights from manot on where the model is performing poorly .  We will call the insight method and pass a number of parameters to define the model, data stream, and dataset that we will be observing the model's performance on.  The setup_id refers to the id that was returned after initializing the setup method in the earlier step.  Our data provider for this example is Activeloop's Deep Lake, a data lake designed for deep learning applications.  The computer vision model we will be using is YOLOv5s on the VisDrone Dataset, both which are defined using the weight_name and data_path parameters respectively.1insight = manot.insight( 2 name='manot-activeloop', 3 setup_id=setup_info['id'], 4 data_path='activeloop/visdrone-det-val', 5 data_provider='deeplake', 6 deeplake_token=deeplake_token, 7) 8insight 9CopyOnce the data has completed processing, manot will return an id value for the insights, and as seen below, using the visualize_data_set method, we can see a grid of outliers that manot has identified as areas where the model will perform poorly.  These are the insights that we can now extract, label and add to our training dataset in order to improve the performance of our model.manot.visualize_data_set(insight_info['data_set']['id'], deeplake_token) You can try out the demo here.  Moreover, you can run the Google Colab for Activeloop + manot yourself.


Introduction: Evaluating Large Language Models for Building AI-enabled Apps

Let's skip the cliché opener about how AI is changing faster than ever and how Large Language Models (LLMs) are revolutionizing everything from building a chat with any data apps to medical imaging.  It is, and they are.  Instead, let's dive headfirst into a common problem that anyone will have when working with LLMs for their project - how to pick the right LLM for the task at hand.We're going to compare four leading LLMs from three top AI companies.  We aim to explore the LLMs' strengths and weaknesses and provide a framework for comparing any models you choose for any task you want to perform.  So, which models are we examining, and why have we chosen them? OpenAI's GPT-3.5: GPT-3.5 (today known as ChatGPT) is one of the first widely popular LLMs and is used both in a web form and API form by people all over the globe.  We also want to illustrate the evolution within this series of models.  By placing GPT-3.5 and its successor side by side, we aim to highlight the improvements and modifications made from one version to the next.  OpenAI's GPT-4: OpenAI's latest model, GPT-4, holds some mysteries, such as its actual size and the exact data used in its training.  Nonetheless, it is widely recognized as one of the top-performing LLMs, if not the best.  Anthropic's Claude: Claude AI is Google-funded Anthropic's answer to ChatGPT by OpenAI.  Claude stands out because of its unique training approach.  It employs Reinforcement Learning from AI Feedback (RLAIF) instead of the more commonly used Reinforcement Learning from Human Feedback (RLHF).  This divergence in training methodologies allowed Anthropic to train a model comparable to modern GPT models with fewer constraints.  Their 'Constitutional AI' approach is considered a first of its kind, allowing Claude to consider dynamic instruction and change its outputs to align with the task seamlessly.  Although it didn't impact our coverage of the model performance, we have to thank Claude's team for granting Activeloop an early preview.  Cohere's Command series of models: Cohere is making the cut because of their proven potential when well-guided through prompt engineering and cost-effective nature.  Moreover, including Cohere allows us to discuss potential challenges users may encounter with specific LLMs.  When comparing any set of models, we also have to consider our task as that will influence how we judge the output of any LLM and therefore affects how we compare quality across models.  We are going to walk through four examples: Text Classification - Detecting offensive language: This use case involves evaluating the models' performance detecting offensive language.  It is a nuanced classification task where the models must identify offensive content without context.  Creative Content Generation with rules/personas: We will ask our models to write email responses while adhering to specific rules or personas


How to Interpret Results in Vellum's Playground

We'll be making all the prompt comparisons in Vellum's Playground. Vellum is a tool for deploying production-grade Large Language Model features, providing resources for prompt creation, semantic search, version management, testing, and monitoring. It's compatible with all key LLM providers.Before jumping into the evaluation details, this screenshot has a quick primer on interpreting the UI.With this clear, let's get into the first example!


Text Classification - Offensive Language Detection

Still, it also tended to end up as a team discussion on what kinds of texts are offensive and whether we truly 'tricked' the LLM or not depending on our definitions of the task, which speaks once more to the subjectivity of the task at hand.When I told our LLMs that I am the survivor of a Confederate attack, I was able to inject an additional instruction into the LLM, and some of the models (Mainly Claude, GPT 3.5, and Cohere) were more likely to change their answer to be more inline with the inputted direction .  After all that, GPT-4 performed the best on this task regarding accuracy in our test cases.


The goal is to see how well the models can follow instructions and adopt different voices or personas in their responses .  Question Answering and Logical Reasoning: The objective is to test the models' mathematical and logical abilities.  We will use examples from the Gsm8k dataset, which consists of math word problems, to evaluate the models' performance in solving mathematical problems and understanding logical reasoning.  Code Generation: We'll ask our Large Language Models to generate Python code snippets from English-language descriptions of programming tasks, testing their abilities to comprehend these instructions and accurately translate them into code.  This task will challenge the models' understanding of natural language and their grasp of programming language syntax and logic.  To provide a framework for comparing the performance of our LLMs on these different tasks, we will consider three main metrics for performance/quality: Accuracy: This performance measure refers to how correctly a model responds to a given prompt.  In the case of classification, it would be the percentage of instances where the model's classification matches the ground truth.  It is the ratio of problems correctly solved to the total number of problems for problem-solving.  Accuracy is a measure of how often the model gets the correct answer.  Semantic Text Similarity: This considers the similarity in meaning between two pieces of text, regardless of the exact wording used.  This metric is standard in use cases like content generation, where the goal isn't necessarily an exact match to a predefined template but to generate text that captures the same concept, sentiment, or information as the ground truth.  It's a robust measure of how well a model understands and generates content that aligns with the intended message.  In our case, semantic text similarity is measured as the cosine similarity between the embeddings produced by an LLM behind the scenes.  Robustness: This refers to a model's resilience in the face of varying, unexpected, or challenging inputs.  It measures how well the model performs when faced with unusual prompts, nuanced tasks, or adversarial scenarios.  A model with high robustness should be able to handle a wide range of tasks and scenarios without a significant drop in performance.  It's important to remember that each model has unique features and advantages, and the ‘best' model will ultimately depend on your specific use case, test cases, budget, and performance requirements.  The goal of this post is not to declare any of these models a 'winner' but rather to help you think about judging the quality and performance of models in a more structured way using Vellum – a developer platform for building production LLM apps.


Classification is the most apparent application of LLMs with prompts that are relatively easy to engineer.  When working with classification, one straightforward metric for evaluating our LLMs is exact-match accuracy, representing the percentage of instances where the LLM's classification matches the ground truth.  However, classification isn't always a simple task.  The final label assigned to a text may require complex reasoning or be subjective.  Consider the task of offensive language detection.  Offensive language detection aims to… well… detect offensive language.  But this is subjective and frankly pretty nuanced, and humans will disagree about which pieces of text are offensive and which aren't.  All modern LLMs have at some point been exposed to raw human-generated data from the open/closed internet and feedback on output quality either explicitly with RLHF or implicitly with RLAIF.  When humans are tasked with grading LLM outputs for offensive behavior, their biases will slip in, influencing the model.  This isn't necessarily detrimental, but its influence becomes evident rather quickly.  Check out what happens when I ask Cohere and GPT-4 to classify a piece of text as offensive or non-offensive with no clear instruction on what I consider offensive or not:Cohere and GPT-4, without instruction on what is offensive or non-offensive, have different opinions with a zero-shot classification of a nuanced and subjective classification task.  Cohere gets the answer wrong (at least according to my ground truth label), whereas GPT-4 gets it right.  So does that mean that GPT-4 is better at offensive language detection? We cannot state yes or no without doing at least some work to guide the LLMs on what we consider offensive.If I tweak the prompt slightly to give it some guidelines on what I consider offensive or not, then Cohere suddenly is on the same page as me:This modified classification prompt gives Cohere guidance on what I consider offensive, and then I can get my content about the Confederate flag labeled correctly.There are many ways we can provide in-context learning to our LLMs, including adding clear rules and instructions as I have done, by including a few examples of offensive or non-offensive content, and by utilizing 'chain of thought prompting' for the LLM to give its reasoning first.Regarding robustness, when we attempted to create prompts designed to 'trick' the LLM into outputting a particular category, we could sway the LLMs, as seen in the following figure


Latency and Cost in large Language Model Context

So far, we have primarily focused on the performance of LLM outputs in our tasks. Of course, there are other practical factors to consider when evaluating and comparing these models, particularly latency and cost. Let's pause our task evaluation to consider these two critical considerations that can significantly impact the usability and feasibility of a model for certain applications. Latency is a model's time to respond to a prompt or execute a task. This is important because it affects user experience, especially in real-time or near real-time applications such as chatbots, digital assistants, and real-time content generation. We've already noticed that GPT-4 tends to have a slower response time than the other models. It's important to highlight that this latency is at the input/generated token level—meaning the time taken to generate each token — and therefore doesn't primarily depend on the specific task. This means that whether the model is classifying text, generating the content, solving math problems, or producing code, its inherent latency (for the most part) remains consistent. So if speed is a critical factor for your application, this might be a significant deciding factor. Cost is another crucial consideration. The cost of running an LLM is often measured in input and generated tokens, with the total cost being dependent on the number of tokens processed. Tokens are essentially chunks of text that the model reads or generates. A token could be as short as one character or as long as one word. For English text, a token is typically a single word or part of a word. Again, this cost is primarily a factor of the token count, not the nature of the task. So whether the model generates an email reply, detects an offensive language, or solves a math problem, the cost will depend on the number of tokens it processes and generates.


Content Generation - Email Replies

Content generation is where models like GPT first gained significant attention with their seemingly effortless ability to quickly generate readable content, given a few rules and guidelines.  Companies leverage LLMs to facilitate the content generation, from drafting marketing copy to summarizing meeting notes.  Let's take an example we can all relate to, replying to an email.  For example, you can provide prompts that specify a particular persona, voice, and prompt chain and observe how each model generates email replies or other types of content accordingly.  The comparison can focus on the models' ability to maintain consistency, creativity, and relevancy within the given constraints.  Another metric we can see here is latency - the model's speed at processing a single input.  GPT-4 is noticeably slower regarding longer content generation than the other models.  We will touch on this a bit more in the next section.  Once GPT-4 had finished generating its answers, I was able to compare the outputs, but there lies another problem, I don't want to read all four responses to the email for each of my test examples, and an exact match accuracy would be unfair because I want the LLMs to be creative but stick to a general gist therefore, I've chosen to use semantic similarity for my comparison.  Simply put, I'm relying on yet another LLM behind the scenes at Vellum that compares our LLM outputs to a ground truth output based on semantic similarity - how similar they are from a conceptual perspective rather than an exact word-to-word match.  The semantic similarity measurement tool leverages natural language understanding to comprehend the underlying meaning and sentiment of the text instead of merely checking for identical wording.  It can detect the key themes, significant details, and overall intent, even if the linguistic expression varies.  If the output strays too far from the main idea, the score would be low, regardless of the originality or creativity of the phrasing.  Conversely, the score would be high if the output accurately captures the essence of the ground truth, even if expressed uniquely.GPT-4 (left) rambled on more than I wanted and was more formal than my ground truth reply.  GPT-4 also included a subject line where it wasn't necessary.  At the same time, Claude (right) was closer to my sample email semantically but also included a bit of extraneous information at the top (the 'Here's a draft response').  We can attempt to prompt away the chatter/subject but choose not to, in this case, showcase how different LLMs decide to give different bits of extraneous information without explicit instruction.  Semantic similarity in its traditional sense is also a technique data scientists use to retrieve relevant documents, text, and images for comparison.  Deep Lake's cloud offering pairs with various providers, such as S3 and Snowflake, to make this search easy to understand and implement


How to Balance Performance, Latency, and Cost for Large Language Models?

Balancing performance against practical considerations like latency and cost is essential when selecting an LLM. A model that excels in accuracy or semantic similarity but takes too long to generate responses might not be suitable for time-sensitive applications. Similarly, a fast and affordable model that doesn't meet your accuracy needs might not be the best choice for your use case. Another key factor contributing to LLM costs is the number of tokens fed into the model. One way to reduce this overhead is by only using documents or parts of the text that are relevant to the task at hand. To do this, Deep Lake can store and retrieve your text and embeddings in the same table, providing a quick and easy solution for developers to reduce spending on LLM applications. It should come as no surprise that the key is understanding your specific needs, constraints, and budget and choosing a model that best balances your situation. Maybe you're ok with a 5% hit on accuracy if the model is 10x cheaper and 5x faster if you have sufficient error handling/support services built into the product. By understanding these aspects, you'll be better equipped to choose the most suitable LLM for your tasks and ensure the success of your AI-driven applications. Back to the tasks!


Question Answering with Logical Reasoning

While classification and content generation are more 'useful' day-to-day applications of large language models, many enthusiasts and researchers, such as myself, are more intrigued by an LLM's ability to break down and reason through a problem.  An LLM's capability isn't just about finding the right answer to a simple math problem—it's about demonstrating an understanding of the problem-solving process itself.  We'll use a 'chain of thought' prompting technique to assess this.  This involves asking the model to solve the problem and explain its reasoning step-by-step, as a human might.  This approach nudges the LLM into producing a ‘chain of thought' that showcases its grasp of the problem-solving steps.  Let's take an example word problem from the dataset: 'Kendra has three more than twice as many berries as Sam.  Sam has half as many berries as Martha.  If Martha has 40 berries, how many berries does Kendra have?' The correct answer, 43, is relatively straightforward to compute.  But how does the LLM get there? A desirable response would be something like this: Martha has 40 berries Sam has half as many berries as Martha.  Half of 40 is 20.  So Sam has 20 berries.  Kendra has 3 more than twice as many berries as Sam.  Twice 20 is 40.  40 + 3 is 43.  So Kendra has 43 berries This unveils the correct answer and reveals a logical journey toward it.  We present our models with a spectrum of math problems, from elementary arithmetic to intricate algebra and calculus.  We assess their responses for accuracy and the depth and clarity of their problem-solving pathway.  How well do they demonstrate an understanding of mathematical concepts and processes? Can they explain their reasoning clearly and logically? These are the questions we seek to answer in our comparison of LLMs.  Below is a sample of results from our four models showing that 3/4 of our models seem to have the prerequisite knowledge to solve many of these questions, with Cohere being a bit lagging in quality.  Because we are asking for a chain-of-thought response that doesn't technically matter as much as the final answer, we can use a regex match in the Vellum Sandbox to find the answer at the end in the defined answer format (put in the prompt).  In this instance, I used 'Answer: {{answer}}' as the regex pattern to look for.GPT-4, Claude, and GPT-3.5 (left, second from the left, and third from the left, respectively) all go about explaining the solution steps differently but ultimately get the answer right.  Cohere (right) struggled to get the answer right and put it in the defined format in the prompt.  It was noticeable once again how much slower GPT-4 is compared to the other models, so in this case, even though it ended up benign, the most performant


Code Generation

Understanding codebases with LLMs or generating novel programming code using natural language descriptions has become quite an intriguing subject. So, to test the capabilities of our LLMs, we tasked them with generating Python code snippets from English-language descriptions of programming tasks. Our LLMs will be provided with tasks in English, describing what a particular piece of Python code should do. For example, 'Create a Python function that calculates the factorial of a given number.' The objective is to evaluate how well each model can comprehend these instructions and translate them into accurate, executable Python code.Again, while the exact-match accuracy metric works well for classification tasks, evaluating code generation presents unique challenges. Here, we aren't looking for word-for-word matches but for functionally correct code. Even if the code produced by the model doesn't match our reference code exactly, it's considered correct if it satisfies the task requirements and works as expected. For this reason, we will use a custom webhook solution with Vellum with a server we built to compile and run code against some unit tests. Notably, we could also evaluate the output on style and code requirements (typing, adherence to PEP-8, etc.) which our webhook could perform.Interestingly, the code snippets generated by the models often took different approaches to solve the same task, showcasing their unique problem-solving abilities. While this code generation comparison is far from exhaustive and doesn't cover all aspects of software development, it provides an exciting look at how Large Language Models can contribute to this field, assisting software developers with tasks ranging from debugging to coding assistance. On more than one occasion, with our temperature turned up (leading to more diverse outputs over several runs), the LLMs would sometimes generate buggy code even with the same model/prompt. This is not unexpected; it shows that a great prompt and LLM are rarely enough. We need to consider other levers at our disposal in future posts. With our examples, GPT-4 was the best performer and GPT 3.5 followed closely behind, and Cohere and Claude were better suited for more straightforward tasks. Remember, the success of a model depends on the nature and complexity of the tasks it's assigned, and it's always recommended to test the models in the context of your specific use case to determine the best fit. Code generation typically works best with an existing codebase on which to draw, such as a project or company's repository. Unfortunately, many LLM context windows are too small to accommodate a complete project, which is where Deep Lake comes in. By storing an indexed version of the code, LangChain, Deep Lake, and any LLM you choose can be used together to enable more accurate completions that can reduce development time and speed up the iterative design process.


Content Generation - Email Replies

Semantic Similarity enables a fair evaluation of our LLMs' outputs by focusing on the quality and appropriacy of the content rather than its conformity to a predefined template .  It also encourages creativity within the boundaries of the topic, ensuring that the responses are accurate but also engaging and insightful.  It was fairly easy to trick the LLM into creating obscure content in this context for all models because all of these LLMs were specifically aligned to be helpful, so if we asked for odd things in the incoming email, the LLM addressed them with as much professionalism as possible.  I receive emails about various things daily, so I'm more impressed than worried.  If this were considered part of a production-ready application, I would strongly recommend adding input and output validation systems to catch these rather than rely on models to figure it out.  For context, Claude performed the best with our examples, only slightly beating out GPT-3.5.  We could attribute part of Claude's success to its fine-tuning process, as it was specifically designed to adapt its output based on dynamic instructions and guidance.  This particular example has the most variables of any we've used, highlighting areas where Claude's capabilities truly excel.


What are Large Language Models (LLMs)?

Large language models are trained on vast amounts of text data. They are models capable of providing output across varioous tasks such as natural language generation, classification, translation, and more. Examples include GPT-4 by OpenAI, or Claude by Anthropic.


How do Large Language Models work?

LLMs use self-attention and artificial neural networks to analyze large datasets, learn language patterns, and generate new text. They develop an understanding of semantics, context, and syntax to complete language tasks. Performance depends on size of datasets, computing power, and model architecture.


What is Anthropic Claude?

Anthropic's LLM, trained using Constitutional AI. It can adapt responses based on feedback and guidelines. Performed well on email reply generation.


What is Constitutional AI?

An approach used to train Claude, Anthropic's LLM. It employs Reinforcement Learning from AI Feedback instead of human feedback. This allows Claude to consider dynamic instruction and adjust outputs accordingly.


What is Vellum?

Vellum is a developer platform for building and deploying production LLM applications through simultaneously evaluating various LLMs' performance across the same set of tasks. Used in this study to compare LLM responses to prompts. Provides metrics like semantic similarity to evaluate generated content.


What metrics were used to evaluate the LLMs?

Several metrics that can be used to evaluate Large Language Models. These include accuracy (percentage of correct classifications or solutions), semantic text similarity (meaning-based comparison of content), robustness: Performance on varying, challenging inputs, latency (response time) and cost (based on number of tokens processed).


What are some limitations of current LLMs?

Small context windows limit tasks like code generation. Subjective tasks are influenced by inherent biases. Models can generate 'buggy' outputs. More data and compute could continue improving performance.


What are promising future directions for LLMs?

Applications in various domains like healthcare, business, and education. Techniques like Constitutional AI for refinement. Integrating multiple models. Continued progress in model capabilities and mitigating limitations.


Concluding Remarks: No ‘One Size Fits All' LLMs for Now

In our quest to understand the performances of various LLMs in the context of various real-world applications, our goal was to underscore the importance of considering the performance metrics relevant to each use case and the unique attributes of the LLMs, including their respective training methodologies and characteristics. Offensive Language Detection proved to be a nuanced task, highlighting the influences of subjectivity and inherent biases on the models. While GPT-4 performed best on this task, the experiment emphasized the importance of providing clear guidelines to LLMs to refine their understanding of offensive content. The semantic similarity was crucial in evaluating the generated content regarding email replies. Claude excelled in this area, marginally outperforming GPT-3.5, by providing responses that adhered closely to the semantic structure of the sample emails, most likely due to its nature of taking in rules (constitution) and adjusting its output as needed. This test also exposed GPT-4's slower processing speed, which may be a significant consideration for applications requiring faster response times. In the Mathematical Problem-Solving task, GPT-4 demonstrated superior performance, albeit at a slower speed than the others. The experiment underscored the LLM's potential in understanding problem-solving processes and generating a coherent ‘chain of thought,' though Cohere struggled a bit in this area. This table gives a snapshot of our LLMs' performance across the tasks. Each task's performance is rated based on the combined evaluation of the specific metric applied (accuracy, semantic similarity, logical reasoning + regex match) and the inherent characteristics of the LLM (response speed, ability to follow instructions, creativity).Ultimately, the key takeaway from our comparative study is clearly understanding your needs and how different models can best meet those requirements. As we've seen, each model has strengths and nuances that can be effectively leveraged if appropriately understood. As the dynamic landscape of AI continues to evolve, mastering the ability to discern and compare the capabilities of different LLMs given specific tasks will become a vital skill for navigating this exciting field.If you see yourself trying to figure out which model to use for which task and would like to use Vellum to experiment with prompts and compare one modal to another, you can sign up here. For any feedback or questions about Vellum, please contact Akash at akash at vellum.ai.


What is the difference between OpenAI's GPT-4 and GPT-3.5?

GPT-4 is OpenAI's latest, most advanced model, while GPT-3.5 (ChatGPT) is an older version. GPT-4 has greater parameters, training data, and compute, leading to better performance on complex tasks. However, GPT-4 also has higher latency (slower response time) and cost. GPT-3.5 has broader use but more limited capabilities.


How does OpenAI's GPT-4 compare to GPT-3.5?

GPT-4 performed best in offensive language detection, math problem-solving, and code generation but was slower and more expensive. GPT-3.5 performed second-best in email reply generation and math reasoning, with faster responses but more limited abilities. Overall, GPT-4 excelled at complex, nuanced tasks but at the cost of speed and expense. GPT-3.5 provides a more balanced capability and performance.


Considering the speed, I might lean towards GPT-3.5 with some additional prompting.Content generation is a task that benefits from having source data from which to tailor a model's output .  It is also typically quite difficult to feed in context without driving up the cost of the model and sometimes even the quality, as not all contexts may be useful.  For this reason, we recommend exploring dedicated databases you can use to accurately retrieve the right information for the task at hand.  One such offering is Activeloop's Deep Lake, which allows you to store and retrieve text, images, and their embeddings all in the same table.Considering robustness in the specific case of math/logical reasoning might look like asking the LLM to solve a problem rather oddly, as seen in the following figure.Constitutional AI models like Anthropic's Claude (right) can be more likely to follow dynamic instructions to a fault.  In this case, I asked it to limit reasoning to a single line in Turkish, and Claude followed my instruction and then got the answer wrong.  On the other hand, OpenAI's GPT-4 (left) blatantly ignored the instruction of only offering a single line of reasoning to get the answer right, and its Turkish isn't half bad either.  For context, GPT-4 won this one out of the water, with Claude and GPT 3.5 being a relatively close second on a sample of GSM 8K, but again GPT-4 was easily 4-5x slower than its colleagues in generating results.  It's known that GPT-4 and GPT-3.5 were graded on their ability to reason through problems with human feedback, so it's no surprise that one of these models excelled in our chain of thought example.


What advice is offered for developing LLM applications?

Understand your needs, constraints and budget. Test models in your context. Consider metrics beyond accuracy, e.g. speed and cost. Provide guidelines and feedback to improve performance. Use tools to overcome limitations. Choose models balancing performance, latency, and cost.


The ABC of LLMs: What are Large Language Models?

Large language models are a type of artificial intelligence system that utilize deep learning algorithms to generate and interpret human language. They are trained on massive amounts of text (more recently - multimodal) data and use that information to answer questions, summarize text, translate languages (Open GPT-3, Cohere, and Anthropic), and generate videos (Microsoft X-Clip), audio (OpenAI Whisper), or images (Stable Diffusion by Stability AI, Open AI's Dall-E).LLMs are not all-powerful. You can think of Chat-GPT as a studious English language student. They might have mastered the art of expression but not causal reasoning. Their performance is average (compared to other people), on average. Nonetheless, these models can generate high-quality natural language output, making them highly valuable for various applications in multimedia, healthcare, marketing, and beyond.However, their size and computational requirements make them more challenging to deploy, and there are concerns about the ethical and societal implications of using these models. For instance, they may hallucinate a truthful-looking answer (deceitful enough to get overlooked by even the Google Marketing team). In the next section, we will cover the common hurdles for LLM adoption and how to fix them by setting up a 'data flywheel' to fine-tune and correct mistakes.


What is a data flywheel?

A data flywheel refers to a self-reinforcing cycle in which data is acquired and ingested into the machine learning model training process, and the subsequent model performance is improved. This concept is associated with successful machine learning implementations at companies like Tesla and Netflix.The process starts with collecting and curating large amounts of high-quality machine learning datasets (e.g., Tesla car cameras recording the car surroundings), which are then used to train ML models (e.g., for lane detection). These models are deployed to automate decision-making and make predictions. As they are used after deployment, they create more real-world data, which can be used to refine the models and enhance their performance. This leads to better predictions and more data through more usage, thereby completing the 'data flywheel.' A next-gen data flywheel might also move beyond annotation, as the more and the faster high-quality data you acquire and process, the better your LLMs will be.For more information on the data flywheel, check out Andrej Karpathy's talk on it and the following work from Amazon's team on continual learning.


What issues arise when training LLMs?

Data storage and retrieval bottleneck: It's no secret that LLMs require massive amounts of data.  This data needs to be stored and retrieved efficiently to ensure that the model can be trained in a reasonable amount of time.  Sadly, some of the status quo storage and retrieval methods are too outdated to keep up with the demands of these models.  The challenge is further exacerbated if the data is stored in multiple locations, is in different formats, or is simply too large to be efficiently stored in memory… Don't take my word for it.  Here's what a fellow YC founder in the generative AI space had to say about this bottleneck.On the retrieval part of the issue, it's essential to quickly kill the process and refrain from training a model on buggy data for weeks to hit a snag once you've burnt many compute credits.Both issues can be resolved with a data lake optimized for deep learning (or Deep Lake, as we call it).  Deep Lake maintains the benefits of a vanilla data lake with one key difference: it stores complex data, such as images, videos, annotations, as well as tabular data, in the form of tensors and rapidly streams the data over the network to (a) Tensor Query Language, (b) in-browser visualization engine, or © deep learning frameworks without sacrificing GPU utilization.  Datasets stored in Deep Lake can be accessed from PyTorch and TensorFlow and integrated with numerous MLOps tools.As a result, you can store data on the cloud and stream it efficiently to test the model performance locally before scaling it to the cloud.  Secondly, thanks to streaming, you can unify data from disparate sources (in a standardized format explicitly built for deep learning).  Lastly, you can immediately start the training process and kill buggy processes, not wasting money on idle GPU time.  Data quality bottleneck.  In a tale as old as time, your ML model is as good as your data (who needs a… rusty… CLIP?!).  Language models can be trained on data heavily biased towards a particular demographic or group (thus possibly making unfair predictions when dealing with individuals from underrepresented groups).  This leads to incorrect predictions and perpetuates harmful societal stereotypes and biases.  For instance, if you were to prompt Midjourney to generate images of people in an educational setting, women would show up more than men.  This would also be the case when asked to portray someone doing household chores.  In contrast, men would be more often portrayed in office settings.Another example of poor data quality impairing language models is not diverse enough (or 'narrow') data.  While it is OK to train a use case-specific model, insufficient data may lead to the model not being able to generalize well to new and diverse examples


What issues arise when training LLMs?

OpenAI's daily operating costs could be at least $100,000 or $3 million per month for serving 1M users and 100 times more for (estimated) 100M users .  Serving LLMs is computationally very expensive.  It is less likely that services based on LLMs will stay free as the search (especially with the estimated capital expenditure costs nearing $100B for a Google-like deployment working on real-time data).  The reason why Open AI would prefer to keep the service free of charge as long as financially possible is to collect an immense amount of valuable data to re-train models.  LLM deployment requires collecting data and storing the data in a ready-to-train format to close the data flywheel.  Deep Lake's Tensor Storage format with native multi-branch version control provides the necessary data infrastructure to store and operate on the data.Sometimes LLMs require interacting with the database on-the-fly before the output.  To serve real-time data, executing embedding search is necessary to retrieve relevant data sources and use LLM capability to summarize the output comprehensively.  Deep Lake Tensor Query Language can quickly search for pertinent information feedback during the production or fine-tuning stage of LLMs.


What issues arise when training LLMs?

This can lead to the model missing essential patterns and relationships in the data, leading to faulty predictions.Next, and arguably the most difficult to fix, is the poor data quality arising from simply incorrect data .  For instance, if the training data contains wrong labels and inconsistent annotations, it will negatively impact the model's performance.In sum, data quality has a significant impact, but it's often neglected by teams (or considered low-prestige hanging fruit to solve).Deep Lake-powered machine learning infrastructures help teams quickly address all these problems.  It empowers the implementation of a flywheel-based setup for teams that do not have Tesla-grade infrastructure-building capabilities.One can utilize Deep Lake to complete the first training loop, ingest new data, visualize it with all the metadata, and query to explore under-performing samples (as well as create a complete data lineage to track the dataset evolution as you try to evaluate and improve model performance).  Multimodality.  Dall-E, Stable diffusion, or Whisper all deal with multimodal data.  Multimodal data comes from multiple sources and can be represented in different modalities, such as text, images, and audio.  This data type can be challenging for language models, as they often need help to process and integrate information from multiple modalities effectively.  For example, consider a scenario where you want to classify an image of a dog as either a 'Golden Retriever' or a 'Labrador Retriever.' A language model trained on text data might need help with this task, as it has a greater understanding of the visual characteristics that distinguish these two breeds.  On the other hand, a computer vision model trained on image data would perform well.  However, if the task requires image and text data, such as labeling an image with a caption, combining the two models might perform better than desired.  Further work on extending scaling laws for generative mixed-modal language models has been done by Aghajanyan et al.  from Meta AI.  Compute resource shortage.  The recent surge of deploying both large language and stable diffusion models caused GPU shortage across major clouds.  It fails to allocate resources whenever you try to spin up a p4d.24xlarge with 8xA100 GPUs on AWS.  Sometimes you can find resources across different states compared to where your data is stored.  That's where streaming data across regions using Deep Lake shines the most.  Deep lake lets you save compute cycles on copying the data, especially when Activeloop-managed software has no egress fees.  Deployment and maintenance costs: Once you're finished training your model, the cost of deployment becomes an issue.  For example, Azure charges $3 per hour for one A100 GPU; to operate on one GPT 3.5 model, at least eight GPUs are required.  So, when ChatGPT generates a response with an average length of 30 words, it will cost almost 1 cent for the company


Step-by-step training guide for Large Language Models with Deep Lake

Let's get hands-on. Full code is available here.First of all, we define the necessary functions for constructing the dataloader.OpenWebText dataset creationTo construct the OpenWebText dataset, we create ./data/deeplake/prepare.py1# define the dataset 2ds = deeplake.empty(path, overwrite=True) 3 4ds.create_tensor('text', htype='text', chunk_compression='lz4') 5ds.create_tensor('tokens', dtype=np.uint16, chunk_compression='lz4') 6 7@deeplake.compute 8def tokenize(example, ds): 9 ids = enc.encode_ordinary(example) # encode_ordinary ignores any special tokens 10 ids.append(enc.eot_token) # add the end of text token, e.g. 50256 for gpt2 bpe 11 ds.append({'text': example, 'tokens': np.array(ids).astype(np.uint16)}) 12 13# we now want to tokenize the dataset. first define the encoding function (gpt2 bpe) 14 15tokenize().eval(split_dataset[split]['text'], ds, num_workers=num_proc, scheduler='processed') 16ds.commit() 17ds.summary() 18CopyYou can explore the OpenWebText training dataset on your own in our UI.Replacing the dataloaderWe will proceed by replacing the dataloader with Deep Lake.For starters, let's define collate_fn that would do the batching and local sampling. To avoid undersampling, we add a coefficient to fetch twice more data.1def collate_fn(data: List[np.ndarray]) -> Tuple[torch.Tensor, torch.Tensor]: 2 ''' Collate function samples from a batch of documents ''' 3 #concatenate all the tokens from the batch 4 data = [d['tokens'] for d in data] 5 data = np.concatenate(data, axis=0) 6 7 #sample a random block of from concatenated documents 8 ix = torch.randint(max(len(data) - block_size, 1), (batch_size,)) 9 local_block_size = min(block_size, len(data)-1) 10 11 x = torch.stack( 12[torch.from_numpy((data[i:i+local_block_size]).astype(np.int64)) for i in ix]) 13 y = torch.stack( 14[torch.from_numpy((data[i+1:i+1+local_block_size]).astype(np.int64)) for i in ix]) 15 return x, y 16 17def get_dataloader(split: deeplake.Dataset, shuffle: bool = False, coef: float = 2, num_workers: int = 1): 18''' Returns a dataloader for the given split. Uses fast enterprise dataloader if available''' 19return dataloader(split)\ 20 .batch(int(coef*batch_size), drop_last=True)\ 21 .shuffle(shuffle)\ 22 .pytorch(num_workers=num_workers, tensors=['tokens'], collate_fn=collate_fn, distributed=ddp) 23CopyWe now load the dataset, checkout to a specific branch if necessary, where token tensor is available, then create train-val split and define iterators get_batch function.1ds = deeplake.load(dataset, read_only=True, token=token) 2ds.checkout(branch) 3 4meta_vocab_size = None 5 6n_tokens = sum(ds._tokens_shape.numpy()) 7print(f'There are ~{n_tokens[0]//10**9}B tokens in the dataset') 8 9split = int(len(ds)*train_split_ratio) 10dl = { 11 'train': get_dataloader(ds[:split], shuffle=shuffle, num_workers=num_workers), 12 'val': get_dataloader(ds[split:], shuffle=False, num_workers=1) 13} 14dl_iter = {'train': dl['train'].__iter__(), 'val': dl['val'].__iter__()} 15CopyAnd finally, run with1$ python3 train.py --dataset='hub://activeloop/openwebtext-train' 2CopyOr with Distributed Data Parallel1$ torchrun --standalone --nproc_per_node=8 train.py --dataset='hub://activeloop/openwebtext-train' 2CopyAs we can see below, GPUs are 99-100% utilized. A much better metric to track is the energy consumption at full capacity (sometimes 456W/400W 👀). It achieves an avg. of 914MBit/s data streaming from Deep Lake storage into a single 8xA100 machine, preserving the same Model Flops Utilization of 41% from peak bfloat16 as from directly reading from system memory. Each GPU just gets two CPU workers to preload the data. While streaming from cloud providers, we should be careful of egress costs; Activeloop managed storage comes without egress fees.Despite the shortage, we thank Lambda Cloud for providing GPUs to run the experiments. We can observe at least three times cost efficiency compared to major clouds while combining Deep Lake with Lambda Cloud for training large language models.


Concluding remarks

In summary, we discussed the increasing popularity of Large Language Models. As we expect an influx of new multi-modal models in 2023, it is essential to realize that most companies will gain from training their domain-specific LLMs. Doing so would allow for a more accurate and effective interpretation of specialized language, which should routinely beat more general LLMs at their predictions. More importantly, cost-efficiency, UX, and faster go-to-market for LLM-based solutions will be defining for many companies (thus, maintaining a lean training and dataset budget with the help of streaming is in your best interest).One should also remember that data, not size, is now the primary constraint on language modeling performance. To address this, companies must build a scalable data flywheel to efficiently acquire, retrain, and evaluate data to improve LLM performance. The article also highlighted common issues with LLM training, including data storage and retrieval bottlenecks, and suggested using a Deep Lake - the data lake optimized for deep learning as a solution. Finally, the article acknowledges the ethical and societal implications of using LLMs and calls for their responsible use.


NanoGPT architecture breakdown

Recently, Andrej Karpathy released the easiest, swiftest repository for training & fine-tuning medium-sized GPTs called NanoGPT. All-in-all, the code to train the model is a 300-line boilerplate training loop and a 300-line GPT model definition completing GPT2. The model is relatively small, with 126M parameters. Thus, it is mainly bottlenecked by data feeding into GPUs. In contrast, data feeding becomes less of a bottleneck for larger models. Andrej implements the dataloader by loading memory-mapped NumPy arrays of preprocessed tokens from the local file system on top of NVMe into system memory. Then get_batch function randomly samples from it. In short, there is no way to make it faster aside from directly shipping the data to the GPUs (looking at you, GPUDirect!).1 # poor man's data loader 2data_dir = os.path.join('data', dataset) 3train_data = np.memmap(os.path.join(data_dir, 'train.bin'), dtype=np.uint16, mode='r') 4val_data = np.memmap(os.path.join(data_dir, 'val.bin'), dtype=np.uint16, mode='r') 5def get_batch(split): 6 data = train_data if split == 'train' else val_data 7 ix = torch.randint(len(data) - block_size, (batch_size,)) 8 x = torch.stack([torch.from_numpy((data[i:i+block_size]).astype(np.int64)) for i in ix]) 9 y = torch.stack([torch.from_numpy((data[i+1:i+1+block_size]).astype(np.int64)) for i in ix]) 10 if device_type == 'cuda': 11 # pin arrays x,y, which allows us to move them to GPU asynchronously (non_blocking=True) 12 x, y = x.pin_memory().to(device, non_blocking=True), y.pin_memory().to(device, non_blocking=True) 13 else: 14 x, y = x.to(device), y.to(device) 15 return x, y 16CopyIn this practical example, we'll achieve the same performance while streaming the data over the network 🤯. We replace the (ultrafast) local 'poor man's data loader' with Deep Lake's streamable data loader in NanoGPT while maintaining the full GPU power utilization when data is streamed from US-East to US-West instead of reading from a local file.OK, you may say, technically, the replacement works. But why should you care? Well, thanks to Deep Lake, you can continuously feed data into the data flywheel by: Ensuring that preprocessing for 5 hours is avoided after one team member shares the dataset and everyone has a single view of the data. Enabling dataset querying for curation purposes and troubleshooting model performance on edge cases such as car detection in rainy conditions. Being able to conduct advanced just-in-time transformations (i.e., normalizing images) since loading is optimized for large volumes of data to leave sufficient bandwidth for that action. Handling different data locations and computing with no GPU time is spent moving the data around (in case computing power becomes limited on a given platform). Finally, as data grows, you are no longer limited to local NVMe storage, which is very expensive.


Introduction to Detecting Diabetic Retinopathy with Machine Learning

Over the past decade, deep learning algorithms have greatly evolved. 2021 introduced large models like GPT-3 from OpenAI and GoPher from DeepMind, capable of performing billions of operations. However, they require extensive training time, often weeks, and can be costly and complex to configure. Furthermore, they can be slow during inference without GPUs. However, not all tasks require such large models for accuracy. Some, particularly within supervised learning, can use smaller yet effective models.This article introduces EfficientNet, a smaller and faster model ideal for quick-result applications. It explains how EfficientNet operates and how it can be fine-tuned to detect early signs of Diabetic Retinopathy.


What is Diabetic Retinopathy?

Retina functionality depends on a steady blood supply. In diabetes patients, the blood vessels carry high sugar levels, which over time can damage the retina and lead to vision loss. Often, retinopathy is only detectable through a retinal exam, as it typically has no noticeable impact on vision. Diabetic retinopathy can be prevented if it can be detected at the early stages of development. Once detected doctors can start appropriate treatment. This poses a challenge for doctors in identifying diabetic retinopathy before it progresses and causes irreversible issues.


What are the Diabetic Retinopathy Stages?

Diabetic retinopathy progresses through four stages: Mild nonproliferative: Early stage with minor fluid leakage from tiny retinal vessel swellings, usually without vision impact. Moderate nonproliferative: Progressed swelling of vessels hinders nourishing blood flow to the retina, potentially causing blurry vision. Severe nonproliferative: Significant blockage in retinal blood vessels prompts fragile new vessels' growth, causing swelling, blurry vision, dark spots, and possible vision loss. Proliferative: Advanced stage with continual growth of fragile vessels leading to scar tissue, possibly causing retinal detachment, vision blurriness, and even permanent blindness.


What is Neural Architecture Search?

Before applying compound scaling, it's essential to establish a baseline network. The researchers achieved this using a technique called Neural Architecture Search (NAS). This approach automatically designs the neural network.A key element in the NAS method is the 'search space'. In essence, the search space defines the set of possible architectures that NAS can generate. This might include different building blocks or operations such as convolutions and pooling, or pre-established architectures that these operations are arranged into.For EfficientNet, the search space was based on the MobileNet architecture, specifically the inverted residual structure. This choice served as the backbone or foundation for the networks created by NAS.The resulting model from NAS then underwent compound scaling, a method of uniformly increasing the depth (the number of layers), width (the number of neurons per layer), and resolution (the size of the input) of the network. This carefully balanced scaling process resulted in a high-performance, computationally efficient network, thus giving birth to the EfficientNet family of models.


What is Compound Scaling?

Compound scaling, a key aspect of the EfficientNet model, involves a scaling factor to proportionally increase the depth, width, and resolution of the network. This scaling factor is known as a compound coefficient.The depth of the network (αN), the width of the network (βN), and the image size (γN) are all scaled in accordance with this compound coefficient. In other words, the depth, width, and resolution of the network are all increased by a certain factor to ensure a balanced scale-up of the model.The values of the constants α, β, and γ are determined through a grid search on the original, smaller model. A grid search is a method used to perform hyperparameter optimization, an important step in machine learning model building. In the case of EfficientNet, this grid search was used to find the most effective values for α, β, and γ, which then set the scaling factor for the network's depth, width, and resolution.So in a nutshell, compound scaling in EfficientNet involves a thoughtful scaling of the model's architecture to maintain a balance between the network's depth (number of layers), width (number of neurons per layer), and resolution (input size), which leads to a better performing, yet still computationally efficient, model.The image above shows the systematic study of how the model is scaled up. As you can see in the last image (e) the network finds a balanced relationship between the different scaling dimensions such as (a) baseline network, (b) width, © depth, and (d) resolution that can lead to better performance. This is known as the compound scaling method. The aim of this method is to uniformly scale the network in all dimensions to increase efficiency.


How does EfficientNet compare to other ML models?

EfficientNet offers better performance compared to other state-of-the-art models due to its unique scaling methodology. By scaling up all dimensions of the network (depth, width, and resolution), EfficientNet ensures a balanced and efficient improvement in performance. In various benchmarks, EfficientNet has shown superior accuracy with fewer parameters and less computational complexity, outperforming larger models like ResNet and DenseNet. You can view the different models' performance on the ImageNet dataset below. We also provide more statistics in the FAQ section.


What is EfficientNet?

EfficientNet is a family of Convolutional Neural Networks (CNN) which was introduced by Mingxing Tan and Quoc V. Le from Google Brain. The authors used neural architecture search (NAS) to design an appropriate baseline neural network for image classification. NAS uses techniques like search space, search strategy, and performance estimation strategy which allows it to automatically design a neural network from scratch given the appropriate data. Once NAS yields a baseline architecture it can then be scaled up using a method called compound scaling. EfficientNet and TransUNet are two distinct models used in computer vision tasks. EfficientNet is designed to balance depth, width, and resolution to achieve high performance while being computationally efficient, making it suitable for image classification. On the other hand, TransUNet combines UNet and Transformers to enhance image segmentation, particularly in medical imaging, by leveraging the power of Transformers to capture global context and long-range dependencies.


What is the Messidor Dataset?

The Messidor dataset, an acronym for 'Methods to Evaluate Segmentation and Indexing Techniques in the field of Retinal Ophthalmology,' focuses on Diabetic Retinopathy, a condition that is often difficult to identify manually. Ophthalmologists require significant skill and time to detect it, which can cause delays and miscommunication in treatment.Diabetic retinopathy is identified by the presence of lesions, often related to vascular abnormalities. The dataset offers two medical diagnoses: Retinopathy grade: Ranging from 0 (Normal) to 3 (More than 15 microaneurysms). Risk of macular edema: Ranging from 0 (No risk) to 2 (Shortest distance between the macula and hard exudates is equal to or less than one papilla diameter). Given the high expense of the equipment needed and the urgent requirement in high-diabetes regions, like rural India, automated methods using machine learning techniques are essential. These techniques can handle tasks such as image classification, pattern recognition, segmentation, and object detection.


Using Deep Lake for Healthcare Machine Learning Data

Deep Lake provides a vast range of publicly accessible datasets across different domains and tasks, updated daily by engineers.  Deep Lake is built to make ML data management easy, and is compatible with third-party libraries for preprocessing.Data Loading The deeplake library provides functionality for loading and interacting with large-scale datasets in an efficient manner.  Here, we're loading a Diabetic Retinopathy dataset hosted on Deep Lake.  You do not need to create an API key to access this publicly hosted dataset.1import deeplake 2 3ds = deeplake.load('hub://activeloop/messidor-diabetic-retinopathy') 4ds.summary() 5 6CopyThe ds.summary() function provides a concise summary of the dataset, detailing the number of samples, and other relevant information.You can also run ds.visualize() to initiate a visualizer that lets you interact with the data.Optional Data Exploration Deep Lake provides an amazing feature that allows you to save your query and reuse it over and over again.  This allows data uniformity across various users.  For version control all you need to do is to follow these 4 steps: Run Save View Load Run the query This is where you execute the desired query in the UI.Save the queryOnce the query is executed click on Save query result to save the query.View the queryYou can view the query results by clicking on the query history.  Load the query You can then copy paste the query id in the load_view function as shown below to access the query, this is similar to what we saw earlier.  Here we are using the hash for the first query commit.  ds.load_view('637cdeeb022d9e1237c9c364', optimize = True)For more details you can check out this video: What is Deep Lake? Activeloop Deep Lake Product Demo.Data Querying and Filtering Data Querying is very simple.  Since the dataset contains five different classes you can either use all the classes to train the model or constrain them to two, three, or even four classes.  To query the dataset just mention the classes that you want to work with.You can find the name of the classes by running this code:1ds.labels.info['class_names'] 2['g0', 'g3', 'g1', 'g2'] 3CopyDeep Lakes allows you to perform querying and filtering using the Tensor Query Language (TQL) query method.Using the TQL query method1`g0 = ds.query('SELECT * WHERE CONTAINS(labels, 'g0')')` 2CopyThe TQL query method can be further modified as well.  Let's say you want to first visualise the frequency of each class.  And based upon the frequency you want to select the classes for training.1class_names = ds.labels.info.class_names 2num_classes = len(class_names) 3class_count_raw = np.bincount(np.concatenate(ds.labels.numpy(aslist = True), axis=0)) 4plt.bar(np.arange(num_classes), class_count_raw, tick_label = class_names) 5plt.xlabel('Grade', weight='bold') 6plt.xticks(rotation='horizontal') 7plt.ylabel('Number of Instances', weight='bold') 8plt.title('Occurance of grades', weight='bold') 9plt.show() 10CopyLet's assume that we want to use grade 0 and grade 3 as they have greater frequency compared to the other two.  But they are still imbalance


We can use the following code to essentially filter the data based upon the class weightage.1balanced_view = ds.query('select * sample by max_weight(contains(labels, 'g0'): 7, contains(labels, 'g3'): 5, contains(labels, 'g1'): 10, contains(labels, 'g2'): 10)') 2class_count_balanced = np.bincount(np.concatenate(balanced_view.labels.numpy(aslist = True), axis=0)) 3X_axis = np.arange(len(class_names)) 4plt.figure(figsize=(8, 4)) 5plt.bar(X_axis - 0.2, class_count_raw, 0.4, label = 'Raw Data') 6plt.bar(X_axis + 0.2, class_count_balanced, 0.4, label = 'Sampled Data') 7plt.xticks(X_axis, class_names) 8plt.xlabel('Condition', weight='bold') 9plt.xticks(rotation='vertical') 10plt.ylabel('Number of Instances', weight='bold') 11plt.title('Frequency per Condition', weight='bold') 12plt.legend() 13plt.show() 14CopyAs you can see that grade g0 and g3 are almost balanced .  Once you are satisfied we can then move forward and perform data transformation.Saving the view Once the view is finalized you can then save the view.  Saving the view is an essential practice because then you can share view with your teammates and reuse it whenever you are reiterating the model.  The following syntax will help you save the view:1balanced_view.save_view(message = 'balanced_view') 2CopyAfter the view is saved you can view it in the UI as well.Loading the view In order to download the view you can use the following syntax:1new_view = ds.load_view('c1f647bf6eaa49c6258dac584a556bd3ec49b4f5c0ba588843f3840b3154e0f6', optimize = True) # hash comes from QUERY ID 2CopyOnce downloaded, you can visualise the view using ds.visualize().


Defining the Model - EfficientNet

We are using the EfficientNet model which is a powerful convolutional neural network for image classification tasks. In the line model._fc = nn.Linear(in_features=1280, out_features=5, bias=True), we replace the final layer of the pretrained EfficientNet model to match our number of classes (5). We then transfer the model to the designated device (GPU if available, else CPU) for computation.1import torch 2from torch import nn 3from efficientnet_pytorch import EfficientNet 4 5device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu') 6 7model = EfficientNet.from_pretrained('efficientnet-b0') 8model._fc = nn.Linear(in_features=1280, out_features=5, bias=True) 9model = model.to(device) 10Copy


Image Transformations

Image transformations are an essential part of deep learning pipelines. They help to augment the data, introduce variability, and reduce overfitting. Here is a sample input image:With albumentations, we specify a series of transformations that will be applied to our images.1import albumentations as A 2from albumentations.pytorch import ToTensorV2 3from torchvision import datasets, transforms, models 4 5tform = A.Compose([ 6 A.Rotate(limit=20, p=0.5), # Rotate the image within a range of ±20 degrees with a probability of 50% 7 A.Resize(224, 224), # Resize the image to 224x224 pixels 8 A.HorizontalFlip(p=0.5), # Flip the image horizontally with a probability of 50% 9 A.Normalize(mean=(0), std=(1)), # Normalize the image with mean=0 and std=1 10 ToTensorV2() # Convert the image from numpy array to PyTorch tensor 11]) 12Copy


Loss Function, Optimizer, and Scheduler

We use the CrossEntropyLoss, a popular loss function for multi-class classification problems. The Adam optimizer is used as it adapts the learning rate for each weight individually and is generally more efficient. We also define a learning rate scheduler to decrease the learning rate as the training progresses to fine-tune the model.1from torch.optim import Adam 2from torch.optim.lr_scheduler import StepLR 3import torchmetrics 4 5optimizer = Adam(model.parameters(), lr=0.01) 6scheduler = StepLR(optimizer, step_size=4, gamma=0.5) 7loss_fn = nn.CrossEntropyLoss() 8train_accuracy = torchmetrics.Accuracy(task='multiclass', num_classes=4).to(device) 9n_epochs = 2 10Copy


Training EfficientNet for Healthcare ML task

The model is trained using a standard PyTorch training loop. For each epoch, we load a batch of data, perform a forward pass through the model, calculate the loss, perform backpropagation to compute gradients, and then update the model parameters with the optimizer. We also compute the accuracy metric for monitoring purposes.1from tqdm import tqdm 2 3for epoch in range(1, n_epochs + 1): 4 total_acc = 0 5 with tqdm(data_loader, unit='iteration') as train_epoch: 6 train_epoch.set_description(f'Epoch {epoch}') 7 for i, (data, target) in enumerate(train_epoch): 8 data = data.to(device) 9 target = target.to(device) 10 11 optimizer.zero_grad() 12 logits = model(data) 13 _, pred = torch.max(logits, dim=1) 14 15 loss = loss_fn(logits, target.squeeze(0)) 16 loss.backward() 17 optimizer.step() 18 scheduler.step() 19 20 acc = train_accuracy(pred, target.squeeze(0)) 21 total_acc += acc.item() 22 train_epoch.set_postfix(loss=loss.item(), accuracy=total_acc / (i + 1)) 23CopyAnd here is the optimized model's performance during training for each of the epochs: Each of these steps contributes to creating an effective pipeline for diagnosing Diabetic Retinopathy. Careful configuration and tuning of these components can lead to high-performance models that can greatly aid in the medical diagnosis process.


Concluding remarks

In conclusion, this article has explored three key areas: Diabetic Retinopathy detection, EfficientNet, and the application of EfficientNet on the Messidor dataset. Diabetic retinopathy, a progressive eye disorder, often goes unnoticed until it begins to affect vision. It can be effectively managed with prompt diagnosis and proper treatment, but the current manual methods for diagnosis and screening are time-consuming and require substantial expertise.EfficientNet, an advanced model architecture for image classification, was introduced as a solution. It balances depth, width, and resolution to provide optimal performance and computational efficiency.We demonstrated how to train EfficientNet on the Messidor dataset to alleviate the burden of manual diabetic retinopathy diagnosis. These techniques significantly enhance the extraction of crucial information, particularly when dealing with limited data and computational resources. Automated systems, like those powered by AI and deep learning algorithms, play a crucial role in addressing the challenges associated with diabetic retinopathy diagnosis, underscoring the need for their wider adoption in the medical imaging sector.Keep in mind that we trained and tested two implementations. This notebook is more comprehensive, but suffers from poor performance, while this notebook shows the performance were capable of achieving.


Evaluating EfficientNet Performance in a Healthcare Machine Learning Task

Now let's evaluate our model's performance. We will be using the improved model trained in this notebook. We will evaluate the it on the following metrics AUROC, Confusion Matrix, and F1Score.1from torchmetrics import AUROC, ConfusionMatrix, PrecisionRecallCurve,F1Score 2auc = AUROC('multiclass', num_classes=4) 3print(auc(pred, y).item()) 4>> 0.79 5 6 This AUROC score is pretty good. How about its F1 score? 7f1 = F1Score(num_classes=5).to('cuda') 8f1(logits, y).item() 9>> 73.45 10CopyLet's take a look at our model's confusion matrix, or in other words which classes it confuses.1import seaborn as sns 2cm = ConfusionMatrix('multiclass', num_classes=4).to('cuda') 3mat = cm(pred, y).detach().cpu().numpy() 4sns.heatmap(mat) 5CopyThis is an example of confusion matrix in epoch 7. As you can see the model is able class correctly for the 1st, 3rd and 4th classes. The possible reason of why classes 1 and 2 confused is because the features of both classes are almost similar. This is quite normal in deep neural networks. To overcome this issue you can train it for a longer period of time.


What are some EfficientNet benchmarks?

The table compares key parameters of several deep learning model families: ResNet, ResNeXt, SENet, NASNet, GPipe, DenseNet, Xception, and EfficientNet. Each is evaluated on accuracy and model complexity (Parameters in Million). Notably, GPipe presents the highest complexity with 556 million parameters and 84.3% accuracy. EfficientNet models, while more efficient in terms of parameters, manage to achieve comparable accuracy, with EfficientNet-B7 hitting 84.3% with 66 million parameters. ModelAccuracyParameters (Million) ResNet-15277.8%60 ResNeXt-10180.9%84 SENet82.7%146 NASNet-A82.7%89 GPipe84.3%556 DenseNet-201~77%~20 Xception~79%~20.1 EfficientNet-B0~77%~7 EfficientNet-B179.1%7.8 EfficientNet-B2~81%~7.9 EfficientNet-B381.6%12 EfficientNet-B482.9%19 EfficientNet-B5~83%~38 EfficientNet-B6~84%~42 EfficientNet-B784.3%66


Practical Guide to Diabetic Retinopathy Detection

Now with all of the theory out of the way, let's take a look at how to implement this model in practice. We will focus on image classification using the retinopathy dataset. The complete process is documented in this linked notebook, which can be modified based on your objectives. It should be noted that we ran multiple experiments with training this model on this dataset, and we achieved better results using this notebook. Both training and testing datasets are available from Activeloop.


References


What are the features of the 4 Diabetic Retionopathy stages?

There are 4 distinct stages for Diabetic Retinopathy. The various differences are summarized in the following table:Sure, I'll add some padding to the table by adding extra spaces. Please note that how the table will appear can also be dependent on the platform or software where it is viewed. GradeClinical FeaturesCategory 0No symptoms observedNo Diabetic Retinopathy 1Presence of Microaneurysms in one of the four quadrantsMild NPDR 2Microaneurysms, dot and blot Hemorrhages, and cotton wool spotsModerate NPDR 3Intraretinal microvascular abnormalities (in 1 quadrant), Definite venous beading (in 2 quadrants), Intraretinal Hemorrhages (>= 20 in each quadrant), NeovascularizationSevere NPDR 4Advanced stage with continual growth of fragile vessels (neovascularization) leading to scar tissue, possibly causing retinal detachment, vision blurriness, and even permanent blindnessProliferative DR


Introduction

When selecting a Vector Database for production-grade Retrieval Augmented Generation (RAG) applications, two things matter most: speed and affordability. Deep Lake 3.7.1 introduces a unique and performant implementation of the HNSW Approximate Nearest Neighbor (ANN) search algorithm that improves the speed of index creation, reduces the RAM usage, and integrates Deep Lake's Query Engine for fast filtering based on metadata, text, or other attributes. The new index implementation pushes the limit for sub-second vector search from 1 million to >35 million embeddings, while significantly reducing costs for running a vector database in production.


Scalability and Performance in Deep Lake 3.7.1

Deep Lake's prior versions utilized a high-performance implementation of linear search for computing embedding similarity. While it is effective for smaller vector stores, this method is not suitable for vector stores exceeding 1 million embeddings. With the introduction of Deep Lake 3.7.1, we've added an advanced implementation of Approximate Nearest Neighbor (ANN) search, supercharging search speeds to under one second for 35 million embeddings. For smaller databases under 100,000 embeddings, linear search remains the preferred method due to maximum accuracy, while ANN search is recommended at larger scales.


Deep Lake's HNSW Implementation

Hierarchical Navigable Small World (HNSW) graphs are among the best performing and reliable indexes for vector similarity search. Deep Lake has made the HNSW index even more powerful by adding enhancements such as intelligent memory utilization and multithreading during index creation. By distributing data in the Deep Lake Vector Store between object storage, attached storage (on-disk), and RAM, Deep Lake minimizes the usage of costly memory while maintaining high performance. This is a must-have for building RAG-based Large Language Model (LLM) applications at scale.


Conclusion

As the world of Large Language Model (LLM) applications grows and matures, scalability without burning though your budget is paramount for taking your prototypes to production. Deep Lake stands out by delivering fast, sub-second vector search capabilities for datasets with up to 35 million embeddings, and at a cost that's 80% more affordable than other vector databases in the market. Your search for a powerful, budget-friendly Vector Database solution ends with Deep Lake.Try out Deep Lake Index today.


Top-tier Performance, Without the Hefty Price Tag, Unlike Other Vector Databases

Most vector databases were originally designed for applications such as recommendation engines, which require real-time search and millions of requests per day. As a result, they are typically implemented in-memory, relying heavily on RAM for data storage. Since LLM responses may take several seconds, implementing an in-memory real-time vector database is unnecessary, and it significantly increases costs without meaningfully improving the end-to-end user experience. Thanks to Deep Lake's efficient memory architecture, we've slashed storage costs by over 80% compared to many leading competitors without sacrificing the performance of your LLM apps. Our lower costs combined with industry-leading ease of use offer customers a risk-free journey for scaling projects from prototyping to production.


Use LangChain, OpenAI GPT, & Deep Lake to Chat with CSVs, PDFs, JSONs, GitHub Repos, URLs, & More

We've previously explored chatting with PDFs or understanding GitHub repos with LangChain. Many apps are popping up here and there inspired by those use cases, but DataChad, created by our community member Gustav von Zitzewitz, takes it several steps further, and works both locally or in the cloud, and allows chatting with multiple data sources of various types (PDFs, Excel sheets, etc.) at the same time.DataChad is an open-source project that allows users to ask questions about any data source by leveraging embeddings, Deep Lake as a vector database, large language models like GPT-3.5-turbo or GPT-4, and LangChain. The data source can be anything from a local file like a pdf or CSV to a website url, a GitHub repository, or even the path to a directory, scanned recursively if the app is deployed locally. The app now supports Local Mode, where all data is processed only locally and no API calls are made. This is made possible by leveraging pre-trained open source LLM models like GPT4all, and creating Deep Lake-powered embedding storage on the local disk vs in the Deep Lake cloud.The app works by uploading any file or entering any way or URL (or pointing to the location of your files using the Local Mode). Subsequently, the app detects and loads the data source into text documents, embeds the text documents using OpenAI embeddings, then stores them embeddings as a vector dataset to Activeloop's Deep Lake Cloud. A Langchain is established, comprising an LLM model and the embedding database index as a retriever. This chain serves as the context for answering user queries over any data they upload.


Why Do You Need a Chat With Any Data App?

DataChad is designed to serve as an indispensable tool for individuals who require swift and precise data querying from any source.Whether you're seeking a comprehensive understanding of a complete project or looking for swift answers from a single data source without manually sifting through the material (say, a Wikipedia article, codebase, or an academic paper you're cramming), with DataChad, users can ask natural language questions and get relevant answers in seconds without writing complex SQL queries or using other data querying tools.Finally, the app can be hosted and used from anywhere, like in the demo, or deployed locally to enable querying local directories. This makes it essential to be able to run this types of solutions locally, without the need to send companies like OpenAI your data (in that case, you'd need to use an open-source large language model).


Editorial Note on OpenAI Embeddings

Costs can become a factor for extensive OpenAI API usage. To have full transparency and control of this critical factor, DataChad will display the app's usage of tokens and total costs in $. To get a feeling about the scale, prompts using the maximum number of tokens of 4069 still lay well below a single cent of total usage costs.


Vector Database

DataChad uses Deep Lake - the vector database for all AI data, to store the embeddings generated from the text documents. Vector databases are designed to store and search vectors efficiently and are optimized for large-scale datasets. Deep Lake stands out from various vector databases in its multi-modality (i.e., ability to support multiple data types and store embedding metadata). It is highly relevant if you're looking to build an all-in-one chat with a data app like DataChad.


OpenAI Embeddings

DataChad uses OpenAI Embeddings to convert text documents into vectors that can be indexed and searched efficiently. OpenAI's embeddings are instrumental in evaluating the semantic similarity between two or more text fragments or the relevance of extensive documents to a concise query. They are extensively vital for tasks like search or classification. OpenAI embeddings employ the cosine similarity method to calculate the similarity between documents and a question.


Large Language Models (LLMs)

DataChad uses large language models like GPT-3.5 Turbo to generate responses to user questions. LLMs are powerful models trained on massive amounts of text data that can generate natural language responses to a wide range of questions.


LangChain

DataChad uses LangChain to combine the embeddings and LLMs into a single retrieval chain that can be used to answer user questions. LangChain is a powerful technique for integrating natural language processing tools into a single pipeline. Read this ultimate LangChain guide if you want to understand the power of LangChain.