EMBEDDING SEARCH

Deep Lake supports several vector operations for embedding search. Typically, vector operations are called by returning data ordered by the score based on the vector search method.select * from (select tensor_1, tensor_2, <VECTOR_OPERATION> as score) order by score desc limit 10# THE SUPPORTED VECTOR_OPERATIONS ARE:l1_norm(<embedding_tensor> - ARRAY[<search_embedding>]) # Order should be ascl2_norm(<embedding_tensor> - ARRAY[<search_embedding>]) # Order should be asclinf_norm(<embedding_tensor> - ARRAY[<search_embedding>]) # Order should be asccosine_similarity(<embedding_tensor>, ARRAY[<search_embedding>]) # Order should be desc


VIRTUAL TENSORS

Virtual tensors are the result of a computation and are not tensors in the Deep Lake dataset. However, they can be treated as tensors in the API.# 'score' is a virtual tensorselect * from (select tensor_1, tensor_2, <VECTOR_OPERATION> as score) order by score desc limit 10# 'box_beyond_image' is a virtual tensorselect *, any(boxes[:,0])<0 as box_beyond_image where ....# 'tensor_sum' is a virtual tensorselect *, tensor_1 + tensor_3 as tensor_sum where ......When combining embedding search with filtering (where conditions), the filter condition is evaluated prior to the embedding search.


GROUP BY AND UNGROUP BY

creates a sequence of data based on the common properties that are being grouped (i.e. frames into videos). Ungroup by splits sequences into their individual elements (i.e. videos into images).select * group by label, video_id # Groups all data with the same label and video_id in to the same sequenceselect * ungroup by split # Splits sequences into their original pieces


EXPAND BY

includes samples before and after a query condition is satisfied.select * where <condition> expand by rows_before, rows_after


How to sample datasets using Deep Lake's query engine

Sampling is often used when training models in order to modify the distribution of data that models are trained on. A common objective of samplers is to rebalance the data in order to achieve an more uniform distribution of classes in the training loop. Deep Lake provides a powerful API for several sampling methods via the query engine. Sampler queries in Deep Lake are only accessible to registered and authenticated users, and it applies usage restrictions based on your Deep Lake Plan.The general syntax for sampling is using the sample by keywords:select * sample by weight_choice(expression_1: weight_1, expression_2: weight_2, ...) replace True limit Nweight_choice resolves the weight that is used when multiple expressions evaluate to True for a given sample. Options are max_weight, sum_weight. For example, if weight_choice is max_weight, then the maximum weight will be chosen for that sample.replace determines whether samples should be drawn with replacement. It defaults to True.limit specifies the number of samples that should be returned. If unspecified, the sampler will return the number of samples corresponding to the length of the datasetSampling can be performed in the query interface in the Deep Lake UI, or in the Python API as shown below.


Example Usage

Suppose we're working with a medical imaging dataset such as the NIH Chest X-Ray. Let's use samplers to create a more balanced view of the dataset that we can use for training a model. First, let's load the dataset:import deeplakeimport numpy as npfrom matplotlib import pyplot as pltds = deeplake.load('hub://activeloop/nih-chest-xray-train')Next, let's calculate the a histogram of the medical findings (findings tensor) and plot it.# Extract the list of class_names in a separate variable for re-useclass_names = ds.findings.info.class_namesnum_classes = len(class_names)class_count_raw = np.bincount(np.concatenate(ds.findings.numpy(aslist = True), axis=0))plt.bar(np.arange(num_classes), class_count_raw, tick_label = class_names)plt.xlabel('Condition', weight='bold')plt.xticks(rotation='vertical')plt.ylabel('Number of Instances', weight='bold')plt.title('Frequency per Condition', weight='bold')plt.show()We observe that findings such as Hernia, Pneumonia, Fibrosis, Edema , and Emphysema are very rare, which may cause our model to underperform when predicting these conditions. Note that even though many images have No_Finding, this is desirable for avoiding false positives when training models for medical imaging applications.We can use Deep Lake Tensor-Query-Language to upsample the under-represented findings in order to create a more balanced dataset.select * sample by max_weight(contains(findings, 'Hernia'): 20, contains(findings, 'Pneumonia'): 8, contains(findings, 'Fibrosis'): 5, contains(findings, 'Edema'): 5, contains(findings, 'Emphysema'): 2, True: 1)We can run this query in the UI or in the Python API using ds.query(...):balanced_view = ds.query('select * sample by max_weight(contains(findings, 'Hernia'): 20, contains(findings, 'Pneumonia'): 8, contains(findings, 'Fibrosis'): 5, contains(findings, 'Edema'): 5, contains(findings, 'Emphysema'): 2, True: 1)')In this sampler query, we're upsampling Hernia, by 20x, Pneumonia by 8x, Fibrosis by 5x, Edema by 5x, and Emphysema by 2x. Let's recalculate the histogram for the balanced dataset and compare it to the raw data histogram. class_count_balanced = np.bincount(np.concatenate(balanced_view.findings.numpy(aslist = True), axis=0))X_axis = np.arange(len(class_names))plt.figure(figsize=(8, 4))plt.bar(X_axis - 0.2, class_count_raw, 0.4, label = 'Raw Data')plt.bar(X_axis + 0.2, class_count_balanced, 0.4, label = 'Sampled Data') plt.xticks(X_axis, class_names)plt.xlabel('Condition', weight='bold')plt.xticks(rotation='vertical')plt.ylabel('Number of Instances', weight='bold')plt.title('Frequency per Condition', weight='bold')plt.legend()plt.show()The data in the upsampled dataset has much better representation of the rare conditions. Note that since a given image may have multiple conditions, and since conditions can be correlated, upsampling by one condition may implicitly upsample another condition, if they tend to occur in the same image.


Training Models on Sampled Views

The sampled dataset view can be passed to a dataloader just like an ordinary Deep Lake dataset. Examples of dataset training can be found in our training tutorials.pytorch_dataloader = balanced_view.pytorch(...)for data in pytorch_dataloader: # Training loop


How to Filter and Query Data in Deep Lake

Filtering and querying is an important aspect of data engineering because analyzing and utilizing data in smaller units is much more productive than executing workflows on all data all the time. Queries can be performed in Deep Lake enables with user-defined functions, or they can be executed in Activeloop Platform using our highly-performance SQL-style query language.


Filtering using our Tensor Query Language (TQL)

Deep Lake offers a highly-performant SQL-style query language that is built in C++ and is optimized for Deep Lake datasets. Queries and their results are executed and saved in the UI, and they can be accessed in Deep Lake using using the the Dataset Views API described below.Full details about the query language are described in a standalone tutorial.


Filtering with user-defined-functions (UDF)

The first step for querying using UDFs is to define a function that returns a boolean depending on whether an dataset sample meets the user-defined condition. In this example, we define a function that returns True if the labels in a tensor are in the desired labels_list. If there are inputs to the filtering function other than sample_in, it must be decorated with @deeplake.compute.import deeplakefrom PIL import Image# Let's create a local copy of the dataset (Explanation is in the next section)ds = deeplake.deepcopy('hub://activeloop/mnist-train', './mnist-train-local') labels_list = ['0', '8'] # Desired labels for filtering@deeplake.computedef filter_labels(sample_in, labels_list): return sample_in.labels.data()['text'][0] in labels_listThe filtering function is executed using the ds.filter() command below, and it returns a Dataset View that only contains the indices that met the filtering condition (more details below). Just like in the Parallel Computing API, the sample_in parameter does not need to be passed into the filter function when evaluating it, and multi-processing can be specified using the scheduler and num_workers parameters.ds_view = ds.filter(filter_labels(labels_list), scheduler = 'threaded', num_workers = 0)print(len(ds_view))In most cases, multi-processing is not necessary for queries that involve simple data such as labels or bounding boxes. However, multi-processing significantly accelerates queries that must load rich data types such as images and videos.


Dataset Views

A Dataset View is any subset of a Deep Lake dataset that does not contains all of the samples. It can be an output of a query, filtering function, or regular indexing like ds[0:2:100].In the filtering example above, we copied mnist-train locally in order to gain write access to the dataset. With write access, the views are saved as part of the dataset. Without write access, views are stored elsewhere or in custom paths, and full details are available here. Users have write access to their own datasets, regardless of whether the datasets are local or in the cloud.The data in the returned ds_view can be accessed just like a regular dataset. Image.fromarray(ds_view.images[10].numpy())A Dataset View can be saved permanently using the method below, which stores its indices without copying the data:ds_view.save_view(message = 'Samples with 0 and 8')In order to maintain data lineage, Dataset Views are immutable and are connected to specific commits. Therefore, views can only be saved if the dataset has a commit and there are no uncommitted changes in the HEAD. Each Dataset View has a unique id, and views can be examined or loaded using:views = ds.get_views()print(views)ds_view = views[0].load()# OR# ds_view = ds.load_view(id)print(len(ds_view))Congrats! You just learned to filter and query data with Deep Lake! 🎈


How Deep Memory Works

Deep Memory computes transformation of your embeddings based on several examples of embeddings for which the most relevant embedding in the Vector Store is known. The transformation is computed computed on a Deep Lake Managed Service, and it is applied at inference in Deep Lake's Tensor Query Language (TQL).


Tuning the Index Parameters

The parameters of the HSNW index can be tuned using the index_params shown below:vectorstore = VectorStore(path, index_params = {'threshold': <DEFAULT_-1>, # Index turned off by default 'distance_metric':<DEFAULT_COS>, 'additional_params': { 'efConstruction': <DEFAULT_200>, 'M': <DEFAULT_16>}})Further information about the impact of the index parameters can be found here.


Caching of Embeddings and Index

Either of the following operations caches the embeddings are on disk and the index in RAM:The index is createdThe first vector search is executed after the Vector Store is loadedSince the first query caches critical information, subsequent queries will execute much faster compared to the first query. Since the cache is invalidated after the Vector Store is loaded or initialized, the optimal access pattern is not to re-load the Vector Store each search, unless you believe it was updated by another client.The embeddings are cached on disk in the following locations:Mac: /tmp/....Linux: /var/folders/


Caching of Other Tensors

Tensors containing other information such as text and metadata are also cached in memory when they are used in queries. As a result, the first query that utilized this data will be slowest, with subsequent queries running much faster. If the data size exceeds the cache size, it will be re-fetched with every query, thus reducing query performance. The default cache size is 2 MB, and you may increase the cache size using the parameter below:vectorstore = VectorStore(path, memory_cache_size = <cache_in_MB))


Querying via the REST API

The primary input to the query API is a query string that contains all the necessary information for executing the query, including the path to the Deep Lake data. Full details on the query syntax are available here.


Overview of the Managed Database REST API

The REST API is currently in Alpha, and the syntax may change without announcement.The Deep Lake Tensor Database can be accessed via REST API. The datasets must be stored in the Tensor Database by specifying the deeplake_path = hub://org_id/dataset_name and runtime = {'tensor_db': True}. Full details on path and storage management are available here.


Input

url = 'https://app.activeloop.ai/api/query/v1'headers = { 'Authorization': f'Bearer {user_token}' }# Format the embedding array or list as a string, so it can be passed in the REST API request.embedding_string = ','.join([str(item) for item in embedding])request = { 'query': f'select * from (select text, cosine_similarity(embedding, ARRAY[{embedding_string}]) as score from \'{dataset_path}\') order by score desc limit 5', 'as_list': True/False # Defaults to True. }


Response

If as_list = True (default).  Returns a list of jsons, one per row.{ 'message': 'Query successful.', 'tensors': [ 'text', 'score' ], 'data': [ { 'text': '# Twitter's Recommendation Algorithm\n\nTwitter's Recommendation Algorithm is a set of services and jobs that are responsible for constructing and serving the\nHome Timeline.  For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm).  The\ndiagram below illustrates how major services and jobs interconnect.\n\n![](docs/system-diagram.png)\n\nThese are the main components of the Recommendation Algorithm included in this repository:', 'score': 22.59016227722168 }, { 'text': '![](docs/system-diagram.png)\n\nThese are the main components of the Recommendation Algorithm included in this repository:', 'score': 22.5976619720459 },...  ]If as_list = False.  Returns a list of values per tensor.{ 'message': 'Query successful.', 'tensors': [ 'text', 'score' ], 'data': { 'text': [ '# Twitter's Recommendation Algorithm\n\nTwitter's Recommendation Algorithm is a set of services and jobs that are responsible for constructing and serving the\nHome Timeline.  For an introduction to how the algorithm works, please refer to our [engineering blog](https://blog.twitter.com/engineering/en_us/topics/open-source/2023/twitter-recommendation-algorithm).  The\ndiagram below illustrates how major services and jobs interconnect.\n\n![](docs/system-diagram.png)\n\nThese are the main components of the Recommendation Algorithm included in this repository:', '![](docs/system-diagram.png)\n\nThese are the main components of the Recommendation Algorithm included in this repository:', '| Type | Component | Description |\n|------------|------------|------------|\n| Feature | [SimClusters](src/scala/com/twitter/simclusters_v2/README.md) | Community detection and sparse embeddings into those communities.  |\n| | [TwHIN](https://github.com/twitter/the-algorithm-ml/blob/main/projects/twhin/README.md) | Dense knowledge graph embeddings for Users and Tweets.  |\n| | [trust-and-safety-models](trust_and_safety_models/README.md) | Models for detecting NSFW or abusive content.  |\n| | [real-graph](src/scala/com/twitter/interaction_graph/README.md) | Model to predict the likelihood of a Twitter User interacting with another User.  |\n| | [tweepcred](src/scala/com/twitter/graph/batch/job/tweepcred/README) | Page-Rank algorithm for calculating Twitter User reputation.  |\n| | [recos-injector](recos-injector/README.md) | Streaming event processor for building input streams for [GraphJet](https://github.com/twitter/GraphJet) based services.  |\n| | [graph-feature-service](graph-feature-service/README.md) | Serves graph features for a directed pair of Users (e.g.  how many of User A's following liked Tweets from User B).  |\n| Candidate Source | [search-index](src/java/com/twitter/search/README.md) | Find and rank In-Network Tweets.  ~50% of Tweets come from this candidate source.  |\n| | [cr-mixer](cr-mixer/README.md) | Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services.  |\n| | [user-tweet-entity-graph](src/scala/com/twitter/recos/user_tweet_entity_graph/README.md) (UTEG)| Maintains an in memory User to Tweet interaction graph, and finds candidates based on traversals of this graph.  This is built on the [GraphJet](https://github.com/twitter/GraphJet) framework.  Several other GraphJet based features and candidate sources are located [here](src/scala/com/twitter/recos).  |\n| | [follow-recommendation-service](follow-recommendations-service/README.md) (FRS)| Provides Users with recommendations for accounts to follow, and Tweets from those accounts.  |\n| Ranking | [light-ranker](src/python/twitter/deepbird/projects/timelines/scripts/models/earlybird/README.md) | Light Ranker model used by search index (Earlybird) to rank Tweets.  |\n| | [heavy-ranker](https://github.com/twitter/the-algorithm-ml/blob/main/projects/home/recap/README.md) | Neural network for ranking candidate tweets.  One of the main signals used to select timeline Tweets post candidate sourcing.  |\n| Tweet mixing & filtering | [home-mixer](home-mixer/README.md) | Main service used to construct and serve the Home Timeline.  Built on [product-mixer](product-mixer/README.md)


Response

|\n| | [visibility-filters](visibilitylib/README.md) | Responsible for filtering Twitter content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking .  |\n| | [timelineranker](timelineranker/README.md) | Legacy service which provides relevance-scored tweets from the Earlybird Search Index and UTEG service.  |\n| Software framework | [navi](navi/README.md) | High performance, machine learning model serving written in Rust.  |\n| | [product-mixer](product-mixer/README.md) | Software framework for building feeds of content.  |\n| | [twml](twml/README.md) | Legacy machine learning framework built on TensorFlow v1.  |', 'We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file.\n\n## Contributing', 'We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file.\n\n## Contributing\n\nWe invite the community to submit GitHub issues and pull requests for suggestions on improving the recommendation algorithm.  We are working on tools to manage these suggestions and sync changes to our internal repository.  Any security concerns or issues should be routed to our official [bug bounty program](https://hackerone.com/twitter) through HackerOne.  We hope to benefit from the collective intelligence and expertise of the global community in helping us identify issues and suggest improvements, ultimately leading to a better Twitter.\n\nRead our blog on the open source initiative [here](https://blog.twitter.com/en_us/topics/company/2023/a-new-era-of-transparency-for-twitter).' ], 'score': [ 22.843185424804688, 22.83962631225586, 22.835460662841797, 22.83342170715332, 22.832916259765625 ] }}


How to migrate existing Deep Lake datasets to the Tensor Database

Datasets are created in the Tensor Database by specifying the dataset_path = 'hub://<org_id>/<dataset_name>' and runtime = {'tensor_db': True}) during dataset creation. If datasets are currently stored locally, in your could, or in non-database Activeloop storage, they can be migrated to the Tensor Database using:import deeplakeds_tensor_db = deeplake.deepcopy(src = <current_path>, dest = 'hub://<org_id>/<dataset_name>', runtime = {'tensor_db': True}, src_creds = {<creds_dict>}, # Only necessary if src is in your cloud )


How to use Deep Lake as a Vector Store for LLM applications

Deep Lake can be used as a Vector Store for storing embeddings and their metadata including text, jsons, images, audio, video, and more. Its serverless architecture can be self-hosted, and it is also available via fully managed service. Deep Lake's designated API for Vector Store applications is shown in these tutorials. Details on the low-level Deep Lake API are available here.


Vector Store Tutorials:

Deep Lake Vector Store in LangChainVector Search Options


How to use Deep Lake for Deep Learning Applications

Deep Lake can be used as tool for managing Deep Learning data, including rapidly training models while streaming data, running queries, tracking dataset versions, visualizing datasets, and more.These tutorials show how to use Deep Lake's low-level API for deep-learning use cases.


Deep Learning Tutorials:

Creating DatasetsTraining ModelsUpdating DatasetsData Processing Using Parallel Computing


How to use queries and version control to train models with reproducible data lineage.

The road from raw data to a trainable deep-learning dataset can be treacherous, often involving multiple tools glued together with spaghetti code. Activeloop simplifies this journey so you can create high-quality datasets and train production-level deep-learning models.


This playbook demonstrates how to use Activeloop Deep Lake to:

Create a Deep Lake dataset from data stored in an S3 bucketVisualize the data to gain insights about the underlying data challenges Update, edit, and store different versions of the data with reproducibilityQuery the data, save the query result, and materialize it for training a model.Train a object detection model while streaming data


Prerequisites

In addition to installation of commonly used packages, this playbook requires installation of: pip3 install deeplakepip3 install albumentationspip3 install opencv-python-headless==4.1.2.30 #In order for Albumentations to work properlypip install 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'The required python imports are:import deeplakeimport numpy as npimport boto3import mathimport timeimport osfrom tqdm import tqdmfrom pycocotools.coco import COCOimport albumentations as Afrom albumentations.pytorch import ToTensorV2import torchimport torchvisionfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictorYou should also register with Activeloop and create an API token in the UI.


Creating the Dataset

Since many real-world datasets use the COCO annotation format, the COCO training dataset is used in this playbook. To avoid data duplication, linked tensors are used to store references to the images in the Deep Lake dataset from the S3 bucket containing the original data. For simplicity, only the bounding box annotations are copied to the the Deep Lake dataset.To convert the original dataset to Deep Lake format, let's establish a connection to the original data in S3.dataset_bucket = 'non-hub-datasets's3 = boto3.resource('s3', aws_access_key_id=os.environ.get('aws_access_key_id'), aws_secret_access_key=os.environ.get('aws_secret_access_key'))s3_bucket = s3.Bucket(dataset_bucket)Next, let's load the annotations so we can access them later:ann_path = 'coco/annotations/instances_train2017.json'local_ann_path = 'anns_train.json's3_bucket.download_file(ann_path, local_ann_path)coco = COCO(local_ann_path)category_info = coco.loadCats(coco.getCatIds())Moving on, let's create an empty Deep Lake dataset and pull managed credentials from Platform, so that we don't have to manually specify the credentials to access the s3 links every time we use this dataset. Since the Deep Lake dataset is stored in Deep Lake storage, we also provide an API token to identify the user.ds = deeplake.empty('hub://dl-corp/coco-train', token = 'Insert API Token')creds_name = 'my_s3_creds'ds.add_creds_key(creds_name, managed = True)The UI for managed credentials in Platform is shown below, and more details are available here.Last but not least, let's create the Deep Lake dataset's tensors. In this example, we ignore the segmentations and keypoints from the COCO dataset, only uploading the bounding box annotations as well as their labels.img_ids = sorted(coco.getImgIds()) # Image ids for uploadingwith ds: ds.create_tensor('images', htype = 'link[image]', sample_compression = 'jpg') ds.create_tensor('boxes', htype = 'bbox') ds.create_tensor('categories', htype = 'class_label')Finally, let's iterate through the data and append it to our Deep Lake dataset. Note that when appending data, we directly pass the s3 URL and the managed credentials key for accessing that URL using deeplake.link(url, creds_key)with ds: ## ---- Iterate through each image and upload data ----- ## for img_id in tqdm(img_ids): anns = coco.loadAnns(coco.getAnnIds(img_id)) img_coco = coco.loadImgs(img_id)[0] #First create empty objects for all the annotations boxes = np.zeros((len(anns),4)) categories = [] #Then populate the objects with the annotations data for i, ann in enumerate(anns): boxes[i,:] = ann['bbox'] categories.append([category_info[i]['name'] for i in range(len(category_info)) if category_info[i]['id']==ann['category_id']][0]) #If there are no categories present, append the empty list as None. if categories == []: categories = None img_url = 's3://{}/coco/train2017/{}'.format(dataset_bucket, img_coco['file_name']) ds.append({'images': deeplake.link(img_url, creds_key=creds_name), 'boxes': boxes.astype('float32'), 'categories': categories})Note: if dataset creation speed is a priority, it can be accelerated using 2 options:By uploading the dataset in parallel. An example is available here.By setting the optional parameters below to False. In this case, the upload machine will not load any of the data before creating the dataset, thus speeding the upload by up to 100X. The parameters below are defaulted to True because they improve the query speed on image shapes and file metadata, and they also verify the integrity of the data before uploading. More information is available here:ds.create_tensor('images', htype = 'link[image]', verify = False, create_shape_tensor = False, create_sample_info_tensor = False )


Inspecting the Dataset

In this example, we will train an object detection model for driving applications. Therefore, we are interested in images containing cars, busses, trucks, bicycles, motorcycles, traffic lights, and stop signs, which we can find by running a SQL query on the dataset in Platform. More details on the query syntax are available here.(select * where contains(categories, 'car') limit 1000) union (select * where contains(categories, 'bus') limit 1000) union (select * where contains(categories, 'truck') limit 1000) union (select * where contains(categories, 'bicycle') limit 1000) union (select * where contains(categories, 'motorcycle') limit 1000) union (select * where contains(categories, 'traffic light') limit 1000) union (select * where contains(categories, 'stop sign') limit 1000)A quick visual inspection of the dataset reveals several problems with the data including:Sample 61 but is a-low quality image where it's very difficult to discern the features, and it is not clear whether the small object in the distance is an actual traffic light. Images like this do not positively contribute to model performance, so let's delete all the data in this sample.ds.pop(61)ds.commit('Deleted index 61 because the image is low quality.')In sample 8, a road sign is labeled as a stop sign, even though the sign is facing away from the camera. Even though it may be a stop sign, computer vision systems should positively identify the type of a road sign based on its visible text. Therefore, let's remove the stop sign label from this image. ds.categories[8] = ds.categories[8].numpy()[np.arange(0,4)!=2]ds.boxes[8] = ds.boxes[8].numpy()[np.arange(0,4)!=2,:]ds.commit('Deleted bad label at index 8')Both changes are now evident in the visualizer, and they were both logged as separate commits in the version control history. A summary of this inspection workflow is shown below:


Optimizing the Dataset for Training

Now that the dataset has been improved, we save the query result containing the samples of interest and optimize the data for training. Since query results are associated with a particular commit, they are immutable and can be retrieved at any point in time.First, let's re-run the query and save the result as a dataset view, which is uniquely identified by an id.The dataset is currently storing references to the images in S3, so the images are not rapidly streamable for training. Therefore, we materialize the query result (Dataset View) by copying and re-chunking the data for maximum performance:ds.load_view('62d6d490e49d0d7bab4e251f', optimize = True, num_workers = 4)Once we're finished using the materialized dataset view, we may choose to delete it using:# ds.delete_view('62d6d490e49d0d7bab4e251f')


Training an Object Detection Model

An object detection model can be trained using the same approach that is used for all Deep Lake datasets, with several examples in our tutorials.  Typically the training would occur on another machine with more GPU power, so we start by loading the dataset and and corresponding dataset view:ds = deeplake.load('hub://dl-corp/coco-train', token = 'Insert API Token')ds_view = ds.load_view('62d6d490e49d0d7bab4e251f')When using subsets of datasets, it's advised to remap the input classes for model training.  In this example, the source dataset has 81 classes, but we are only interested in 7 classes (cars, busses, trucks, bicycles, motorcycles, traffic lights, and stop signs).  Therefore, we remap the classes of interest to values 0,1,2,3,4,6 before feeding them into the model for training.  We also specify resolution for resizing the data before training the model.WIDTH = 128HEIGHT = 128# These are the classes we care about and they will be remapped to 0,1,2,3,4,6 in the modelCLASSES_OF_INTEREST = ['car', 'truck', 'bus', 'motorcycle', 'bicycle', 'traffic light', 'stop sign']# The classes of interest correspond to the following array values in the current datasetINDS_OF_INTEREST = [ds.categories.info.class_names.index(item) for item in CLASSES_OF_INTEREST]Next, let's specify an augmentation pipeline, which mostly utilizes Albumentations.  We perform the remapping of the class labels inside the transformation function.# Augmentation pipeline using Albumentationstform_train = A.Compose([ A.RandomSizedBBoxSafeCrop(width=WIDTH, height=HEIGHT, erosion_rate=0.2), A.Rotate(limit=20, p=0.5), A.RandomBrightnessContrast(brightness_limit=0.1, contrast_limit=0.1, p=0.5), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), ToTensorV2()], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['class_labels', 'bbox_ids'], min_area=16, min_visibility=0.6)) # 'label_fields' and 'box_ids' are all the fields that will be cut when a bounding box is cut.# Transformation function for pre-processing the deeplake sample before sending it to the modeldef transform_train(sample_in): # Convert any grayscale images to RGB image = sample_in['images'] shape = image.shape if shape[2] == 1: image = np.repeat(image, int(3/shape[2]), axis = 2) # Convert boxes to Pascal VOC format boxes = coco_2_pascal(sample_in['boxes'], shape) # Filter only the labels that we care about for this training run labels_all = sample_in['categories'] indices = [l for l, label in enumerate(labels_all) if label in INDS_OF_INTEREST] labels_filtered = labels_all[indices] labels_remapped = [INDS_OF_INTEREST.index(label) for label in labels_filtered] boxes_filtered = boxes[indices,:] # Make sure the number of labels and boxes is still the same after filtering assert(len(labels_remapped)) == boxes_filtered.shape[0] # Pass all data to the Albumentations transformation transformed = tform_train(image = image, bboxes = boxes_filtered, bbox_ids = np.arange(boxes_filtered.shape[0]), class_labels = labels_remapped, ) # Convert boxes and labels from lists to torch tensors, because Albumentations does not do that automatically


Training an Object Detection Model

# Be very careful with rounding and casting to integers, becuase that can create bounding boxes with invalid dimensions labels_torch = torch.tensor(transformed['class_labels'], dtype = torch.int64) boxes_torch = torch.zeros((len(transformed['bboxes']), 4), dtype = torch.int64) for b, box in enumerate(transformed['bboxes']): boxes_torch[b,:] = torch.tensor(np.round(box)) # Put annotations in a separate object target = {'labels': labels_torch, 'boxes': boxes_torch} return transformed['image'], target# Conversion script for bounding boxes from coco to Pascal VOC formatdef coco_2_pascal(boxes, shape): # Convert bounding boxes to Pascal VOC format and clip bounding boxes to make sure they have non-negative width and height return np.stack((np.clip(boxes[:,0], 0, None), np.clip(boxes[:,1], 0, None), np.clip(boxes[:,0]+np.clip(boxes[:,2], 1, None), 0, shape[1]), np.clip(boxes[:,1]+np.clip(boxes[:,3], 1, None), 0, shape[0])), axis = 1)def collate_fn(batch): return tuple(zip(*batch))You can now create a PyTorch dataloader that connects the Deep Lake dataset to the PyTorch model using the provided method ds_view.pytorch() .  This method automatically applies the transformation function and takes care of random shuffling (if desired).  The num_workers parameter can be used to parallelize data preprocessing, which is critical for ensuring that preprocessing does not bottleneck the overall training workflow.train_loader = ds_view.pytorch(num_workers = 8, shuffle = True, transform = transform_train, tensors = ['images', 'categories', 'boxes'], batch_size = 16, collate_fn = collate_fn)This playbook uses a pre-trained torchvision neural network from the torchvision.models module.  We define helper functions for loading the model and for training 1 epoch.# Helper function for loading the modeldef get_model_object_detection(num_classes): # Load an instance segmentation model pre-trained on COCO model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) # Get number of input features for the classifier in_features = model.roi_heads.box_predictor.cls_score.in_features # Replace the pre-trained head with a new one model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) return model # Helper function for training for 1 epochdef train_one_epoch(model, optimizer, data_loader, device): model.train() start_time = time.time() for i, data in enumerate(data_loader): images = list(image.to(device) for image in data[0]) targets = [{k: v.to(device) for k, v in t.items()} for t in data[1]] loss_dict = model(images, targets) losses = sum(loss for loss in loss_dict.values()) loss_value = losses.item() # Print performance statistics if i%10 ==0: batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] loss: %.3f, speed: %.2f' % (i, loss_value, speed)) if not math.isfinite(loss_value): print(f'Loss is {loss_value}, stopping training') print(loss_dict) break optimizer.zero_grad() losses.backward() optimizer.step()Training is performed on a GPU if possible.  Otherwise, it's on a CPU.device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')print(device)Let's initialize the model and optimizer.model = get_model_object_detection(len(CLASSES_OF_INTEREST))model.to(device)# Specify the optimizerparams = [p for p in model.parameters() if p.requires_grad]optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)The model and data are ready for training 🚀!# Train the model for 1 epochnum_epochs = 3lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1) for epoch in range(num_epochs): # Loop over the dataset multiple times print('------------------ Training Epoch {} ------------------'.format(epoch+1)) train_one_epoch(model, optimizer, train_loader, device) lr_scheduler.step() # --- Insert Testing Code Here ---print('Finished Training')


How to evaluate model performance and compare ground-truth annotations with model predictions.

Models are never perfect after the first training, and model predictions need to be compared with ground-truth annotations in order to iterate on the training process. This comparison often reveals incorrectly annotated data and sheds light on the types of data where the model fails to make the correct prediction.This playbook demonstrates how to use Activeloop Deep Lake to:Improve training data by finding data for which the model has poor performanceTrain an object detection model using a Deep Lake datasetUpload the training loss per image to a branch on the dataset designated for evaluating model performanceSort the training dataset based on model loss and identify bad samplesEdit and clean the bad training data and commit the changesEvaluate model performance on validation data and identify difficult dataCompute model predictions of object detections for a validation Deep Lake datasetUpload the model predictions to the validation dataset, compared them to ground truth annotations, and identify samples for which the model fails to make the correct predictions.


Prerequisites

In addition to installation of commonly user packages, this playbook requires installation of: pip3 install deeplakepip3 install albumentationspip3 install opencv-python-headless==4.1.2.30 #In order for Albumentations to work properlyThe required python imports are:import deeplakeimport numpy as npimport mathimport sysimport timefrom PIL import Imageimport albumentations as Afrom albumentations.pytorch import ToTensorV2import torchimport torchvisionfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictorYou should also register with Activeloop and create an API token in the UI.


Training an Object Detection Model

An object detection model can be trained using the same approach that is used for all Deep Lake datasets, with several examples in our tutorials.  First, let's specify an augmentation pipeline, which mostly utilizes Albumentations.  We also define several helper functions for resizing and converting the format of bounding boxes.WIDTH = 128HEIGHT = 64NUM_CLASSES = len(ds_train.labels.info.class_names)# Augmentation pipeline for training using Albumentationstform_train = A.Compose([ A.RandomSizedBBoxSafeCrop(width=WIDTH, height=HEIGHT, erosion_rate=0.2), A.Rotate(limit=20, p=0.5), A.RandomBrightnessContrast(brightness_limit=0.1, contrast_limit=0.1, p=0.5), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), ToTensorV2()], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['class_labels', 'bbox_ids'], min_area=8, min_visibility=0.6)) # 'label_fields' and 'box_ids' are all the fields that will be cut when a bounding box is cut.# Augmentation pipeline for validation using Albumentationstform_val = A.Compose([ A.Resize(width=WIDTH, height=HEIGHT), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), ToTensorV2()], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['class_labels', 'bbox_ids'], min_area=8, min_visibility=0.6)) # 'label_fields' and 'box_ids' are all the fields that will be cut when a bounding box is cut.# Transformation function for pre-processing the Deep Lake training sample before sending it to the modeldef transform_train(sample_in): # Convert any grayscale images to RGB image = sample_in['images'] shape = image.shape if shape[2] == 1: image = np.repeat(image, 3, axis = 2) # Convert boxes to Pascal VOC format boxes = coco_2_pascal(sample_in['boxes'], shape)  # Pass all data to the Albumentations transformation transformed = tform_train(image = image, bboxes = boxes, bbox_ids = np.arange(boxes.shape[0]), class_labels = sample_in['labels'], ) # Convert boxes and labels from lists to torch tensors, because Albumentations does not do that automatically.  # Be very careful with rounding and casting to integers, becuase that can create bounding boxes with invalid dimensions labels_torch = torch.tensor(transformed['class_labels'], dtype = torch.int64) boxes_torch = torch.zeros((len(transformed['bboxes']), 4), dtype = torch.int64) for b, box in enumerate(transformed['bboxes']): boxes_torch[b,:] = torch.tensor(np.round(box))  # Put annotations in a separate object target = {'labels': labels_torch, 'boxes': boxes_torch} return transformed['image'], target# Transformation function for pre-processing the Deep Lake validation sample before sending it to the modeldef transform_val(sample_in): # Convert any grayscale images to RGB image = sample_in['images'] shape = image.shape if shape[2] == 1: image = np.repeat(images, 3, axis = 2) # Convert boxes to Pascal VOC format boxes = coco_2_pascal(sample_in['boxes'], shape)  # Pass all data to the Albumentations transformation transformed = tform_val(image = image, bboxes = boxes, bbox_ids = np.arange(boxes.shape[0]), class_labels = sample_in['labels'], ) # Convert boxes and labels from lists to torch tensors, because Albumentations does not do that automatically


Training an Object Detection Model

# Be very careful with rounding and casting to integers, becuase that can create bounding boxes with invalid dimensions labels_torch = torch.tensor(transformed['class_labels'], dtype = torch.int64) boxes_torch = torch.zeros((len(transformed['bboxes']), 4), dtype = torch.int64) for b, box in enumerate(transformed['bboxes']): boxes_torch[b,:] = torch.tensor(np.round(box))  # Put annotations in a separate object target = {'labels': labels_torch, 'boxes': boxes_torch} # We also return the shape of the original image in order to resize the predictions to the dataset image size return transformed['image'], target, sample_in['index'], shape# Conversion script for bounding boxes from coco to Pascal VOC formatdef coco_2_pascal(boxes, shape): # Convert bounding boxes to Pascal VOC format and clip bounding boxes to make sure they have non-negative width and height return np.stack((np.clip(boxes[:,0], 0, None), np.clip(boxes[:,1], 0, None), np.clip(boxes[:,0]+np.clip(boxes[:,2], 1, None), 0, shape[1]), np.clip(boxes[:,1]+np.clip(boxes[:,3], 1, None), 0, shape[0])), axis = 1)# Conversion script for resizing the model predictions back to shape of the dataset imagedef model_2_image(boxes, model_shape, img_shape): # Resize the bounding boxes convert them from Pascal VOC to COCO m_h, m_w = model_shape i_h, i_w = img_shape x0 = boxes[:,0]*(i_w/m_w) y0 = boxes[:,1]*(i_h/m_h) x1 = boxes[:,2]*(i_w/m_w) y1 = boxes[:,3]*(i_h/m_h) return np.stack((x0, y0, x1-x0, y1-y0), axis = 1)def collate_fn(batch): return tuple(zip(*batch))We can now create a PyTorch dataloader that connects the Deep Lake dataset to the PyTorch model using the provided method ds.pytorch() .  This method automatically applies the transformation function and takes care of random shuffling (if desired).  The num_workers parameter can be used to parallelize data preprocessing, which is critical for ensuring that preprocessing does not bottleneck the overall training workflow.train_loader = ds_train.pytorch(num_workers = 8, shuffle = True, transform = transform_train, tensors = ['images', 'labels', 'boxes'], batch_size = 4, collate_fn = collate_fn)This playbook uses a pre-trained torchvision neural network from the torchvision.models module.  We define helper functions for loading the model and for training 1 epoch.# Helper function for loading the modeldef get_model_object_detection(num_classes): # Load an instance segmentation model pre-trained on COCO model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) # Get number of input features for the classifier in_features = model.roi_heads.box_predictor.cls_score.in_features # Replace the pre-trained head with a new one model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) return model # Helper function for training for 1 epochdef train_one_epoch(model, optimizer, data_loader, device): model.train() start_time = time.time() for i, data in enumerate(data_loader): images = list(image.to(device) for image in data[0]) targets = [{k: v.to(device) for k, v in t.items()} for t in data[1]] loss_dict = model(images, targets) losses = sum(loss for loss in loss_dict.values()) loss_value = losses.item() # Print performance statistics if i%100 ==0: batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] loss: %.3f, speed: %.2f' % (i, loss_value, speed)) if not math.isfinite(loss_value): print(f'Loss is {loss_value}, stopping training') print(loss_dict) break optimizer.zero_grad() losses.backward() optimizer.step()Training is performed on a GPU if possible


Creating the Dataset

In this playbook we will use the svhn-train and -test datasets that are already hosted by Activeloop. Let's copy them to our own organization dl-corp in order to have write access:ds_train = deeplake.deepcopy('hub://activeloop/svhn-train', 'hub://dl-corp/svhn-train', )ds_test = deeplake.deepcopy('hub://activeloop/svhn-test', 'hub://dl-corp/svhn-test')These are object detection datasets that localize address numbers on buildings:Let's create a branch called training_run on both datasets for storing the model results.ds_train.checkout('training_run', create = True)ds_test.checkout('training_run', create = True)Since we will write the model results back to the Deep Lake datasets, let's create a group called model_evaluation in the datasets and add tensors that will store the model results.Putting the model results in a separate group will prevent the visualizer from confusing the predictions and ground-truth data.# Store the loss in the training datasetds_train.create_group('model_evaluation')ds_train.model_evaluation.create_tensor('loss')# Store the predictions for the labels, boxes, and the average iou of the # boxes, for the test datasetds_test.create_group('model_evaluation')ds_test.model_evaluation.create_tensor('labels', htype = 'class_label', class_names = ds_test.labels.info.class_names)ds_test.model_evaluation.create_tensor('boxes', htype = 'bbox', coords = {'type': 'pixel', 'mode': 'LTWH'})ds_test.model_evaluation.create_tensor('iou')


Evaluating Model Performance on Training Data

Evaluating the performance of the model on a per-image basis can be a powerful tool for identifying bad or difficult data. First, we define a helper function that does a forward-pass through the model and computes the loss per image, without updating the weights. Since the model outputs the loss per batch, this functions requires that the batch size is 1.def evaluate_loss(model, data_loader, device): # This function assumes the data loader may be shuffled, and it returns the loss in a sorted fashion # using knowledge of the indices that are being trained in each batch. # Set the model to train mode in order to get the loss, even though we're not training. model.train() loss_list = [] indices_list = [] assert data_loader.batch_size == 1 start_time = time.time() for i, data in enumerate(data_loader): images = list(image.to(device) for image in data[0]) targets = [{k: v.to(device) for k, v in t.items()} for t in data[1]] indices = data[2] with torch.no_grad(): loss_dict = model(images, targets) losses = sum(loss for loss in loss_dict.values()) loss_value = losses.item() loss_list.append(loss_value) indices_list.append(indices) # Print performance statistics if i%100 ==0: batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] loss: %.3f, speed: %.2f' % (i, loss_value, speed)) loss_list = [x for _, x in sorted(zip(indices_list, loss_list))] return loss_listNext, let's create another PyTorch dataloader on the training dataset that is not shuffled, has a batch size of 1, uses the evaluation transform, and returns the indices of the current batch the dataloader using return_index= True:train_loader_eval = ds_train.pytorch(num_workers = 8, shuffle = False, transform = transform_val, tensors = ['images', 'labels', 'boxes'], batch_size = 1, collate_fn = collate_fn, return_index = True)Finally, we evaluate the loss for each image, write it back to the dataset, and add a commit to the training_run branch that we created at the start of this playbook:loss_per_image = evaluate_loss(model, train_loader_eval, device)with ds_train: ds_train.model_evaluation.loss.extend(loss_per_image) ds_train.commit('Trained the model and computed the loss for each image.')


Inspecting the Training Dataset based on Model Results

The dataset can be sorted based on loss in Activeloop Platform. An inspection of the high-loss images immediately reveals that many of them have poor quality or are incorrectly annotated.The sort feature in the video below was removed. To sort, please run the query:We can edit some of the bad data by deleting the incorrect annotation of '1' at index 14997 , and by removing the poor quality samples at indices 2899 and 32467. # Remove label '1' from 14997. It's in the first positions in the labels and boxes arraysds_train.labels[14997] = ds_train.labels[14997].numpy()[1:]ds_train.boxes[14997] = ds_train.boxes[14997].numpy()[1:,:]# Delete bad samplesds_train.pop(32467)ds_train.pop(2899)Lastly, we commit the edits in order to permanently store this snapshot of the data. ds_train.commit('Updated labels at index 14997 and deleted samples at 2899 and 32467')The next step would be perform a more exhaustive inspection of the high-loss data and make further improvements to the dataset, after which the model should be re-trained.


Training an Object Detection Model

Otherwise, it's on a CPU.device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')print(device)Let's initialize the model and optimizer:model = get_model_object_detection(NUM_CLASSES)model.to(device)# Specify the optimizerparams = [p for p in model.parameters() if p.requires_grad]optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)The model and data are ready for training 🚀!# Train the modelnum_epochs = 3lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.1) for epoch in range(num_epochs): # loop over the dataset multiple times print('------------------ Training Epoch {} ------------------'.format(epoch+1)) train_one_epoch(model, optimizer, train_loader, device) lr_scheduler.step() print('Finished Training')torch.save(model.state_dict(), 'model_weights_svhn_first_train.pth')


Evaluating Model Performance on Validation Data

After iterating on the training data re-training the model, a general assessment of model performance should be performed on validation data that was not used to train the model.  We create a helper function for running an inference of the model on the validation data that returns the model predictions and the average IOU (intersection-over-union) for each sample:# Run an inference of the model and compute the average IOU (intersection-over-union) for each sampledef evaluate_iou(model, data_loader, num_classes, device = 'cpu', score_thresh = 0.5): # This function removes predictions in the output and IUO calculation that are below a confidence threshold. # This function assumes the data loader may be shuffled, and it returns the loss in a sorted fashion # using knowledge of the indices that are being trained in each batch.  # Set the model to eval mode.  model.eval() ious_list = [] boxes_list = [] labels_list = [] indices_list = [] start_time = time.time() for i, data in enumerate(data_loader): images = list(image.to(device) for image in data[0]) ground_truths = [{k: v.to(device) for k, v in t.items()} for t in data[1]] indices = data[2] model_start = time.time() with torch.no_grad(): predictions = model(images) model_end = time.time() assert len(ground_truths) == len(predictions) == len(indices) # Check if data in dataloader is consistent for j, pred in enumerate(predictions): # Ignore boxes below the confidence threshold thresh_inds = pred['scores']>score_thresh pred_boxes = pred['boxes'][thresh_inds] pred_labels = pred['labels'][thresh_inds] pred_scores = pred['scores'][thresh_inds] # Find the union of prediceted and groud truth labels and iterate through it all_labels = np.union1d(pred_labels.to('cpu'), ground_truths[j]['labels'].to('cpu')) ious = np.zeros((len(all_labels))) for l, label in enumerate(all_labels): # Find the boxes corresponding to the label boxes_1 = pred_boxes[pred_labels == label] boxes_2 = ground_truths[j]['boxes'][ground_truths[j]['labels'] == label] iou = torchvision.ops.box_iou(boxes_1, boxes_2).cpu() # This method returns a matrix of the IOU of each box with every other box.  # Consider the IOU as the maximum overlap of a box with any other box.  Find the max along the axis that has the most boxes.  if 0 in iou.shape: ious[l] = 0 else: if boxes_1.shape>boxes_2.shape: max_iou, _ = iou.max(dim=0) else: max_iou, _ = iou.max(dim=1) # Compute the average iou for that label ious[l] = np.mean(np.array(max_iou)) #Take the average iou for all the labels.  If there are no labels, set the iou to 0


if len(ious)>0: ious_list.append(np.mean(ious)) else: ious_list.append(0) boxes_list.append(model_2_image(pred_boxes.cpu(), (HEIGHT, WIDTH), (data[3][j][0], data[3][j][1]))) # Convert the bounding box back to teh shape of the original image labels_list.append(np.array(pred_labels.cpu())) indices_list.append(indices[j]) # Print progress if i%100 ==0: batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] speed: %.2f' % (i, speed)) # Sort the data based on index, just in case shuffling was used in the dataloader ious_list = [x for _, x in sorted(zip(indices_list, ious_list))] boxes_list = [x for _, x in sorted(zip(indices_list, boxes_list))] labels_list = [x for _, x in sorted(zip(indices_list, labels_list))] return ious_list, boxes_list, labels_listLet's create a PyTorch dataloader using the validation data and run the inference using evaluate_iou above.val_loader = ds_test.pytorch(num_workers = 8, shuffle = False, transform = transform_val, tensors = ['images', 'labels', 'boxes'], batch_size = 16, collate_fn = collate_fn, return_index = True)iou_val, boxes_val, labels_val = evaluate_iou(model, val_loader, NUM_CLASSES, device, score_thresh = 0.5)Finally, we write the predictions back to the dataset and add a commit to the training_run branch that we created at the start of this playbook:with ds_test: ds_test.model_evaluation.labels.extend(labels_eval_test) ds_test.model_evaluation.boxes.extend(boxes_eval_test) ds_test.model_evaluation.iou.extend(iou_eval_test) ds_test.commit('Added model predictions.')


Comparing Model Results to Ground-Truth Annotations.

When sorting the model predictions based on IOU, we observe that the model successfully makes the correct predictions in images with one street number and where the street letters are large relative to the image. However, the model predictions are very poor for data with small street numbers, and there exist artifacts in the data where the model interprets vertical objects, such as narrow windows that the model thinks are the number '1'.The sort feature in the video below was removed. To sort, please run the query:Understanding the edge cases for which the model makes incorrect predictions is critical for improving the model performance. If the edge cases are irrelevant given the model's intended use, they should be eliminated from both the training and validation data. If they are applicable, more representative edge cases should be added to the training dataset, or the edge cases should be sampled more frequently while training.


Congratulations 🚀. You can now use Activeloop Deep Lake to evaluate the performance of your Deep-Learning models and compare their predictions to the ground-truth!


Cleanup and Reverting Mistakes in The Workflow

If you make a mistake you can use the following commands to start over or delete the new data:Delete data in a tensor: ds.<tensor_name>.clear()Delete the entire tensor and its data: ds.delete_tensor(<tensor_name>)Reset all edits since the prior commit: ds.reset()Delete the branch you just created: ds.delete_branch(<branch_name>)Must be on another branch, and deleted branch must not have been merged to another.


How to manage video datasets and train models using Deep Lake.

Performing deep-learning on video data can be challenging due to the large size of video files, especially when they are uncompressed to raw numeric data that is fed into neural networks. Deep Lake abstracts these challenges away from the user so you can focus on building performant models.


Setup

Make sure to install Deep Lake with pip install 'deeplake[av]' in order to use Deep Lake's audio and video features.import deeplakeds = deeplake.empty('demo/video') # create a local dataset


Adding annotations

See a complete example for this section in this notebook.Annotations like bounding boxes can be added and visualized in Deep Lake along with the video samples. We use tensors of htype sequence[bbox] for this purpose. Every sample in a sequence[bbox] tensor will be a sequence of bounding boxes which represents the annotations for the corresponding video sample in the video tensor.Learn more about sequences here.ds.create_tensor('boxes', htype='sequence[bbox]', coords={'type': 'pixel', 'mode': 'LTWH'})See this page for more details about the bbox htype.Next, here's an example of an annotations file taken from the LaSOT dataset. It contains annotations for every frame of a video.Example of an annotations fileWe convert this to a numpy array and append it to our boxes tensor.import pandas as pddf = pd.read_csv('groundtruth.txt', header=None)boxes = df.to_numpy().astype(np.float32)[:, np.newaxis]# boxes.shape == (2788, 1, 4) == (number of frames, number of boxes, 4)ds.boxes.append(boxes)Visualize the bounding boxes within your notebook using ds.visualize().The shapes of the samples in the video and sequence[bbox]tensors have to match in order for visualization to work properly.If the shape of video tensor is (# frames, height, width, 3), the shape of the sequence tensor should be(# frames, # of boxes in a frame, 4)Visualize video annotations in notebook


Shape

We can get the shape of a video sample in (N, H, W, C) format usingds.videos[0].shape(400, 360, 640, 3)


Creating a video tensor

To create a video tensor, we specify an htype of 'video' and set sample_compression to the format of the video.ds.create_tensor('videos', htype='video', sample_compression='mp4')


Adding video samples

We append videos to the newly created tensor by reading the video files with deeplake.readds.videos.append(deeplake.read('./videos/example1.mp4'))ds.videos.append(deeplake.read('./videos/example2.mp4')) can also read videos from http://, gcs:// and s3:// urls given you have the credentials to access them. Examples include:ds.videos.append( deeplake.read( 'http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4', creds=None, ))ds.videos.append( deeplake.read( 's3://bucket-name/sample_video.mp4', creds={ 'aws_access_key_id': '...', 'aws_secret_access_key': '...', 'aws_session_token': '...', }, ))See deeplake.read and check out this notebook to see this in action.


Sample info

Info about a video sample can be accessed usingds.videos[0].sample_infoThis returns info about the first sample as a dict:{ 'duration': 400400, 'fps': 29.97002997002997, 'timebase': 3.3333333333333335e-05, 'shape': [400, 360, 640, 3], 'format': 'mp4', 'filename': './videos/example1.mp4', 'modified': False} is in units oftimebase


Accessing video timestamps

Presentation timestamps (PTS) of frames can be obtained (in seconds) through a video tensor's .timestamp attribute after indexing it just like in the previous section:# timestamps of frames 10 - 15 of 0th sampleds.videos[0, 10:15].timestamparray([0.36703333, 0.4004 , 0.43376666, 0.46713334, 0.5005 ], dtype=float32)


Accessing video frames

The most important part of working with videos on Deep Lake is retrieving the frames of a video sample as a numpy array.video = ds.videos[0].numpy()This decompresses the entire first video sample and returns the frames as a numpy array.print(type(video))print(video.shape)<class 'numpy.ndarray'>(400, 360, 640, 3)Be careful when decompressing an entire large video sample because it can blow up your memory.Deep Lake allows you to index the video tensor like a numpy array and return the frames you want. Only the required frames are decompressed. See a few examples below:Getting a 100 frames from index 100 - 200# 1st sample, frames 100 - 200video = ds.videos[1, 100:200].numpy()video.shape(100, 360, 640, 3)Indexing with step# 0th sample, frames 100 - 200 with step of 5 framesvideo = ds.videos[0, 100:200:5].numpy()video.shape(20, 360, 640, 3)Getting a single frame# 1st sample, last framelast_frame = ds.videos[1, -1].numpy()last_frame.shape(360, 640, 3)


.data()

Calling ds.videos[index].data() will return a dict with keys 'frames' and 'timestamps' with the corresponding numpy arrays as values. Indexing works the same way as it does with .numpy().data = ds.videos[1, 15:20].data()data['frames'].shapedata['timestamps'](5, 360, 640, 3)array([0.5005 , 0.5672333 , 0.6006 , 0.6339667 , 0.76743335], dtype=float32)


.play()

Individual video samples can be instantly visualized by calling .play() on them:ds.videos[1].play()This will play the video on your web browser:video playback in browserOn a jupyter notebook this will look like:video playback on jupyter notebook


ds.visualize()

The whole Deep Lake dataset can be visualized by calling .visualize() on your dataset in a jupyter or colab notebook.ds.visualize()ds.visualize() on colabTry this out for yourself here!On colab, we only support visualizing hub://datasets


Linked videos

Tensors of Deep Lake type link[video] can be used to store links to videos. All of the above features are supported for linked videos. https://, gcs://, s3://and gdrive://links are accepted.# create linked tensorlinks = ds.create_tensor('video_links', htype='link[video]')# append linked sampleslinks.append(deeplake.link('http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4', creds_key=None)) # link to public video# .numpy()video = links[0].numpy()# shape of numpy arrayvideo.shape(360, 720, 1280, 3)You will need to set credentials to link to private data on your S3 or GCS.


For Activeloop cloud datasets

This proccess is easy and streamlined for deeplake:// datasets.First, go to your Activeloop platform, login and choose 'Managed credentials' in settings.And then choose 'Add Credentials'.Select a credentials provider, set the credentials name (say, 'MY_KEY'), fill the fields and save it.Done! Your credentials have now been set.


Add managed credentials to your dataset

Use ds.add_creds_key with managed set to True to add the credentials to your dataset. Multiple credentials can be added.ds.add_creds_key('MY_KEY', managed=True)ds.add_creds_key('S3_KEY', managed=True)


Use credentials

And when adding linked data using deeplake.link, simply mention which credentials to use through the creds_key argument.ds.links.append(deeplake.link('s3://my-bucket/sample-video.mp4', creds_key='MY_KEY'))


Video streaming

This section describes some implementation details regarding how video data is fetched and decompressed in Deep Lake.Large video samples (> 16MB by default) stored in remote Deep Lake datasets are not downloaded in their entirety on calling .numpy(). Instead, they are streamed from storage. Only the required packets are decompressed and converted to numpy arrays based on how the tensor is indexed.also streams videos from storage.


For non Activeloop cloud datasets

For non-hub:// datasets, you can use credentials set in your environment by mentioning creds_key='ENV'ds.links.append(deeplake.link('s3://my-bucket/sample-video.mp4', creds_key='ENV'))Or you can temporarily add creds to your datasetcreds={ 'aws_access_key_id': '...', 'aws_secret_access_key': '...', 'aws_session_token': '...', }# add creds key (Note that managed is False)ds.add_creds_key('TEMP_KEY')# populate creds with a credentials dictds.populate_creds('TEMP_KEY', creds)and thends.links.append(deeplake.link('s3://my-bucket/sample-video.mp4', creds_key='TEMP_KEY'))See deeplake.link


How to Implement External Locks using Zookeeper

Apache Zookeeper is a tool that can be used to manage Deep Lake locks and ensure that only 1 worker is writing to a Deep Lake dataset at a time. It offers a simple API for managing locks using a few lines of code.


Setup

First, let's install Zookeper and launch a local server using Docker in the CLI.pip install zookeeperdocker run --rm -p 2181:2181 zookeeper


Write Locks

All write operations should be executed while respecting the lock.Let's connect a Python client to the local server and create a WriteLock using:from kazoo.client import KazooClientzk = KazooClient(hosts='127.0.0.1:2181')zk.start()deeplake_writelock = zk.WriteLock('/deeplake')The client can be blocked from performing operations without a WriteLock using the code below. The code will wait until the lock becomes available, and the internal Deep Lake lock should be disabled by specifying lock_enabled=False:from deeplake.core.vectorstore import VectorStorewith deeplake_writelock: # Initialize the Vector Store vector_store = VectorStore(<vector_store_path>, lock_enabled=False) # Add data vector_store.add(text = <your_text>, metadata = <your_metadata>, embedding_function = <your_embedding_function>) # This code can also be used with the Deep Lake LangChain Integration # from langchain.vectorstores import DeepLake # db = DeepLake(<dataset_path>, embedding = <your_embedding_function>) # db.add_texts(tests = <your_texts>, metadatas = <your_metadatas>, ...) # This code can also be used with the low-level Deep Lake API # import deeplake # ds = deeplake.load(dataset_path) # ds.append({...})


When Writes Update and Delete Data

If the write operations are updating or deleting rows of data, the read operations should also lock the dataset in order to avoid corrupted read operations. Let's connect a Python client to the same local server above and create a ReadLock . Multiple clients can have a ReadLock without blocking each other, but they will all be blocked by the WriteLock above.from kazoo.client import KazooClientzk = KazooClient(hosts='127.0.0.1:2181')zk.start()deeplake_readlock = zk.ReadLock('/deeplake')The syntax for restricting operations using the ReadLock is:from deeplake.core.vectorstore import VectorStorewith deeplake_readlock: # Initialize the Vector Store vector_store = VectorStore(<vector_store_path>, read_only = True) # Search for data search_results = vector_store.search(embedding_data = <your_prompt>, embedding_function = <your_embedding_function>) # This code can also be used with the Deep Lake LangChain Integration # from langchain.vectorstores import DeepLake # db = DeepLake(<dataset_path>, embedding = <your_embedding_function>, read_only = True) # retriever = db.as_retriever() # qa = RetrievalQA.from_llm(llm = <your_model>, retriever = retriever) # This code can also be used with the low-level Deep Lake API # import deeplake # ds = deeplake.load(<dataset_path>, read_only = True) # dataloader = ds.dataloader().pytorch(...)Congrats! You just learned how manage your own lock for Deep Lake using Zookeeper! 🎉


When Writes are Append-Only

If the write operations are only appending data, it is not necessary to use locks during read operations like as vector search. However, the Deep Lake datasets must be reloaded or re-initialized in order to have the latest available information from the write operations. from deeplake.core.vectorstore import VectorStore# Initialize the Vector Store vector_store = VectorStore(<vector_store_path>, read_only = True)# Search for datasearch_results = vector_store.search(embedding_data = <your_prompt>, embedding_function = <your_embedding_function>)# This code can also be used with the Deep Lake LangChain Integration# from langchain.vectorstores import DeepLake# db = DeepLake(<dataset_path>, embedding = <your_embedding_function>, read_only = True)# retriever = db.as_retriever()# qa = RetrievalQA.from_llm(llm = <your_model>, retriever = retriever)# This code can also be used with the low-level Deep Lake API# import deeplake# ds = deeplake.load(<dataset_path>, read_only = True)# dataloader = ds.dataloader().pytorch(...)


How Deep Lake Datasets are Synchronized with Long-Term Storage

Using with context when updating Deep Lake datasets is critical for achieving rapid write performance.


BAD PRACTICE - Code without with context

Any standalone update to a Deep Lake dataset is immediately pushed to the dataset's long-term storage location. Due to the high number of write operations, there may be a significant increase in runtime when the data is stored in the cloud. In the example below, an update is pushed to storage for every call to the .append() command.for i in range(10): ds.my_tensor.append(i)


Code using with context

To increase write speeds when using Deep Lake, the with syntax significantly improves performance because it only pushes updates to long-term storage after the code block inside the with statement has been executed, or when the local cache is full. This significantly reduces the number of discreet write operations, thereby increasing the speed by up to 100X. with ds: for i in range(10): ds.my_tensor.append(i)


How to create Deep Lake datasets at scale

This workflow assumes the reader has experience uploading datasets using Deep Lake's distributed framework deeplake.compute.


When creating large Deep Lake datasets, it is recommended to:

Parallelize the ingestion using deeplake.compute with a large num_workers (8-32)Use checkpointing to periodically auto-commit data using .eval(... checkpoint_interval = <commit_every_N_samples>)If there is an error during the data ingestion, the dataset is automatically reset to the last auto-commit with valid data.Additional recommendations are:If upload errors are intermittent and error-causing samples may be skipped (like bad links), you can run .eval(... ignore_errors=True).When uploading linked data, if a data integrity check is not necessary, and if querying based on shape information is not important, you can increase the upload speed by 10-100X by setting the following parameters to False when creating the linked tensor: verify, create_shape_tensor , create_sample_info_tensorWe highly recommend performing integrity checks for linked data during dataset creation, even though it slows data ingestion. This one-time check will significantly reduce debugging during querying, training, or other workflows.


Example Dataset Creation Using Checkpointing

The dataset has a total of ~118000 samples.coco_2_deeplake(coco_api = coco, bucket = coco_bucket, category_names = category_names, category_info = category_info, creds_key = creds_key).eval(img_ids, ds, num_workers = 8, checkpoint_interval=25000)After the upload is complete, we see commits like the one below in ds.log().Commit : firstdbf9474d461a19e9333c2fd19b46115348f (main) Author : <username>Time : 2023-03-27 19:18:14Message: Auto-commit during deeplake.compute of coco_2_deeplake after 20.0% progressTotal samples processed in transform: 25000If an upload error occurs but the script completes, the dataset will be reset to the prior checkpoint and you will see a message such as:If the script does not complete due to a system failure or keyboard interrupt, you should load the dataset and run ds.reset(), or load the dataset using ds = deeplake.load(.. .  reset = True).  This will restore the dataset to the prior checkpoint.  You may find how many samples were successfully processed using:len(ds) -> length of the shortest tensords.max_len -> length of the longest tensords.log() -> Prints how many samples were processed by the checkpointing


In this example we upload the COCO dataset originally stored as an S3 bucket to a Deep Lake dataset stored in another S3 bucket.  The images are uploaded as links and the annotations (categories, masks, bounding boxes) are stored in the Deep Lake dataset.  Annotations such as pose keypoints or supercategories are omitted.import deeplakeimport numpy as npimport boto3import osfrom pycocotools.coco import COCOimport getpassFirst, let's define the S3 buckets where the source COCO data is stored, and where the Deep Lake dataset will be stored.  Let's also connect to the source data via boto3 and define a credentials dictionary (on some systems credentials, can be automatically pulled from the environment).coco_bucket = <bucket_containing_the_source_data>deeplake_bucket = <bucket_for_storing_the_deep_lake_dataset>creds = {'aws_access_key_id': os.environ.get('aws_access_key_id'), 'aws_secret_access_key': os.environ.get('aws_secret_access_key')}# Create the connection to the source datas3 = boto3.resource('s3', aws_access_key_id = creds['aws_access_key_id'], aws_secret_access_key = creds['aws_secret_access_key'])s3_bucket = s3.Bucket(coco_bucket)The annotations are downloaded locally for simplifying the upload code, since the COCO API was designed to read the annotations from a local file.cloud_ann_path = 'coco/annotations/instances_train2017.json'local_ann_path = 'anns_train.json's3_bucket.download_file(ann_path, local_ann_path)coco = COCO(local_ann_path)category_info = coco.loadCats(coco.getCatIds())Next, let's create an empty Deep Lake dataset at the desired path and connect it to the Deep Lake backend.  We also add managed credentials for accessing linked data.  In this case, the managed credentials for accessing the dataset are the same as those for accessing the linked data, but that's not a general requirement.  More details on managed credentials are available here.  ds = deeplake.empty('s3://{}/coco-train'.format(deeplake_bucket), creds = creds, overwrite = True)creds_key = <managed_creds_key>ds.connect(org_id = <org_id>, creds_key = creds_key, token = <your_token>)ds.add_creds_key(creds_key, managed = True)Next, we define the list category_names that maps the numerical annotations to the index in this list.  If label annotations are uploaded as text (which is not the case here), the list is auto-populated.  We pass category_names to the class_names parameter during tensor creation, though it can also be updated later, or omitted entirely if the numerical labels are sufficient.category_names = [category['name'] for category in category_info]with ds: ds.create_tensor('images', htype = 'link[image]', sample_compression = 'jpg') ds.create_tensor('categories', htype = 'class_label', class_names = category_names) ds.create_tensor('boxes', htype = 'bbox') ds.create_tensor('masks', htype = 'binary_mask', sample_compression = 'lz4')Next, we define the input iterable and deepake.compute function.  The elements in the iterable are parallelized among the workers during the execution of the function.img_ids = sorted(coco.getImgIds())@deeplake.computedef coco_2_deeplake(img_id, sample_out, coco_api, category_names, category_info, bucket, creds_key): anns = coco_api.loadAnns(coco_api.getAnnIds(img_id)) img_coco = coco_api.loadImgs(img_id)[0] # First create empty arrays for all annotations categories = np.zeros((len(anns)), dtype = np.uint32) boxes = np.zeros((len(anns),4), dtype = np.float32) masks = np.zeros((img_coco['height'], img_coco['width'], len(anns)), dtype = bool) # Then populate the arrays with the annotations data for i, ann in enumerate(anns): mask = coco.annToMask(ann) # Convert annotation to binary mask masks[:, :, i] = mask boxes[i,:] = ann['bbox'] # Find the deep lake category_names index from the coco category_id categories[i] = category_names.index([category_info[i]['name'] for i in range(len(category_info)) if category_info[i]['id']==ann['category_id']][0]) # Append the data to a deeplake sample sample_out.append({'images': deeplake.link('s3://{}/coco/train2017/{}'.format(bucket, img_coco['file_name']), creds_key = creds_key), 'categories': categories, 'boxes': boxes, 'masks': masks})Finally, execute the deeplake.compute function and set checkpoint_interval to 25000


How to restore a corrupted Deep Lake dataset

Deliberate of accidental interruption of code may make a Deep Lake dataset or some of its tensors unreadable. At scale, code interruption is more likely to occur, and Deep Lake's version control is the primary tool for recovery.


How to Use Version Control to Retrieve Data

When manipulating Deep Lake datasets, it is recommended to commit periodically in order to create snapshots of the dataset that can be accessed later. This can be done automatically when creating datasets with deeplake.compute, or manually using our version control API.If a dataset becomes corrupted, when loading the dataset, you may see an error like:DatasetCorruptError: Exception occured (see Traceback). The dataset maybe corrupted. Try using `reset=True` to reset HEAD changes and load the previous commit. This will delete all uncommitted changes on the branch you are trying to load.To reset the uncommitted corrupted changes, load the dataset with the reset = True flag:ds = deeplake.load(<dataset_path>, reset = True)Note: this operation deletes all uncommitted changes.


How to Use Version Control in Deep Lake

Deep Lake dataset version control allows you to manage changes to datasets with commands very similar to Git. It provides critical insights into how your data is evolving, and it works with datasets of any size!Let's check out how dataset version control works in Deep Lake! If you haven't done so already, please download and unzip the animals dataset from Step 2. First let's create a Deep Lake dataset in the ./version_control_deeplake folder.import deeplakeimport numpy as npfrom PIL import Image# Set overwrite = True for re-runabilityds = deeplake.dataset('./version_control_deeplake', overwrite = True)# Create a tensor and add an imagewith ds: ds.create_tensor('images', htype = 'image', sample_compression = 'jpeg') ds.images.append(deeplake.read('./animals/cats/image_1.jpg'))The first image in this dataset is a picture of a cat:Image.fromarray(ds.images[0].numpy())


Commit

To commit the data added above, simply run ds.commit:first_commit_id = ds.commit('Added image of a cat')print('Dataset in commit {} has {} samples'.format(first_commit_id, len(ds)))Next, let's add another image and commit the update:with ds: ds.images.append(deeplake.read('./animals/dogs/image_3.jpg')) second_commit_id = ds.commit('Added an image of a dog')print('Dataset in commit {} has {} samples'.format(second_commit_id, len(ds)))The second image in this dataset is a picture of a dog: Image.fromarray(ds.images[1].numpy())


Log

The commit history starting from the current commit can be show using ds.log:log = ds.log()This command prints the log to the console and also assigns it to the specified variable log. The author of the commit is the username of the Activeloop account that logged in on the machine.


Branch

Branching takes place by running the ds.checkout command with the parameter create = True . Let's create a new branch dog_flipped, flip the second image (dog), and create a new commit on that branch.ds.checkout('dog_flipped', create = True)with ds: ds.images[1] = np.transpose(ds.images[1], axes=[1,0,2])flipped_commit_id = ds.commit('Flipped the dog image')The dog image is now flipped and the log shows a commit on the dog_flipped branch as well as the previous commits on main: Image.fromarray(ds.images[1].numpy())ds.log()


Checkout

A previous commit of the branch can be checked out using ds.checkout:ds.checkout('main')Image.fromarray(ds.images[1].numpy())As expected, the dog image on main is not flipped.


Diff

Understanding changes between commits is critical for managing the evolution of datasets. Deep Lake's ds.diff function enables users to determine the number of samples that were added, removed, or updated for each tensor. The function can be used in 3 ways:ds.diff() # Diff between the current state and the last commitds.diff(commit_id) # Diff between the current state and a specific commitds.diff(commit_id_1, commit_id_2) # Diff between two specific commits


Let's see how this works:

You should currently be on the main branch, which has 2 samples. You can check for uncommited changes using:ds.has_head_changesLet's add another image:print('Dataset on {} branch has {} samples'.format('main', len(ds)))with ds: ds.images.append(deeplake.read('./animals/dogs/image_4.jpg')) print('After updating, the HEAD commit on {} branch has {} samples'.format('main', len(ds)))The 3rd sample is also an image of a dog:Image.fromarray(ds.images[2].numpy())Next, if you checkout dog_flipped branch, the dataset contains 2 samples, which is sample count from when that branch was created. Therefore, the additional uncommitted third sample that was added to the main branch above is not reflected when other branches or commits are checked out.ds.checkout('dog_flipped')print('Dataset in {} branch has {} samples'.format('dog_flipped', len(ds)))Finally, when checking our the main branch again, the prior uncommitted changes and available and they are stored in the HEAD commit on main:ds.checkout('main')print('Dataset in {} branch has {} samples'.format('main', len(ds)))The dataset now contains 3 samples and the uncommitted dog image is visible:Image.fromarray(ds.images[2].numpy())You can delete any uncommitted changes using the reset command below, which will bring the main branch back to the state with 2 samples.ds.reset()print('Dataset in {} branch has {} samples'.format('main', len(ds)))


HEAD Commit

Unlike Git, Deep Lake's dataset version control does not have a local staging area because all dataset updates are immediately synced with the permanent storage location (cloud or local). Therefore, any changes to a dataset are automatically stored in a HEAD commit on the current branch. This means that the uncommitted changes do not appear on other branches, and uncommitted changes are visible to all users.


Merge

Merging is a critical feature for collaborating on datasets. It enables you to modify data on separate branches before making those changes available on the main branch, thus enabling you to experiment on your data without affecting workflows by other collaborators.We are currently on the main branch where the picture of the dog is right-side-up.ds.log()Image.fromarray(ds.images[1].numpy())We can merge the dog_flipped branch into main using the command below:ds.merge('dog_flipped')After merging the dog_flipped branch, we observe that the image of the dog is flipped. The dataset log now has a commit indicating that a commit from another branch was merged to main.Image.fromarray(ds.images[1].numpy())ds.log()Congrats! You just are now an expert in dataset version control! 🎓


How to Visualize Datasets in Deep Lake

One of Deep Lake's core features is to enable users to visualize and interpret large amounts of data. Let's load the COCO dataset, which is one of the most popular datasets in computer vision.import deeplakeds = deeplake.load('hub://activeloop/coco-train')The tensor layout for this dataset can be inspected using:ds.summary()The dataset can be visualized in the Activeloop UI or using an iframe in a jupyter notebook. If you don't already have flask and ipython installed, make sure to install Deep Lake using pip install deeplake[visualizer].ds.visualize()Visualizing datasets in Activeloop Platform will unlock more features and faster performance compared to visualization in Jupyter notebooks.


Visualizing your own datasets

Any Deep Lake dataset can be visualized using the methods above as long as it follows the conventions necessary for the visualization engine to interpret and parse the data. These conventions are explained in the link below:Dataset Visualization


How to use Deeplake with PyTorch or TensorFlow in Python

Deep Lake Datasets can be connected to popular ML frameworks such as PyTorch and TensorFlow, so you can train models while streaming data from the cloud without bottlenecking the training process!Data Streaming using Deep Lake


Training models with PyTorch

There are two syntaxes that can be used to train models in Pytorch using Deep Lake datasets:1.Deep Lake Data Loaders are highly-optimized and unlock the fastest streaming and shuffling using Deep Lake's internal shuffling method. However, they do not support custom sampling or fully-random shuffling that is possible using PyTorch datasets + data loaders.2.Pytorch Datasets + PyTorch Data Loaders enable all the customizability supported by PyTorch. However, they have highly sub-optimal streaming using Deep Lake datasets and may result in 5X+ slower performance compared to using Deep Lake data loaders.


1. Deep Lake Data Loaders for PyTorch

Best option for fast streaming!The fastest streaming of data to GPUs using PyTorch is achieved using Deep Lake's built-in PyTorch dataloader ds.pytorch() . If your model training is highly sensitive to the randomization of the input data, please pre-shuffle the data, or explore our writeup onShuffling in dataloaders.import deeplakefrom torchvision import datasets, transforms, modelsds = deeplake.load('hub://activeloop/cifar100-train') # Deep Lake Dataset


Transform syntax #2 - For complex or dependent transforms per tensor

Sometimes a single transformation function might need to be applied to all tensors, or tensors need to be combined in a transform. In this case, you can use the syntax below to perform the exact same transform as above:def transform(sample_in): return {'images': tform(sample_in['images']), 'labels': sample_in['labels']}#PyTorch Dataloaderdataloader= ds.pytorch(batch_size = 16, num_workers = 2, transform = transform, tensors = ['images', 'labels'], shuffle = True)Some datasets such as ImageNet contain both grayscale and color images, which can cause errors when the transformed images are passed to the model. To convert only the grayscale images to color format, you can add this Torchvision transform to your pipeline:


Transform syntax #1 - For independent transforms per tensor

The transform parameter in ds.pytorch() is a dictionary where the key is the tensor name and the value is the transformation function for that tensor. If a tensor does not need to be returned, the tensor should be omitted from the keys. If no transformation is necessary on a tensor, the transformation function is set as None.tform = transforms.Compose([ transforms.ToPILImage(), # Must convert to PIL image for subsequent operations to run transforms.RandomRotation(20), # Image augmentation transforms.ToTensor(), # Must convert to pytorch tensor for subsequent operations to run transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),])#PyTorch Dataloaderdataloader= ds.pytorch(batch_size = 16, num_workers = 2, transform = {'images': tform, 'labels': None}, shuffle = True)


Training models with TensorFlow

Deep Lake Datasets can be converted to TensorFlow Datasets using ds.tensorflow(). Downstream, functions from the tf.Data API such as map, shuffle, etc. can be applied to process the data before training.ds # Deep Lake Dataset object, to be used for trainingds_tf = ds.tensorflow() # A TensorFlow Dataset


2. PyTorch Datasets + PyTorch Data Loaders using Deep Lake

Best option for full customizability.Deep Lake datasets can be integrated in the PyTorch Dataset class by passing the ds object to the PyTorch Dataset's constructor and pulling data in the __getitem__ method using self.ds.image[ids].numpy():from torch.utils.data import DataLoader, Datasetclass ClassificationDataset(Dataset): def __init__(self, ds, transform = None): self.ds = ds self.transform = transform def __len__(self): return len(self.ds) def __getitem__(self, idx): image = self.ds.images[idx].numpy() label = self.ds.labels[idx].numpy(fetch_chunks = True).astype(np.int32) if self.transform is not None: image = self.transform(image) sample = {'images': image, 'labels': label} return sampleWhen loading data sequentially, or when randomly loading samples from a tensor that fits into the cache (such as class_labels) it is recommended to set fetch_chunks = True. This increases the data loading speed by avoiding separate requests for each individual sample. This is not recommended when randomly loading large tensors, because the data is deleted from the cache before adjacent samples from a chunk are used.The PyTorch dataset + data loader is instantiated using the built-in PyTorch functions:cifar100_pytorch = ClassificationDataset(ds_train, transform = tform)dataloader_pytroch = DataLoader(dataset_pt, batch_size = 16, num_workers = 2, shuffle = True)


Iteration and Training

You can iterate through both data loaders above using the exact same syntax. Loading the first batch of data using the Deep Lake data loader may take up to 30 seconds because the shuffle buffer is filled before any data is returned.for data in dataloader: print(data) break # Training Loopfor data in dataloader_pytorch: print(data) break # Training LoopFor more information on training, check out the tutorial on Training an Image Classification Model in PyTorch


How to Use Deep Lake as a Vector Store for Images

Deep Lake is a unique Vector Store because it supports storage of various data types including images, video, and audio. In this tutorial we show how to use Deep Lake to perform similarity search for images.


Similarity Search

Let's perform a similarity search on a reference image to find similar images in our Vector Store. First we download the image:The similarity search will return data for the top k (defaults to 4) similar samples, including numpy arrays for the underlying images.image_path = '/image_similarity.jpg'result = vector_store.search(embedding_data = image_path, embedding_function = embedding_function)The key-value pairs in the result contains the tensor as the key and a list of values for the data:result.keys() # Returns: dict_keys(['filename', 'id', 'image', 'score'])len(result['score']) # Returns: 4result['image'][0].shape# Returns: (427, 640, 3)Since images can be quite large, and we may not want to return them as numpy arrays, so we use return_tensors to specify that only the filename and id tensors should be returned:result = vector_store.search(embedding_data = image_path, embedding_function = embedding_function, return_tensors = ['id', 'filename'])result.keys() # Returns: dict_keys(['filename', 'id', 'image', 'score'])


Creating the Vector Store

We will use ~5k images in the COCO Validation Dataset as a source of diverse images. First, let's download the data.!wget -O '<download_path>' http://images.cocodataset.org/zips/val2017.zip# MAC !curl -o '<download_path>' http://images.cocodataset.org/zips/val2017.zipWe must unzip the images and specify their parent folder below.images_path = <download_path>Next, let's define a ResNet18 PyTorch model to embed the images based on the output from the second-to-last layer. We use the torchvision feature extractor to return the output of the avgpool layer to the embedding key, and we run on a GPU if available. (Note: DeepLakeVectorStore class was deprecated, but you can still use it. The new API for calling Deep Lake's Vector Store is: VectorStore.)from deeplake.core.vectorstore.deeplake_vectorstore import VectorStoreimport osimport torchfrom torchvision import transforms, modelsfrom torchvision.models.feature_extraction import create_feature_extractorfrom PIL import Imagedevice = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')model = models.resnet18(pretrained=True)return_nodes = { 'avgpool': 'embedding'}model = create_feature_extractor(model, return_nodes=return_nodes)model.eval()model.to(device)Let's define an embedding function that will embed a list of image filenames and return a list of embeddings. A transformation must be applied to the images so they can be fed into the model, including handling of grayscale images.tform= transforms.Compose([ transforms.Resize((224,224)), transforms.ToTensor(), transforms.Lambda(lambda x: torch.cat([x, x, x], dim=0) if x.shape[0] == 1 else x), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),])def embedding_function(images, model = model, transform = tform, batch_size = 4): '''Creates a list of embeddings based on a list of image filenames. Images are processed in batches.''' if isinstance(images, str): images = [images] #Proceess the embeddings in batches, but return everything as a single list embeddings = [] for i in range(0, len(images), batch_size): batch = torch.stack([transform(Image.open(item)) for item in images[i:i+batch_size]]) batch = batch.to(device) with torch.no_grad(): embeddings+= model(batch)['embedding'][:,:,0,0].cpu().numpy().tolist() return embeddingsNow we can create the vector store for storing the data. The Vector Store does not have the default configuration with text, embedding, and metadata tensors, so we use the tensor_params input to define the structure of the Vector Store. vector_store_path = 'hub://<org_id>/<dataset_name>'vector_store = VectorStore( path = vector_store_path, tensor_params = [{'name': 'image', 'htype': 'image', 'sample_compression': 'jpg'}, {'name': 'embedding', 'htype': 'embedding'}, {'name': 'filename', 'htype': 'text'}],)Finally, we can create a list of images from the source data and add it to the vector store. image_fns = [os.path.join(images_path, item) for item in os.listdir(images_path) if os.path.splitext(item)[-1]=='.jpg']vector_store.add(image = image_fns, filename = image_fns, embedding_function = embedding_function, embedding_data = image_fns)We observe in the automatically printed summary that the Vector Store has tensors for the image, their filename, their embedding, and an id, with 5000 samples each. This summary is also available via vector_store.summary(). tensor htype shape dtype compression ------- ------- ------- ------- ------- embedding embedding (5000, 512) float32 None filename text (5000, 1) str None id text (5000, 1) str None image image (5000, 145:640, 200:640, 1:3) uint8 jpeg


Visualizing the Similarity Results

Instead of returning the results of the similarity search directly, we can use return_view = True to get the Deep Lake dataset view, which is a lazy pointer to the underlying data that satisfies the similarity search (no data is retrieved locally).view = vector_store.search(embedding_data = image_path, embedding_function = embedding_function, return_view = True)We can then save the view and visualize it in the Deep Lake UI:view.save_view()The images are all fairly similar to the reference image, so it looks like the similarity search worked well!Congrats! You just used the Deep Lake VectorStore in for image similarity search! 🎉


This tutorial is also available as a Colab Notebook

Deep Memory computes a transformation that converts your embeddings into an embedding space that is tailored for your use case, based on several examples for which the most relevant embedding is known. This can increase the accuracy of your Vector Search by up to 22%.In this example, we'll use Deep Memory to improve the accuracy of Vector Search on the SciFact dataset, where the input prompt is a scientific claim, and the search result is the corresponding abstract.


Downloading the Data

First let's specify out Activeloop and OpenAI tokens. Make sure to install pip install datasets because we'll download the source data from HuggingFace.from deeplake import VectorStoreimport osimport getpassimport datasetsimport openaifrom pathlib import Pathos.environ['OPENAI_API_KEY'] = getpass.getpass()# Skip this step if you logged in through the CLIos.environ['ACTIVELOOP_TOKEN'] = getpass.getpass()Next, let's download the dataset locally:corpus = datasets.load_dataset('scifact', 'corpus')


Creating the Vector Store

Now let's define an embedding function for the text data and create a Deep Lake Vector Store in our Managed Database. Deep Memory is only available for Vector Stores in our Managed Database.def embedding_function(texts, model='text-embedding-ada-002'): if isinstance(texts, str): texts = [texts] texts = [t.replace('\n', ' ') for t in texts] return [data['embedding']for data in openai.Embedding.create(input = texts, model=model)['data']]path = 'hub://<org_id>/<vector_store_name>'vectorstore = VectorStore( path=path, embedding_function=embedding_function, runtime={'tensor_db': True},)


Adding data to the Vector Store

Next, let's extract the data from the SciFact dataset and add it to our Vector Store. In this example, we embed the abstracts of the scientific papers. Normally, the id tensor is auto-populated, but in this case, we want to use the ids in the SciFact dataset, in order to use the internal connection between ids, abstracts, and claims, that already exists in SciFact.ids = [f'{id_}' for id_ in corpus['train']['doc_id']]texts = [text[0] for text in corpus['train']['abstract']]metadata = [{'title': title} for title in corpus['train']['title']]vectorstore.add( text=texts, id=ids, embedding_data=texts, embedding_function=embedding_function, metadata=metadata,)


Generating claims

We must create a relationship between the claims and their corresponding most relevant abstracts. This correspondence already exists in the SciFact dataset, and we extract that information using the helper function below.def preprocess_scifact(claims_dataset, dataset_type='train'): # Using a dictionary to store unique claims and their associated relevances claims_dict = {} for item in claims_dataset[dataset_type]: claim = item['claim'] relevance = (item['evidence_doc_id'], 1) # 1 indicates that the evidence is relevant to the question # Check for non-empty relevance if relevance[0] != '': if claim not in claims_dict: claims_dict[claim] = [relevance] else: # If the does not exist in the dictionary, append the new relevance if relevance not in claims_dict[claim]: claims_dict[claim].append(relevance) # Split the dictionary into two lists: claims and relevances claims = list(claims_dict.keys()) relevances = list(claims_dict.values()) return claims, relevancesclaims_dataset = datasets.load_dataset('scifact', 'claims')claims, relevances = preprocess_scifact(claims_dataset, dataset_type='train')Let's print the first 10 claims and their relevant abstracts. The relevances are a list of tuples, where each the id corresponds to the id tensor value in the Abstracts Vector Store, and 1 indicates a positive relevance.claims[:10]['1 in 5 million in UK have abnormal PrP positivity.', '32% of liver transplantation programs required patients to discontinue methadone treatment in 2001.', '40mg/day dosage of folic acid and 2mg/day dosage of vitamin B12 does not affect chronic kidney disease (CKD) progression.', '76-85% of people with severe mental disorder receive no treatment in low and middle income countries.', 'A T helper 2 cell (Th2) environment impedes disease development in patients with systemic lupus erythematosus (SLE).', 'A breast cancer patient's capacity to metabolize tamoxifen influences treatment outcome.', 'A country's Vaccine Alliance (GAVI) eligibility is not indictivate of accelerated adoption of the Hub vaccine.', 'A deficiency of folate increases blood levels of homocysteine.', 'A diminished ovarian reserve does not solely indicate infertility in an a priori non-infertile population.', 'A diminished ovarian reserve is a very strong indicator of infertility, even in an a priori non-infertile population.']relevances[:10][[('13734012', 1)], [('44265107', 1)], [('33409100', 1)], [('6490571', 1)], [('12670680', 1)], [('24341590', 1)], [('12428497', 1)], [('11705328', 1)], [('13497630', 1)], [('13497630', 1)]]