Deep Lake as a Vector Store for LLM Applications

Store and search embeddings and their metadata including text, jsons, images, audio, video, and more. Save the data locally, in your cloud, or on Deep Lake storage.Build LLM Apps using or integrations with LangChain and LlamaIndexRun computations locally or on our Managed Tensor Database


Deep Lake as a Data Lake For Deep Learning

Store images, audios, videos, text and their metadata (i.e. annotations) in a data format optimized for Deep Learning. Save the data locally, in your cloud, or on Activeloop storage.Rapidly train PyTorch and TensorFlow models while streaming data with no boilerplate code.Run version control, dataset queries, and distributed workloads using a simple Python API.Deep Lake Architecture for Inference and Model Development Applications.


To start using Deep Lake ASAP, check out our Vector Store Quickstart, Deep Learning Quickstart, Getting Started Guides, Tutorials, and Playbooks.

Please check out Deep Lake's GitHub repository and give us a ⭐ if you like the project. Join our Slack Community if you need help or have suggestions for improving documentation!


Deep Lake Docs Overview

Vector Store QuickstartDeep Learning QuickstartStorage & CredentialsGetting StartedTutorials (w Colab)PlaybooksDataset VisualizationBest PracticesLow-Level API Summary


Deep Lake API Reference

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Vector Store Creating a Deep Lake Vector Store Vector Store Operations Vector Store Properties VectorStore.DeepMemory Creating a Deep Memory Deep Memory Operations Deep Memory Properties Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype 3D Bounding Box Htype Intrinsics Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Nifti Htype Point Cloud Htype Mesh Htype Embedding Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read MMDetection High-Performance Features Dataloader Sampler Tensor Query Language Random Split Deep Memory API Reference deeplake deeplake.VectorStore deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log deeplake.core.transform deeplake.core.vectorstore.deep_memory deeplake.random.seed


Indices and tables

Index Module Index Search Page


How to Get Started with Vector Search in Deep Lake in Under 5 Minutes

If you prefer to use Deep Lake with LangChain, check out this tutorial. This quickstart focuses on vector storage and search, instead of end-2-end LLM apps, and it offers more customization and search options compared to the LangChain integration.


Installing Deep Lake

Deep Lake can be installed using pip. By default, Deep Lake does not install dependencies for the compute engine, google-cloud, and other features. Details on all installation options are available here. This quickstart also requires OpenAI.!pip3 install deeplake!pip3 install openai


Performing Vector Search

Deep Lake offers highly-flexible vector search and hybrid search options discussed in detail in these tutorials. In this Quickstart, we show a simple example of vector search using default options, which performs cosine similarity search in Python on the client. prompt = 'What are the first programs he tried writing?'search_results = vector_store.search(embedding_data=prompt, embedding_function=embedding_function)The search_results is a dictionary with keys for the text, score, id, and metadata, with data ordered by score. If we examine the first returned text using search_results['text'][0], it appears to contain the answer to the prompt.What I Worked OnFebruary 2021Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.The first programs I tried writing were on the IBM 1401 that our school district used for what was then called 'data processing.' This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in


Creating Your First Vector Store

Let's embed and store one of Paul Graham's essays in a Deep Lake Vector Store stored locally. First, we download the data:Next, let's import the required modules and set the OpenAI environmental variables for embeddings:from deeplake.core.vectorstore import VectorStoreimport openaiimport osos.environ['OPENAI_API_KEY'] = <OPENAI_API_KEY>Next, lets specify paths for the source text and the Deep Lake Vector Store. Though we store the Vector Store locally, Deep Lake Vectors Stores can also be created in memory, in the Deep Lake Managed Tensor Database, or in your cloud. Further details on storage options are available here. Let's also read and chunk the essay text based on a constant number of characters. source_text = 'paul_graham_essay.txt'vector_store_path = 'pg_essay_deeplake'with open(source_text, 'r') as f: text = f.read()CHUNK_SIZE = 1000chunked_text = [text[i:i+1000] for i in range(0,len(text), CHUNK_SIZE)]Next, let's define an embedding function using OpenAI. It must work for a single string and a list of strings, so that it can both be used to embed a prompt and a batch of texts. def embedding_function(texts, model='text-embedding-ada-002'): if isinstance(texts, str): texts = [texts] texts = [t.replace('\n', ' ') for t in texts] return [data.embedding for data in openai.embeddings.create(input = texts, model=model).data]Finally, let's create the Deep Lake Vector Store and populate it with data. We use a default tensor configuration, which creates tensors with text (str), metadata(json), id (str, auto-populated), embedding (float32). Learn more about tensor customizability here. vector_store = VectorStore( path = vector_store_path,)vector_store.add(text = chunked_text, embedding_function = embedding_function, embedding_data = chunked_text, metadata = [{'source': source_text}]*len(chunked_text))The path parameter is bi-directional:When a new path is specified, a new Vector Store is createdWhen an existing path is specified, the existing Vector Store is loadedThe Vector Store's data structure can be summarized using vector_store.summary(), which shows 4 tensors with 76 samples: tensor htype shape dtype compression ------- ------- ------- ------- ------- embedding embedding (76, 1536) float32 None id text (76, 1) str None metadata json (76, 1) str None text text (76, 1) str None To create a vector store using pre-compute embeddings instead of the embedding_data and embedding_function, you may run# vector_store.add(text = chunked_text, # embedding = <list_of_embeddings>, # metadata = [{'source': source_text}]*len(chunked_text))


Authentication

To use Deep Lake features that require authentication (Deep Lake storage, Tensor Database storage, connecting your cloud dataset to the Deep Lake UI, etc.) you should register in the Deep Lake App and authenticate on the client using the methods in the link below:User Authentication


Creating Vector Stores in the Deep Lake Managed Tensor Database

Deep Lake provides Managed Tensor Database that stores and runs queries on Deep Lake infrastructure, instead of the client. To use this service, specify runtime = {'tensor_db': True} when creating the Vector Store.# vector_store = VectorStore(# path = vector_store_path,# runtime = {'tensor_db': True}# )# vector_store.add(text = chunked_text, # embedding_function = embedding_function, # embedding_data = chunked_text, # metadata = [{'source': source_text}]*len(chunked_text)) # search_results = vector_store.search(embedding_data = prompt, # embedding_function = embedding_function)


Next Steps

Check out our Getting Started Guide for a comprehensive walk-through of Deep Lake Vector Stores. For scaling Deep Lake to production-level applications, check out our Managed Tensor Database and Support for Concurrent Writes.Congratulations, you've created a Vector Store and performed vector search using Deep Lake🤓


Visualizing your Vector Store

Visualization is available for Vector Stores stored in or connected to Deep Lake. The vector store above is stored locally, so it cannot be visualized, but here's an example of visualization for a representative Vector Store.


Installing Deep Lake

Deep Lake can be installed using pip. By default, Deep Lake does not install dependencies for video, google-cloud, compute engine, and other features. Details on all installation options are available here. !pip3 install deeplake


Reading Samples From a Deep Lake Dataset

Data is not immediately read into memory because Deep Lake operates lazily. You can fetch data by calling the .numpy() or .data() methods:# Indexingimage = ds.images[0].numpy() # Fetch the first image and return a numpy arraylabels = ds.labels[0].data() # Fetch the labels in the first image# Slicingimg_list = ds.labels[0:100].numpy(aslist=True) # Fetch 100 labels and store # them as a list of numpy arraysOther metadata such as the mapping between numerical labels and their text counterparts can be accessed using:labels_list = ds.labels.info['class_names']


Fetching Your First Deep Lake Dataset

Let's load the Visdrone dataset, a rich dataset with many object detections per image. Datasets hosted by Activeloop are identified by the host organization id followed by the dataset name: activeloop/visdrone-det-train.import deeplakedataset_path = 'hub://activeloop/visdrone-det-train'ds = deeplake.load(dataset_path) # Returns a Deep Lake Dataset but does not download data locally


Visualizing a Deep Lake Dataset

Deep Lake enables users to visualize and interpret large datasets. The tensor layout for a dataset can be inspected using:ds.summary()The dataset can be visualized in the Deep Lake UI, or using an iframe in a Jupyter notebook:ds.visualize()Visualizing datasets in the Deep Lake UI will unlock more features and faster performance compared to visualization in Jupyter notebooks.


Creating Your Own Deep Lake Datasets

You can access all of the features above and more with your own datasets! If your source data conforms to one of the formats below, you can ingest them directly with 1 line of code. The ingestion functions support source data from the cloud, as well as creation of Deep Lake datasets in the cloud.YOLOCOCOClassificationsFor example, a COCO format dataset can be ingested using:dataset_path = 's3://bucket_name_deeplake/dataset_name' # Destination for the Deep Lake datasetimages_folder = 's3://bucket_name_source/images_folder'annotations_files = ['s3://bucket_name_source/annotations.json'] # Can be a list of COCO jsons.ds = deeplake.ingest_coco(images_folder, annotations_files, dataset_path, src_creds = {...}, dest_creds = {...})For creating datasets that do not conform to one of the formats above, you can use our methods for manually creating datasets, tensors, and populating them with data.


Authentication

To use Deep Lake features that require authentication (Activeloop storage, Tensor Database storage, connecting your cloud dataset to the Deep Lake UI, etc.) you should register in the Deep Lake App and authenticate on the client using the methods in the link below:User Authentication


Next Steps

Check out our Getting Started Guide for a comprehensive walk-through of Deep Lake. Also check out tutorials on Running Queries, Training Models, and Creating Datasets, as well as Playbooks about powerful use-cases that are enabled by Deep Lake.Congratulations, you've got Deep Lake working on your local machine🤓


Storing Datasets in Your Own Cloud

Deep Lake can be used as a pure OSS package without any registration or relationship with Activeloop. However, registering with Activeloop offers several benefits:Storage provided by ActiveloopAccess to the Tensor Database for performant vector searchAccess to Deep Lake App, which provides dataset visualization, querying, version control UI, dataset analytics, and other powerful featuresManaged credentials for Deep Lake datasets stored outside of ActiveloopWhen connecting data from your cloud using Managed Credentials, the data is never stored or cached in Deep Lake. All Deep Lake user interfaces (browser, python, etc.) fetch data directly from long-term storage.Authentication Using Managed CredentialsStorage OptionsStoring Deep Lake Data in Your Own Cloud


Compute Engine offers high-performance implementations of compute-heavy Deep Lake features, such as distributed dataloading, large queries, and indexing. The engine is built in C++ and the user-interface is in Python.

The Deep Lake Compute Engine is only accessible to registered and authenticated users, and it applies usage restrictions based on your Deep Lake Plan.


Features Optimized in the Compute Engine:

Performant DataloaderTensor Query Language (TQL)Index for ANN SearchManaged Tensor Database


How to use Deep Lake's performant Dataloader built and optimized in C++

Deep Lake offers an optimized dataloader implementation built in C++, which is 1.5-3X faster than the pure-python implementation, and it supports distributed training. The C++ and Python dataloaders can be used interchangeably, and their syntax varies as shown below.


Pure-Python Dataloader

train_loader = ds_train.pytorch(num_workers = 8, transform = transform, batch_size = 32, tensors=['images', 'labels'], shuffle = True)


C++ Dataloader

The C++ dataloader is only accessible to registered and authenticated users, and it applies usage restrictions based on your Deep Lake Plan.The Deep Lake query engine is only accessible to registered and authenticated users, and it applies usage restrictions based on your Deep Lake Plan.


TensorFlow

train_loader = ds.dataloader()\ .transform(transform)\ .batch(32)\ .shuffle(True)\ .offset(10000)\ .tensorflow(tensors=['images', 'labels'], num_workers = 8)


Further Information

Training ModelsTraining Reproducibility Using Deep Lake and Weights & Biases


PyTorch (returns PyTorch Dataloader)

train_loader = ds.dataloader()\ .transform(transform)\ .batch(32)\ .shuffle(True)\ .offset(10000)\ .pytorch(tensors=['images', 'labels'], num_workers = 8)


How to query datasets using the Deep Lake Tensor Query Language (TQL)

Querying datasets is a critical aspect of data science workflows that enables users to filter datasets and focus their work on the most relevant data. Deep Lake offers a highly-performant query engine built in C++ and optimized for the Deep Lake data format. The Deep Lake query engine is only accessible to registered and authenticated users, and it applies usage restrictions based on your Deep Lake Plan.


Querying in the Vector Store Python API

view = vector_store.search(query = <query_string>, exec_option = 'compute_engine')


Query Syntax

TQL Syntax


Querying in the low-level Python API

Queries can also be performed in the Python API using:view = ds.query(<query_string>)


Saving and utilizing dataset query results in the low-level Python API

The query results (Dataset Views) can be saved in the UI as shown above, or if the view is generated in Python, it can be saved using the Python API below. Full details are available here.ds_view.save_view(message = 'Samples with monarchs')In order to maintain data lineage, Dataset Views are immutable and are connected to specific commits. Therefore, views can only be saved if the dataset has a commit and there are no uncommitted changes in the HEAD. You can check for this using ds.has_head_changes can be loaded in the python API and they can passed to ML frameworks just like regular datasets:ds_view = ds.load_view(view_id, optimize = True, num_workers = 2)for data in ds_view.pytorch(): # Training loop hereThe optimize parameter in ds.load_view(..., optimize = True) materializes the Dataset View into a new sub-dataset that is optimized for streaming. If the original dataset uses linked tensors, the data will be copied to Deep Lake format.Optimizing the Dataset View is critical for achieving rapid streaming.If the saved Dataset View is no longer needed, it can be deleted using:ds.delete_view(view_id)


How to Use Deep Memory to Improve Retrieval Accuracy in Your LLM Apps

Deep Memory is a suite of tools that enables you to optimize your Vector Store for your use-case and achieve higher accuracy in your LLM apps.


Embedding Transformation

Deep Memory computes a transformation that converts your embeddings into an embedding space that is tailored for your use case. This increases the accuracy of your Vector Search by up to 22%, which significantly impacts the user experience of your LLM applications. Furthermore, Deep Memory can also be used to decrease costs by reducing the amount of context (k) that must be injected into the LLM prompt to achieve a given accuracy, thereby reducing token usage.


How Deep Lake Implements an Index for ANN Search

Deep Lake implements the Hierarchical Navigable Small World (HSNW) index for Approximate Nearest Neighbor (ANN) search. The index is based on the OSS Hsnwlib package with added optimizations. The implementation enables users to run queries on >35M embeddings in less than 1 second.


Unique aspects of Deep Lake's HSNW implementation

Rapid index creation with multi-threading optimized for Deep LakeEfficient memory management that reduces RAM usage


Memory Management in Deep Lake

RAM Cost >> On-disk Cost >> Object Storage CostMinimizing RAM usage and maximizing object store significantly reduces costs of running a Vector Database. Deep Lake has a unique implementation of memory allocation that minimizes RAM requirement without any performance penalty:Memory Architecture for the Deep Lake Vector Store


Limitations

The following limitations of the index are being implemented in upcoming releases:Index does not support incremental updates. If any update is made to the dataset, the index is re-created.If the search is performed using a combination of attribute and vector search, the index is not used and linear search is applied instead.


Using the Index

By default, the index is turned off in Deep Lake. To enable the index, during Vector Store initialization or loading, specify the Vector Store length threshold above which the index will be applied:vectorstore = VectorStore(path, index_params = {threshold: <threshold_int>})


LangChain and LlamaIndex

To use the Managed Vector Database in LangChain or Llama Index, specify dataset_path = hub://org_id/dataset_name and runtime = {'tensor_db': True} during Vector Store creation.


Overview of Deep Lake's Managed Tensor Database

Deep Lake offers a serverless Managed Tensor Database that eliminates the complexity of self-hosting and substantially lowers costs. Currently, it only supports dataset queries, including vector search, but additional features for creating and modifying data being added in December 2023.Comparison of Deep Lake as a Managed Database vs Embedded Database


REST API

A standalone REST API is available for interacting with the Managed Database:REST API


Further Information:

Migrating Datasets to the Tensor Database


Vector Store use-cases are implemented using an API that balances simplicity and customizability

Vector Store


Deep Learning use-cases are implemented using the low-level API that offers maximum customizability

Deep Learning


Deep Lake Tutorials Based on Use-Case:

Vector Store TutorialsDeep Learning Tutorials


Playbooks are comprehensive examples of end-to-end workflows using Activeloop products

Querying, Training and Editing Datasets with Data LineageEvaluating Model PerformanceTraining Reproducibility Using Deep Lake and Weights & BiasesWorking with Videos


Import and Installation

By default, Deep Lake does not install dependencies for audio, video, google-cloud, and other features. Details on installation options are available here. !pip3 install deeplakeimport deeplake


Loading Deep Lake Datasets

Deep Lake datasets can be stored at a variety of storage locations using the appropriate dataset_path parameter below. We support S3, GCS, Activeloop storage, and are constantly adding to the list.# Load a Deep Lake Datasetds = deeplake.load('dataset_path', creds = {'optional'}, token = 'optional')


Deleting Datasets

ds.delete()deeplake.delete('dataset_path', creds = {'optional'}, token = 'optional', token = 'optional')API deletions of Deep Lake Cloud datasets are immediate, whereas UI-initiated deletions are postponed by 5 minutes. Once deleted, dataset names can't be reused in the Deep Lake Cloud.


Creating Deep Lake Datasets

# Create an empty Deep Lake datasetds = deeplake.empty('dataset_path', creds = {'optional'}, token = 'optional')# Create an Deep Lake Dataset with the same tensors as another datasetds = deeplake.like(ds_object or 'dataset_path', creds = {'optional'}, token = 'optional')# Automatically create a Deep Lake Dataset from another data sourceds = deeplake.ingest(source_folder, deeplake_dataset_path, ... 'see API reference for details')ds = deeplake.ingest_coco(images_folder, 'annotations.json', deeplake_dataset_path, ... 'see API reference for details')ds = deeplake.ingest_yolo(data_directory, deeplake_dataset_path, class_names_file, ... 'see API reference for details')


Visualizing and Inspecting Datasets

ds.visualize()ds.summary()


Appending Data to Datasets

ds.append({'tensor_1': np.ones((1,4)), 'tensor_2': deeplake.read('image.jpg')})ds.my_group.append({'tensor_1': np.ones((1,4)), 'tensor_2': deeplake.read('image.jpg')})


Appending/Updating Data in Individual Tensors

# Append a single sampleds.my_tensor.append(np.ones((1,4)))ds.my_tensor.append(deeplake.read('image.jpg'))# Append multiple samples. The first axis in the # numpy array is assumed to be the sample axis for the tensords.my_tensor.extend(np.ones((5,1,4)))# Editing or adding data at a specific indexds.my_tensor[i] = deeplake.read('image.jpg')


Deleting data

# Removing samples by indexds.pop[i]# Delete all data in a tensords.<tensor_name>.clear()# Delete tensor and all of its datads.delete_tensor(<tensor_name>)


Creating Tensors

# Specifying htype is recommended for maximizing performance.ds.create_tensor('my_tensor', htype = 'bbox')# Specifiying the correct compression is critical for images, videos, audio and # other rich data types. ds.create_tensor('songs', htype = 'audio', sample_compression = 'mp3')


Appending Empty Samples or Skipping Samples

# Data appended as None will be returned as an empty arrayds.append('tensor_1': deeplake.read(...), 'tensor_2': None)ds.my_tensor.append(None)# Empty arrays can be explicitly appended if the length of the shape # of the empty array matches that of the other samplesds.boxes.append(np.zeros((0,4))


Connecting Deep Lake Datasets to ML Frameworks

# PyTorch Dataloaderdataloader = ds.pytorch(batch_size = 16, transform = {'images': torchvision_tform, 'labels': None}, num_workers = 2, scheduler = 'threaded')# TensorFlow Datasetds_tensorflow = ds.tensorflow()# Enterprise Dataloaderdataloader = ds.dataloader().batch(batch_size = 64).pytorch(num_workers = 8)


Accessing Tensor Data

# Read the i-th tensor samplenp_array = ds.my_tensor[i].numpy()text = ds.my_text_tensor[i].data() # More comprehensive view of the databytes = ds.my_tensor[i].tobytes() # More comprehensive view of the data# Read the i-th dataset sample as a numpy arrayimage = ds[i].images.numpy()# Read the i-th labels as a numpy array or list of stringslabels_array = ds.labels[i].numpy()labels_array = ds.labels[i].data()['value'] # same as .numpy()labels_string_list = ds.labels[i].data()['text']# Read a tensor sample from a hierarchical groupnp_array = ds.my_group.my_tensor_1[i].numpy()np_array = ds.my_group.my_tensor_2[i].numpy()# Read multiple tensor samples into numpy arraynp_array = ds.my_tensor[0:10].numpy() # Read multiple tensor samples into a list of numpy arraysnp_array_list = ds.my_tensor[0:10].numpy(aslist=True)


Creating Tensor Hierarchies

ds.create_group('my_group')ds.my_group.create_tensor('my_tensor')ds.create_tensor('my_group/my_tensor') #Automatically creates the group 'my_group'


Querying Datasets and Saving Dataset Views

A full list of supported queries is shown here. view = ds.query('Select * where contains(labels, 'giraffe')')view.save_view(optimize = True)view = ds.load_view(id = 'query_id')# Return the original dataset indices that satisfied the query conditionindices = list(view.sample_indices)


Adding Tensor and Dataset-Level Metadata

# Add or update dataset metadatads.info.update(key1 = 'text', key2 = number)# Also can run ds.info.update({'key1'='value1', 'key2' = num_value})# Add or update tensor metadatads.my_tensor.info.update(key1 = 'text', key2 = number)# Delete metadatads.info.delete() #Delete all metadatads.info.delete('key1') #Delete 1 key in metadata


Copying datasets

# Fastest option - copies everything including version historyds = deeplake.deepcopy('src_dataset_path', 'dest_dataset_path', src_creds, dest_creds, token)# Slower option - copies only data on the last commitds = deeplake.copy('src_dataset_path', 'dest_dataset_path', src_creds, dest_creds, token)


Advanced

# Load a Deep Lake Dataset if it already exists (same as deeplake.load), or initialize # a new Deep Lake Dataset if it does not already exist (same as deeplake.empty)ds = deeplake.dataset('dataset_path', creds = {'optional'}, token = 'optional')# Append multiple samples using a listds.my_tensor.extend([np.ones((1,4)), np.ones((3,4)), np.ones((2,4)# Fetch adjacent data in the chunk -> Increases speed when loading # sequantially or if a tensor's data fits in the cache.numeric_label = ds.labels[i].numpy(fetch_chunks = True)


Versioning Datasets

# Commit datacommit_id = ds.commit('Added 100 images of trucks')# Print the commit loglog = ds.log()# Checkout a branch or commit ds.checkout('branch_name' or commit_id)# Create a new branchds.checkout('new_branch', create = True)# Examine differences between commitsds.diff()# Delete all changes since the previous commitds.reset()# Delete a branch and its commits - Only allowed for branches that have not been mergedds.delete_branch('branch_name')


Maximizing performance

Make sure to use the with context when making any updates to datasets. with ds: ds.create_tensor('my_tensor') for i in range(10): ds.my_tensor.append(i)


How to use Deep Lake at Scale with best practices


Tensors

Deep Lake uses a columnar storage architecture, and the columns in Deep Lake are referred to as tensors. Data in the tensors can be added or modified, and the data in different tensors are independent of each other.


Hidden Tensors

When data is appended to Deep Lake, certain important information is broken up and duplicated in a separate tensor, so that the information can be accessed and queried without loading all of the data. Examples include the shape of a sample (i.e. width, height, and number of channels for an image), or the metadata from file headers that were passed to deeplake.read('filename').


Indexing and Samples

Deep Lake datasets and their tensors are indexed, and data at a given index that spans multiple tensors are referred to as samples. Data at the same index are assumed to be related. For example, data in a bbox tensor at index 100 is assumed to be related to data in the tensor image at index 100.


Chunking

Most data in Deep Lake format is stored in chunks, which are a blobs of data of a pre-defined size. The purpose of chunking is to accelerate the streaming of data across networks by increasing the amount of data that is transferred per network request.Each tensors has its own chunks, and the default chunk size is 8MB. A single chunk consists of data from multiple indices when the individual data points (image, label, annotation, etc.) are smaller than the chunk size. Conversely, when individual data points are larger than the chunk size, the data is split among multiple chunks (tiling). Exceptions to chunking logic are video data. Videos that are larger than the specified chunk size are not broken into smaller pieces, because Deep Lake uses efficient libraries to stream and access subsets of videos, thus making it unnecessary to split them apart.


Groups

Multiple tensor can be combined into groups. Groups do not fundamentally change the way data is stored, but they are useful for helping Activeloop Platform understand how different tensors are related.


Length of a Dataset

Deep Lake allows for ragged tensors (tensors of different length), so it is important to understand the terminology around dataset length:length (ds.len or len(ds)) - The length of the shortest tensor, as determined by its last index.minimum length (ds.min_len) - Same as lengthminimum length (ds.max_len) - The length of the longest tensor, as determined by its last index. By default, Deep Lake throws an error if a tensor is accessed at an index at which data (empty or non-empty) has not been added. In the example below, ds.bbox[3].numpy() would throw an error. To pad the unspecified data and create a virtual view where the missing samples are treated as empty data, use ds.max_view(). In the example below, the length of this virtual view would be 6.


Understanding the Interaction Between Deep Lake's Versions, Queries, and Dataset Views.

Version control is the core of the Deep Lake data format, and it interacts with queries and view as follows:Datasets have commits and branches, and they can be traversed or merged using Deep Lake's Python API. Queries are applied on top of commits, and in order to save a query result as a view, the dataset cannot be in an uncommitted state (no changes were performed since the prior commit). Each saved view is associated with a particular commit, and the view itself contains information on which dataset indices satisfied the query condition.This logical approach was chosen in order to preserve data lineage. Otherwise, it would be possible to change data on which a query was executed, thereby potentially invalidating the saved view, since the indices that satisfied the query condition may no longer be correct after the dataset was changed. Please check out our Getting Stated Guide to learn how to use the Python API to version your data, run queries, and save views. An example workflow using version control and queries is shown below.


Version Control HEAD Commit

Unlike Git, Deep Lake's dataset version control does not have a local staging area because all dataset updates are immediately synced with the permanent storage location (cloud or local). Therefore, any changes to a dataset are automatically stored in a HEAD commit on the current branch. This means that the uncommitted changes do not appear on other branches, and uncommitted changes are visible to all users.


How to visualize machine learning datasets

Deep Lake has a web interface for visualizing, versioning, and querying machine learning datasets. It utilizes the Deep Lake format under-the-hood, and it can be connected to datasets stored in all Deep Lake storage locations.


Visualization can be performed in 3 ways:

1.In the Deep Lake UI (most feature-rich and performant option)2.In the python API using ds.visualize()3.In your own application using our integration options.


Requirements for correctly visualizing your own datasets

Deep Lake makes assumptions about underlying data types and relationships between tensors in order to display the data correctly. Understanding the following concepts is necessary in order to use the visualizer: 1.Data Types (htypes)2.Relationships between tensors


Downsampling Data for Faster Visualization

For faster visualization of images and masks, tensors can be downsampled during dataset creation. The downsampled data are stored in the dataset and are automatically rendered by the visualizer depending on the zoom level. To add downsampling to your tensors, specify the downsampling factor and the number of downsampling layers during tensor creation:# 3X downsampling per layer, 2X layersds.create_tensor('images', htype = 'image', downsampling = (3,2))Note: since downsampling requires decompression and recompression of data, it will slow down dataset ingestion.


Indexing

Hub datasets and their tensors are indexed like ds[index] or ds.tensor_name[index], and data at the same index are assumed to be related. For example, a bounding_box at index 100 is assumed to apply to the image at index 100.


Relationships Between Tensors

For datasets with multiple tensors, it is important to follow the conventions below in order for the visualizer to correctly infer how tensors are related.By default, in the absence of groups, the visualizer assumes that all tensors are related to each other. This works well for simple use cases. For example, it is correct to assume that the images, labels, and boxes tensors are related in the dataset below:ds-> images (htype = image)-> labels (htype = class_label)-> boxes (htype = bbox)However, if datasets are highly complex, assuming that all tensor are related may lead to visualization errors, because every tensor may not be related to every other tensor:ds-> images (htype = image)-> vehicle_labels (htype = class_label)-> vehicle_boxes (htype = bbox)-> people_labels (htype = class_label)-> people_masks (htype = binary_mask)In the example above, only some of the annotation tensors are related to each other: vehicle_labels -> vehicle_boxes: Boxes and labels describing cars, trucks, etc.people_labels -> people_masks: Binary masks and labels describing adults, toddlers, etc.The best method for disambiguating the relationships between tensors is to place them in groups, because the visualizer assumes that annotation tensors in different groups are not related.In the example above, the following groups could be used to disambiguate the annotations:ds-> images (htype = image)-> vehicles (group) -> vehicle_labels (htype = class_label) -> vehicle_boxes (htype = bbox)-> people (group) -> people_labels (htype = class_label) -> people_masks (htype = binary_mask)


HTML iframe (Alpha)

To embed into your html page, you can use our iframe integration:<iframe src='https://app.activeloop.ai/visualizer/iframe?url=hub://activeloop/imagenet-train' width='800px' height='600px'>iframe URL: Params:url - The url of the dataset vs - Visualizer state, which can be obtained from the platform url token - User token, for private datasets. If the value is ask then the UI will be populated for entering the token checkpoint - Dataset checkpoint query - Query string to apply on the dataset


How to embed the Activeloop visualizer into your own web applications

Visualization engine allows the user to visualize, explore, and interact with Deep Lake datasets. In addition to using through the Activeloop UI or in Python, the Activeloop visualizer can also be embedded into your application.


Javascript API (Alpha)

To have more fine grained control, you can embed the visualizer using Javascript:

or to visualize private datasets with authentication

InterfaceBelow you can find definitions of the arguments./// ds - Dataset url/// commit - optional commit id/// state - optional initial state of the visualizer/// container - HTML element serving as container for visualizer elements/// options - optional Visualization optionsstatic visualize( ds: string, commit: string | null = null, state: string | null = null, container: HTMLElement, options: VisOptions | null): Promise<Vis>;/// backlink - Show backlink to platform button/// singleSampleView - Enable single sample view through enter key/// requireSignin - Requires signin to get access token/// token - Token id/// gridMode - Canvas vs Grid/// queryString - Query to apply on the iframeexport type VisOptions = { backlink?: Boolean singleSampleView?: Boolean requireSignin?: Boolean token: string | null gridMode?: 'canvas' | 'grid' queryString?: string}This visualize returns Promise<Vis> which can be used to dynamically change the visualizer state. Vis supports only query functions for nowclass Vis{	/// Asynchronously runs a query and resolves the promise when query completed. /// In case of error in query, rejects the promise.	query(queryString: string): Promise<void>}


How Shuffling Works in Deep Lake's PyTorch DataLoader

The Deep Lake shuffling algorithm is based upon a shuffle buffer that preloads a specified amount of data (in MB) determined by the buffer_size parameter in ds.pytorch(buffer_size = 2048). First, the dataloader randomly selects chunks from the applicable tensors until the shuffle buffer is full. Next, the indices in shuffle buffer are randomly sampled to construct the batches that are returned by the dataloader. As the data in the shuffle buffer is consumed, new chunks are randomly selected and added to the buffer.In the OSS dataloader, the shuffle buffer contains the decompressed, decoded, and transformed samples. When using the PyTorch dataloaders, this corresponds to torch tensors. In the Performant dataloader, the shuffle buffer contains the non-decompressed data in the format they are stored in. For images, this typically corresponds to compressed bytes in jpeg, png, or other compressions. Since compressed data is stored more efficiently than uncompressed data, there are typically more distinct samples of data in the Performant dataloader shuffle buffer compared to the OSS shuffle buffer. If many chunks in the buffer contain data from the same class, which may occur if data was uploaded in non-random order, the shuffle buffer may contain fewer unique classes than if the samples were chosen fully randomly based on index. The most extreme case of reduced randomness occurs when datasets are much larger than the shuffle buffer, when they have many classes, and when those classes occur in sequence within the dataset indices. One example dataset is Unshuffled ImageNet, which has 1000 classes, 1.2M images, 140GB of data, and approximately 140 images per 16MB chunk. When the images are uploaded in sequence, the plot below shows how many unique classes are returned by the loader vs the number of images that have been returned in total. It is evident that fully randomly sampling returns more unique values than the Deep Lake dataloader. If reduced randomness has an impact on model performance in your workflows, the recommended countermeasures are:Store the dataset in a shuffled fashion such that the data does not appear in order by class. This completely mitigates the randomness concerns at the output of the data loader.Store the dataset with a smaller chunk size. This increases randomness because the shuffle buffer selects more discreet chunks before filling up. The current default size is 8, and reducing chunk size to 4MB significantly increases randomness (see plot above) with only a modest slowdown in data transfer speed.Increase the size of the shuffle buffer. This mitigates the randomness concerns but may not completely alleviate them.


Providing Feedback

We love feedback! Please join our Slack Community or raise an issue in Github.


Getting Started With Development

Clone the repository:git clone https://github.com/activeloopai/deeplake cd deeplake If you are using Linux, install environment dependencies:apt-get -y updateapt-get -y install git wget build-essential python-setuptools python3-dev libjpeg-dev libpng-dev zlib1g-devapt install build-essentialIf you are planning to work on videos, install codecs:apt-get install -y ffmpeg libavcodec-dev libavformat-dev libswscale-devInstall the package locally with plugins and development dependencies:pip install -r deeplake/requirements/plugins.txtpip install -r deeplake/requirements/tests.txtpip install -e .Run local tests to ensure everything is correct:pytest -x --local .


Using Docker (optional)

You can use docker-compose for running testsdocker-compose -f ./bin/docker-compose.yaml up --build localand even work inside the docker by building the image and bashing into.docker build -t activeloop-deeplake:latest -f ./bin/Dockerfile.dev .docker run -it -v $(pwd):/app activeloop-deeplake:latest bash$ python3 -c 'import deeplake'Now changes done on your local files will be directly reflected into the package running inside the docker.


Linting

Deep Lake uses the black python linter. You can auto-format your code by running pip install black, and the run black . inside the directory you want to format.


Docstrings

Deep Lake uses Google Docstrings. Please refer to this example to learn more.


Typing

Deep Lake uses static typing for function arguments/variables for better code readability. Deep Lake has a GitHub action that runs mypy ., which runs similar to pytest . to check for valid static typing. You can refer to mypy documentation for more information.


Prerequisites

Understand how to write pytest tests.Understand what a pytest fixture is.Understand what pytest parametrizations are.


Testing

Deep Lake uses pytest for tests. In order to make it easier to contribute, Deep Lake also has a set of custom options defined here.


Options

To see a list of Deep Lake's custom pytest options, run this command: pytest -h | sed -En '/custom options:/,/\[pytest\] ini\-options/p'.


Fixtures

You can find more information on pytest fixtures here.memory_storage: If --memory-skip is provided, tests with this fixture will be skipped. Otherwise, the test will run with only a MemoryProvider.local_storage: If --local is not provided, tests with this fixture will be skipped. Otherwise, the test will run with only a LocalProvider.s3_storage: If --s3 is not provided, tests with this fixture will be skipped. Otherwise, the test will run with only an S3Provider.storage: All tests that use the storage fixture will be parametrized with the enabled StorageProviders (enabled via options defined below). If --cache-chains is provided, storage may also be a cache chain. Cache chains have the same interface as StorageProvider, but instead of just a single provider, it is multiple chained in a sequence, where the last provider in the chain is considered the actual storage.ds: The same as the storage fixture, but the storages that are parametrized are wrapped with a Dataset.Each StorageProvider/Dataset that is created for a test via a fixture will automatically have a root created, and it will be destroyed after the test. If you want to keep this data after the test run, you can use the --keep-storage option.


Fixture Examples

Single storage provider fixture:def test_memory(memory_storage): # Test will skip if `--memory-skip` is provided memory_storage['key'] = b'1234' # This data will only be stored in memorydef test_local(local_storage): # Test will skip if `--local` is not provided memory_storage['key'] = b'1234' # This data will only be stored locallydef test_local(s3_storage): # Test will skip if `--s3` is not provided # Test will fail if credentials are not provided memory_storage['key'] = b'1234' # This data will only be stored in s3Multiple storage providers/cache chains:from deeplake.core.tests.common import parametrize_all_storages, parametrize_all_caches, parametrize_all_storages_and_caches@parametrize_all_storagesdef test_storage(storage): # Storage will be parametrized with all enabled `StorageProvider`s pass@parametrize_all_cachesdef test_caches(storage): # Storage will be parametrized with all common caches containing enabled `StorageProvider`s pass@parametrize_all_storages_and_cachesdef test_storages_and_caches(storage): # Storage will be parametrized with all enabled `StorageProvider`s and common caches containing enabled `StorageProvider`s passDataset storage providers/cache chains:from deeplake.core.tests.common import parametrize_all_dataset_storages, parametrize_all_dataset_storages_and_caches@parametrize_all_dataset_storagesdef test_dataset(ds): # `ds` will be parametrized with 1 `Dataset` object per enabled `StorageProvider` pass@parametrize_all_dataset_storages_and_cachesdef test_dataset(ds): # `ds` will be parametrized with 1 `Dataset` object per enabled `StorageProvider` and all cache chains containing enabled `StorageProvider`s pass


Benchmarks

Deep Lake uses pytest-benchmark for benchmarking, which is a plugin for pytest.


Here's a list of people who are building the future of data!

Deep Lake would not be possible without the work of our community.Activeloop Deep Lake open-source contributors


How to Use Deep Lake as a Vector Store in LangChain

Deep Lake can be used as a VectorStore in LangChain for building Apps that require filtering and vector search. In this tutorial we will show how to create a Deep Lake Vector Store in LangChain and use it to build a Q&A App about the Twitter OSS recommendation algorithm. This tutorial requires installation of:!pip3 install langchain deeplake openai tiktoken


Downloading and Preprocessing the Data

First, let's import necessary packages and make sure the Activeloop and OpenAI keys are in the environmental variables ACTIVELOOP_TOKEN, OPENAI_API_KEY.from langchain.embeddings.openai import OpenAIEmbeddingsfrom langchain.vectorstores import DeepLakefrom langchain.document_loaders import TextLoaderfrom langchain.text_splitter import CharacterTextSplitterfrom langchain.chat_models import ChatOpenAIfrom langchain.chains import RetrievalQA, ConversationalRetrievalChainimport osNext, let's clone the Twitter OSS recommendation algorithm:!git clone https://github.com/twitter/the-algorithmNext, let's load all the files from the repo into a list:repo_path = '/the-algorithm'docs = []for dirpath, dirnames, filenames in os.walk(repo_path): for file in filenames: try: loader = TextLoader(os.path.join(dirpath, file), encoding='utf-8') docs.extend(loader.load_and_split()) except Exception as e: print(e) pass


Use the Vector Store in a Q&A App

We can now use the VectorStore in Q&A app, where the embeddings will be used to filter relevant documents (texts) that are fed into an LLM in order to answer a question.If we were on another machine, we would load the existing Vector Store without recalculating the embeddings:db = DeepLake(dataset_path=dataset_path, read_only=True, embedding=embeddings)We have to create a retriever object and specify the search parameters.retriever = db.as_retriever()retriever.search_kwargs['distance_metric'] = 'cos'retriever.search_kwargs['k'] = 20Finally, let's create an RetrievalQA chain in LangChain and run it:model = ChatOpenAI(model='gpt-4') # 'gpt-3.5-turbo',qa = RetrievalQA.from_llm(model, retriever=retriever)qa.run('What programming language is most of the SimClusters written in?')This returns:We can tune k in the retriever depending on whether the prompt exceeds the model's token limit. Higher k increases the accuracy by including more data in the prompt.


Adding data to to an existing Vector Store

Data can be added to an existing Vector Store by loading it using its path and adding documents or texts. db = DeepLake(dataset_path=dataset_path, embedding=embeddings)# Don't run this here in order to avoid data duplication# db.add_documents(texts)


Creating the Deep Lake Vector Store

First, we specify a path for storing the Deep Lake dataset containing the embeddings and their metadata. dataset_path = 'hub://<org-id>/twitter_algorithm'Next, we specify an OpenAI algorithm for creating the embeddings, and create the VectorStore. This process creates an embedding for each element in the texts lists and stores it in Deep Lake format at the specified path. embeddings = OpenAIEmbeddings()db = DeepLake.from_documents(texts, embeddings, dataset_path=dataset_path)The Deep Lake dataset serving as a VectorStore has 4 tensors including the embedding, its ids, metadata including the filename of the text, and the text itself. tensor htype shape dtype compression ------- ------- ------- ------- ------- embedding generic (23156, 1536) float32 None ids text (23156, 1) str None metadata json (23156, 1) str None text text (23156, 1) str None


Adding Hybrid Search to the Vector Store

Since embeddings search can be computationally expensive, you can simplify the search by filtering out data using an explicit search on top of the embeddings search. Suppose we want to answer to a question related to the trust and safety models. We can filter the filenames (source) in the metadata using a custom function that is added to the retriever:def filter(deeplake_sample): return 'trust_and_safety_models' in deeplake_sample['metadata'].data()['value']['source']retriever.search_kwargs['filter'] = filterqa = RetrievalQA.from_llm(model, retriever=retriever)qa.run('What do the trust and safety models do?')This returns:Filters can also be specified as a dictionary. For example, if the metadata tensor had a key year, we can filter based on that key using:# retriever.search_kwargs['filter'] = {'metadata': {'year': 2020}}


A note on chunking text files:

Text files are typically split into chunks before creating embeddings. In general, more chunks increases the relevancy of data that is fed into the language model, since granular data can be selected with higher precision. However, since an embedding will be created for each chunk, more chunks increase the computational complexity.text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)texts = text_splitter.split_documents(docs)Chunks in the above context should not be confused with Deep Lake chunks!


Using Deep Lake in Applications that Require Concurrency

For applications that require writing of data concurrently, users should set up a lock system to queue the write operations and prevent multiple clients from writing to the Deep Lake Vector Store at the same time. This can be done with a few lines of code in the example below:Concurrency Using Zookeeper Locks


Accessing the Low Level Deep Lake API (Advanced)

When using a Deep Lake Vector Store in LangChain, the underlying Vector Store and its low-level Deep Lake dataset can be accessed via:# LangChain Vector Storedb = DeepLake(dataset_path=dataset_path)# Deep Lake Vector Store objectds = db.vectorstore# Deep Lake Dataset objectds = db.vectorstore.dataset


SelfQueryRetriever with Deep Lake

Deep Lake supports the SelfQueryRetriever implementation in LangChain, which translates a user prompt into a metadata filters


SelfQueryRetriever with Deep Lake

This section of the tutorial requires installation of additional packages:First let's create a Deep Lake Vector Store with relevant data using the documents below.docs = [ Document( page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'rating': 7.7, 'genre': 'science fiction'}, ), Document( page_content='Leo DiCaprio gets lost in a dream within a dream within a dream within a ...', metadata={'year': 2010, 'director': 'Christopher Nolan', 'rating': 8.2}, ), Document( page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'director': 'Satoshi Kon', 'rating': 8.6}, ), Document( page_content='A bunch of normal-sized women are supremely wholesome and some men pine after them', metadata={'year': 2019, 'director': 'Greta Gerwig', 'rating': 8.3}, ), Document( page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'}, ), Document( page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={ 'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction', 'rating': 9.9, }, ),]Since this feature uses Deep Lake's Tensor Query Language under the hood, the Vector Store must be stored in or connected to Deep Lake, which requires registration with Activeloop:org_id = <YOUR_ORG_ID> #By default, your username is an org_iddataset_path = f'hub://{org_id}/self_query'vectorstore = DeepLake.from_documents( docs, embeddings, dataset_path = dataset_path, overwrite = True,)Next, let's instantiate our retriever by providing information about the metadata fields that our documents support and a short description of the document contents.from langchain.llms import OpenAIfrom langchain.retrievers.self_query.base import SelfQueryRetrieverfrom langchain.chains.query_constructor.base import AttributeInfometadata_field_info = [ AttributeInfo( name='genre', description='The genre of the movie', type='string or list[string]', ), AttributeInfo( name='year', description='The year the movie was released', type='integer', ), AttributeInfo( name='director', description='The name of the movie director', type='string', ), AttributeInfo( name='rating', description='A 1-10 rating for the movie', type='float' ),]document_content_description = 'Brief summary of a movie'llm = OpenAI(temperature=0)retriever = SelfQueryRetriever.from_llm( llm, vectorstore, document_content_description, metadata_field_info, verbose=True)And now we can try actually using our retriever!# This example only specifies a relevant queryretriever.get_relevant_documents('What are some movies about dinosaurs')Output:[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'rating': 7.7, 'genre': 'science fiction'}), Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'}), Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'}), Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'director': 'Satoshi Kon', 'rating': 8.6})]Now we can run a query to find movies that are above a certain ranking:# This example only specifies a filterretriever.get_relevant_documents('I want to watch a movie rated higher than 8.5')Output:[Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'director': 'Satoshi Kon', 'rating': 8.6}), Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'})]Congrats! You just used the Deep Lake Vector Store in LangChain to create a Q&A App! 🎉


Installation

Deep Lake can be installed with pip Copy to clipboard Deep Lake has the following extras that you can choose to install according to your needs. Installation commands Install command Description Dependencies installed pip install 'deeplake[av]' Audio and video support via PyAV av pip install 'deeplake[visualizer]' Visualize Deep Lake datasets within notebooks. This is required for Dataset.visualize to work. IPython, flask pip install 'deeplake[gcp]' GCS support google-cloud-storage, google-auth, google-auth-oauthlib pip install 'deeplake[azure]' Azure Blob Storage support azure-storage-blob, azure-cli, azure-identity pip install 'deeplake[medical]' DICOM and NIfTI data support pydicom, nibabel pip install 'deeplake[gdrive]' Google Drive support google-api-python-client, oauth2client, google-auth, google-auth-oauthlib pip install 'deeplake[point_cloud]' Support for LiDAR point cloud data laspy pip install 'deeplake[all]' Installs all of the above


Datasets

Creating Datasets deeplake.dataset Returns a Dataset object referencing either a new or existing dataset.  deeplake.empty Creates an empty dataset deeplake.like Creates a new dataset by copying the source dataset's structure to a new location.  deeplake.ingest_classification Ingest a dataset of images from a local folder to a Deep Lake Dataset.  deeplake.ingest_coco Ingest images and annotations in COCO format to a Deep Lake Dataset.  deeplake.ingest_yolo Ingest images and annotations (bounding boxes or polygons) in YOLO format to a Deep Lake Dataset.  deeplake.ingest_kaggle Download and ingest a kaggle dataset and store it as a structured dataset to destination.  deeplake.ingest_dataframe Convert pandas dataframe to a Deep Lake Dataset.  deeplake.ingest_huggingface Converts Hugging Face datasets to Deep Lake format.  Loading Datasets deeplake.load Loads an existing dataset Deleting and Renaming Datasets deeplake.delete Deletes a dataset at a given path.  deeplake.rename Renames dataset at old_path to new_path.  Copying Datasets deeplake.copy Copies dataset at src to dest.  deeplake.deepcopy Copies dataset at src to dest including version control history.  Dataset Operations Dataset.summary Prints a summary of the dataset.  Dataset.append Append samples to mutliple tensors at once.  Dataset.extend Appends multiple rows of samples to mutliple tensors at once.  Dataset.update Update existing samples in the dataset with new values.  Dataset.query Returns a sliced Dataset with given query results.  Dataset.copy Copies this dataset or dataset view to dest.  Dataset.delete Deletes the entire dataset from the cache layers (if any) and the underlying storage.  Dataset.rename Renames the dataset to path.  Dataset.connect Connect a Deep Lake cloud dataset through a deeplake path.  Dataset.visualize Visualizes the dataset in the Jupyter notebook.  Dataset.pop Removes a sample from all the tensors of the dataset.  Dataset.rechunk Rewrites the underlying chunks to make their sizes optimal.  Dataset.flush Necessary operation after writes if caches are being used.  Dataset.clear_cache Flushes (see Dataset.flush()) the contents of the cache layers (if any) and then deletes contents of all the layers of it.  Dataset.size_approx Estimates the size in bytes of the dataset.  Dataset.random_split Splits the dataset into non-overlapping Dataset objects of given lengths.  Dataset Visualization Dataset.visualize Visualizes the dataset in the Jupyter notebook.  Dataset Credentials Dataset.add_creds_key Adds a new creds key to the dataset.  Dataset.populate_creds Populates the creds key added in add_creds_key with the given creds.  Dataset.update_creds_key Updates the name and/or management status of a creds key.  Dataset.get_creds_keys Returns the set of creds keys added to the dataset.  Dataset Properties Dataset.tensors All tensors belonging to this group, including those within sub groups.  Dataset.groups All sub groups in this group Dataset.num_samples Returns the length of the smallest tensor.  Dataset.read_only Returns True if dataset is in read-only mode and False otherwise.  Dataset.info Returns the information about the dataset.  Dataset.max_len Return the maximum length of the tensor.  Dataset.min_len Return the minimum length of the tensor.  Dataset Version Control Dataset.commit Stores a snapshot of the current state of the dataset


Datasets

Dataset.diff Returns/displays the differences between commits/branches .  Dataset.checkout Checks out to a specific commit_id or branch.  Dataset.merge Merges the target_id into the current dataset.  Dataset.log Displays the details of all the past commits.  Dataset.reset Resets the uncommitted changes present in the branch.  Dataset.get_commit_details Get details of a particular commit.  Dataset.commit_id The lasted committed commit id of the dataset.  Dataset.branch The current branch of the dataset Dataset.pending_commit_id The commit_id of the next commit that will be made to the dataset.  Dataset.has_head_changes Returns True if currently at head node and uncommitted changes are present.  Dataset.commits Lists all the commits leading to the current dataset state.  Dataset.branches Lists all the branches of the dataset.  Dataset Views A dataset view is a subset of a dataset that points to specific samples (indices) in an existing dataset.  Dataset views can be created by indexing a dataset, filtering a dataset with Dataset.filter(), querying a dataset with Dataset.query() or by sampling a dataset with Dataset.sample_by().  Filtering is done with user-defined functions or simplified expressions whereas query can perform SQL-like queries with our Tensor Query Language.  See the full TQL spec here.  Dataset views can only be saved when a dataset has been committed and has no changes on the HEAD node, in order to preserve data lineage and prevent the underlying data from changing after the query or filter conditions have been evaluated.  Example >>> import deeplake >>> # load dataset >>> ds = deeplake.load('hub://activeloop/mnist-train') >>> # filter dataset >>> zeros = ds.filter('labels == 0') >>> # save view >>> zeros.save_view(id='zeros') >>> # load_view >>> zeros = ds.load_view(id='zeros') >>> len(zeros) 5923 Copy to clipboard Dataset.query Returns a sliced Dataset with given query results.  Dataset.sample_by Returns a sliced Dataset with given weighted sampler applied.  Dataset.filter Filters the dataset in accordance of filter function f(x: sample) -> bool Dataset.save_view Saves a dataset view as a virtual dataset (VDS) Dataset.get_view Returns the dataset view corresponding to id.  Dataset.load_view Loads the view and returns the Dataset by id.  Dataset.delete_view Deletes the view with given view id.  Dataset.get_views Returns list of views stored in this Dataset.  Dataset.is_view Returns True if this dataset is a view and False otherwise.  Dataset.min_view Returns a view of the dataset in which all tensors are sliced to have the same length as the shortest tensor.  Dataset.max_view Returns a view of the dataset in which shorter tensors are padded with None s to have the same length as the longest tensor.


Loading Datasets

deeplake.load Loads an existing dataset


Creating Datasets

deeplake.dataset Returns a Dataset object referencing either a new or existing dataset. deeplake.empty Creates an empty dataset deeplake.like Creates a new dataset by copying the source dataset's structure to a new location. deeplake.ingest_classification Ingest a dataset of images from a local folder to a Deep Lake Dataset. deeplake.ingest_coco Ingest images and annotations in COCO format to a Deep Lake Dataset. deeplake.ingest_yolo Ingest images and annotations (bounding boxes or polygons) in YOLO format to a Deep Lake Dataset. deeplake.ingest_kaggle Download and ingest a kaggle dataset and store it as a structured dataset to destination. deeplake.ingest_dataframe Convert pandas dataframe to a Deep Lake Dataset. deeplake.ingest_huggingface Converts Hugging Face datasets to Deep Lake format.


Dataset Visualization

Dataset.visualize Visualizes the dataset in the Jupyter notebook.


Dataset Credentials

Dataset.add_creds_key Adds a new creds key to the dataset. Dataset.populate_creds Populates the creds key added in add_creds_key with the given creds. Dataset.update_creds_key Updates the name and/or management status of a creds key. Dataset.get_creds_keys Returns the set of creds keys added to the dataset.


Dataset Operations

Dataset.summary Prints a summary of the dataset. Dataset.append Append samples to mutliple tensors at once. Dataset.extend Appends multiple rows of samples to mutliple tensors at once. Dataset.update Update existing samples in the dataset with new values. Dataset.query Returns a sliced Dataset with given query results. Dataset.copy Copies this dataset or dataset view to dest. Dataset.delete Deletes the entire dataset from the cache layers (if any) and the underlying storage. Dataset.rename Renames the dataset to path. Dataset.connect Connect a Deep Lake cloud dataset through a deeplake path. Dataset.visualize Visualizes the dataset in the Jupyter notebook. Dataset.pop Removes a sample from all the tensors of the dataset. Dataset.rechunk Rewrites the underlying chunks to make their sizes optimal. Dataset.flush Necessary operation after writes if caches are being used. Dataset.clear_cache Flushes (see Dataset.flush()) the contents of the cache layers (if any) and then deletes contents of all the layers of it. Dataset.size_approx Estimates the size in bytes of the dataset. Dataset.random_split Splits the dataset into non-overlapping Dataset objects of given lengths.


Copying Datasets

deeplake.copy Copies dataset at src to dest. deeplake.deepcopy Copies dataset at src to dest including version control history.


Deleting and Renaming Datasets

deeplake.delete Deletes a dataset at a given path. deeplake.rename Renames dataset at old_path to new_path.


Dataset Version Control

Dataset.commit Stores a snapshot of the current state of the dataset. Dataset.diff Returns/displays the differences between commits/branches. Dataset.checkout Checks out to a specific commit_id or branch. Dataset.merge Merges the target_id into the current dataset. Dataset.log Displays the details of all the past commits. Dataset.reset Resets the uncommitted changes present in the branch. Dataset.get_commit_details Get details of a particular commit. Dataset.commit_id The lasted committed commit id of the dataset. Dataset.branch The current branch of the dataset Dataset.pending_commit_id The commit_id of the next commit that will be made to the dataset. Dataset.has_head_changes Returns True if currently at head node and uncommitted changes are present. Dataset.commits Lists all the commits leading to the current dataset state. Dataset.branches Lists all the branches of the dataset.


Dataset Views

A dataset view is a subset of a dataset that points to specific samples (indices) in an existing dataset. Dataset views can be created by indexing a dataset, filtering a dataset with Dataset.filter(), querying a dataset with Dataset.query() or by sampling a dataset with Dataset.sample_by(). Filtering is done with user-defined functions or simplified expressions whereas query can perform SQL-like queries with our Tensor Query Language. See the full TQL spec here.Dataset views can only be saved when a dataset has been committed and has no changes on the HEAD node, in order to preserve data lineage and prevent the underlying data from changing after the query or filter conditions have been evaluated.Example Copy to clipboard Dataset.query Returns a sliced Dataset with given query results. Dataset.sample_by Returns a sliced Dataset with given weighted sampler applied. Dataset.filter Filters the dataset in accordance of filter function f(x: sample) -> bool Dataset.save_view Saves a dataset view as a virtual dataset (VDS) Dataset.get_view Returns the dataset view corresponding to id. Dataset.load_view Loads the view and returns the Dataset by id. Dataset.delete_view Deletes the view with given view id. Dataset.get_views Returns list of views stored in this Dataset. Dataset.is_view Returns True if this dataset is a view and False otherwise. Dataset.min_view Returns a view of the dataset in which all tensors are sliced to have the same length as the shortest tensor. Dataset.max_view Returns a view of the dataset in which shorter tensors are padded with None s to have the same length as the longest tensor.


Dataset Properties

Dataset.tensors All tensors belonging to this group, including those within sub groups. Dataset.groups All sub groups in this group Dataset.num_samples Returns the length of the smallest tensor. Dataset.read_only Returns True if dataset is in read-only mode and False otherwise. Dataset.info Returns the information about the dataset. Dataset.max_len Return the maximum length of the tensor. Dataset.min_len Return the minimum length of the tensor.


Vector Store

Creating a Deep Lake Vector Store VectorStore.__init__ Creates an empty VectorStore or loads an existing one if it exists at the specified path. Vector Store Operations VectorStore.add Adding elements to deeplake vector store. VectorStore.search VectorStore search method that combines embedding search, metadata search, and custom TQL search. VectorStore.delete Delete the data in the Vector Store. VectorStore.delete_by_path Deleted the Vector Store at the specified path. VectorStore.update_embedding Recompute existing embeddings of the VectorStore, that match either query, filter, ids or row_ids. Vector Store Properties VectorStore.summary Prints a summary of the dataset VectorStore.tensors Returns the list of tensors present in the dataset VectorStore.__len__ Length of the dataset


Creating a Deep Lake Vector Store

VectorStore.__init__ Creates an empty VectorStore or loads an existing one if it exists at the specified path.


Vector Store Properties

VectorStore.summary Prints a summary of the dataset VectorStore.tensors Returns the list of tensors present in the dataset VectorStore.__len__ Length of the dataset


Vector Store Operations

VectorStore.add Adding elements to deeplake vector store. VectorStore.search VectorStore search method that combines embedding search, metadata search, and custom TQL search. VectorStore.delete Delete the data in the Vector Store. VectorStore.delete_by_path Deleted the Vector Store at the specified path. VectorStore.update_embedding Recompute existing embeddings of the VectorStore, that match either query, filter, ids or row_ids.


VectorStore.DeepMemory

Creating a Deep Memory if Deep Memory is available on your plan, it will be automatically initialized when you create a Vector Store. DeepMemory.__init__ Based Deep Memory class to train and evaluate models on DeepMemory managed service. Deep Memory Operations DeepMemory.train Train a model on DeepMemory managed service. DeepMemory.cancel Cancel a training job on DeepMemory managed service. DeepMemory.delete Delete a training job on DeepMemory managed service. Deep Memory Properties DeepMemory.status Get the status of a training job on DeepMemory managed service. DeepMemory.list_jobs List all training jobs on DeepMemory managed service.


Creating a Deep Memory

if Deep Memory is available on your plan, it will be automatically initialized when you create a Vector Store. DeepMemory.__init__ Based Deep Memory class to train and evaluate models on DeepMemory managed service.


Deep Memory Operations

DeepMemory.train Train a model on DeepMemory managed service. DeepMemory.cancel Cancel a training job on DeepMemory managed service. DeepMemory.delete Delete a training job on DeepMemory managed service.


Deep Memory Properties

DeepMemory.status Get the status of a training job on DeepMemory managed service. DeepMemory.list_jobs List all training jobs on DeepMemory managed service.


Tensors

Creating Tensors Dataset.create_tensor Creates a new tensor in the dataset. Dataset.create_group Creates a tensor group. Dataset.create_tensor_like Copies the source tensor's meta information and creates a new tensor with it. Deleting and Renaming Tensors Dataset.delete_tensor Delete a tensor from the dataset. Dataset.delete_group Delete a tensor group from the dataset. Dataset.rename_tensor Renames tensor with name name to new_name Dataset.rename_group Renames group with name name to new_name Adding and deleting samples Tensor.append Appends a single sample to the end of the tensor. Tensor.extend Extends the end of the tensor by appending multiple elements from a sequence. Tensor.pop Removes an element at the given index. Tensor.clear Deletes all samples from the tensor Tensor.__setitem__ Update samples with new values. Retrieving samples Tensor.numpy Computes the contents of the tensor in numpy format. Tensor.data Returns data in the tensor in a format based on the tensor's base htype. Tensor.tobytes Returns the bytes of the tensor. Tensor.text Return text data. Tensor.dict Return json data. Tensor.list Return list data. Tensor._linked_sample Returns the linked sample at the given index. Tensor Properties Tensor.htype Htype of the tensor. Tensor.base_htype Base htype of the tensor. Tensor.dtype Dtype of the tensor. Tensor.shape Get the shape of this tensor. Tensor.shape_interval Returns a ShapeInterval object that describes this tensor's shape more accurately. Tensor.ndim Number of dimensions of the tensor. Tensor.num_samples Returns the length of the primary axis of the tensor. Tensor.__len__ Returns the length of the primary axis of the tensor. Tensor.is_dynamic Will return True if samples in this tensor have shapes that are unequal. Tensor.is_sequence Whether this tensor is a sequence tensor. Tensor.is_link Whether this tensor is a link tensor. Tensor.verify Whether linked data will be verified when samples are added. Info Tensor.info Returns the information about the tensor. Tensor.sample_info Returns info about particular samples in a tensor. Video features Tensor.play Play video sample. Tensor.timestamps Returns timestamps (in seconds) for video sample as numpy array.


Deleting and Renaming Tensors

Dataset.delete_tensor Delete a tensor from the dataset. Dataset.delete_group Delete a tensor group from the dataset. Dataset.rename_tensor Renames tensor with name name to new_name Dataset.rename_group Renames group with name name to new_name


Adding and deleting samples

Tensor.append Appends a single sample to the end of the tensor. Tensor.extend Extends the end of the tensor by appending multiple elements from a sequence. Tensor.pop Removes an element at the given index. Tensor.clear Deletes all samples from the tensor Tensor.__setitem__ Update samples with new values.


Tensor Properties

Tensor.htype Htype of the tensor. Tensor.base_htype Base htype of the tensor. Tensor.dtype Dtype of the tensor. Tensor.shape Get the shape of this tensor. Tensor.shape_interval Returns a ShapeInterval object that describes this tensor's shape more accurately. Tensor.ndim Number of dimensions of the tensor. Tensor.num_samples Returns the length of the primary axis of the tensor. Tensor.__len__ Returns the length of the primary axis of the tensor. Tensor.is_dynamic Will return True if samples in this tensor have shapes that are unequal. Tensor.is_sequence Whether this tensor is a sequence tensor. Tensor.is_link Whether this tensor is a link tensor. Tensor.verify Whether linked data will be verified when samples are added.


Creating Tensors

Dataset.create_tensor Creates a new tensor in the dataset. Dataset.create_group Creates a tensor group. Dataset.create_tensor_like Copies the source tensor's meta information and creates a new tensor with it.


Retrieving samples

Tensor.numpy Computes the contents of the tensor in numpy format. Tensor.data Returns data in the tensor in a format based on the tensor's base htype. Tensor.tobytes Returns the bytes of the tensor. Tensor.text Return text data. Tensor.dict Return json data. Tensor.list Return list data. Tensor._linked_sample Returns the linked sample at the given index.


Video features

Tensor.play Play video sample. Tensor.timestamps Returns timestamps (in seconds) for video sample as numpy array.


Info

Tensor.info Returns the information about the tensor. Tensor.sample_info Returns info about particular samples in a tensor.


Htypes

Htype is the class of a tensor: image, bounding box, generic tensor, etc.The htype of a tensor can be specified at its creation Copy to clipboard If not specified, the tensor's htype defaults to 'generic'.Specifying an htype allows for strict settings and error handling, and it is critical for increasing the performance of Deep Lake datasets containing rich data such as images and videos.Supported htypes and their respective defaults are: Htype configs HTYPE DTYPE COMPRESSION generic None None image uint8 Required arg image.rgb uint8 Required arg image.gray uint8 Required arg video uint8 Required arg audio float64 Required arg class_label uint32 None bbox float32 None bbox.3d float32 None intrinsics float32 None segment_mask uint32 None binary_mask bool None keypoints_coco int32 None point int32 None polygon float32 None text str None json Any None list List None dicom None dcm nifti None Required arg point_cloud None las mesh None ply instance_label uint32 None embedding None None link str None sequence None None Image Htype Sample dimensions: (height, width, # channels) or (height, width).  Images can be stored in Deep Lake as compressed bytes or as raw arrays.  Due to the high compression ratio for most image formats, it is highly recommended to store compressed images using the sample_compression input to the create_tensor method.  Creating an image tensor An image tensor can be created using >>> ds.create_tensor('images', htype='image', sample_compression='jpg') Copy to clipboard OR >>> ds.create_tensor('images', htype='image', chunk_compression='jpg') Copy to clipboard Optional args: dtype: Defaults to uint8.  Supported compressions: >>> [None, 'bmp', 'dib', 'gif', 'ico', 'jpeg', 'jpeg2000', 'pcx', 'png', 'ppm', 'sgi', 'tga', 'tiff', ...  'webp', 'wmf', 'xbm', 'eps', 'fli', 'im', 'msp', 'mpo'] Copy to clipboard Appending image samples Image samples can be of type np.ndarray or Deep Lake Sample which can be created using deeplake.read().  Examples Appending pixel data with array >>> ds.images.append(np.zeros((5, 5, 3), dtype=np.uint8)) Copy to clipboard Appening Deep Lake image sample >>> ds.images.append(deeplake.read('images/0001.jpg')) Copy to clipboard You can append multiple samples at the same time using extend().  >>> ds.images.extend([deeplake.read(f'images/000{i}.jpg') for i in range(10)]) Copy to clipboard Note If the compression format of the input sample does not match the sample_compression of the tensor, Deep Lake will decompress and recompress the image for storage, which may significantly slow down the upload process.  The upload process is fastest when the image compression matches the sample_compression.  image.rgb and image.gray htypes image.rgb and image.gray htypes can be used to force your samples to be of RGB or grayscale type.  i.e., if RGB images are appended to an image.gray tensor, Deep Lake will convert them to grayscale and if grayscale images are appended to an image.rgb tensor, Deep Lake will convert them to RGB format


Htypes

Examples Appending index >>> ds.labels.append(0) >>> ds.labels.append(np.zeros((5,), dtype=np.uint32)) Copy to clipboard Extending with list of indices >>> ds.labels.extend([[0, 1, 2], [1, 3]]) Copy to clipboard Appending text labels >>> ds.labels.append(['cars', 'airplanes']) Copy to clipboard Bounding Box Htype Sample dimensions: (# bounding boxes, 4) Bounding boxes have a variety of conventions such as those used in YOLO, COCO, Pascal-VOC and others .  In order for bounding boxes to be correctly displayed by the visualizer, the format of the bounding box must be specified in the coords key in tensor meta information mentioned below.  Creating a bbox tensor A bbox tensor can be created using >>> ds.create_tensor('boxes', htype='bbox', coords={'type': 'fractional', 'mode': 'CCWH'}) Copy to clipboard Optional args: coords: A dictionary with keys 'type' and 'mode'.  type: Specifies the units of bounding box coordinates.  'pixel': is in unit of pixels.  'fractional': is in units relative to the width and height of the image, such as in YOLO format.  mode: Specifies the convention for the 4 coordinates 'LTRB': left_x, top_y, right_x, bottom_y 'LTWH': left_x, top_y, width, height 'CCWH': center_x, center_y, width, height dtype: Defaults to float32.  sample_compression or chunk_compression.  Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set the class names after tensor creation.  >>> ds.boxes.info.update(coords = {'type': 'pixel', 'mode': 'LTRB'}) Copy to clipboard Note If the bounding box format is not specified, the visualizer will assume a YOLO format (fractional + CCWH) if the box coordinates are < 1 on average.  Otherwise, it will assume the COCO format (pixel + LTWH).  Appending bounding boxes Bounding boxes can be appended as np.ndarrays or list or lists of arrays.  Examples Appending one bounding box >>> box array([[462, 123, 238, 98]]) >>> ds.boxes.append(box) Copy to clipboard Appending sample with 3 bounding boxes >>> boxes array([[965, 110, 262, 77], [462, 123, 238, 98], [688, 108, 279, 116]]) >>> boxes.shape (3, 4) >>> ds.boxes.append(boxes) Copy to clipboard 3D Bounding Box Htype In order for 3D bounding boxes to be correctly displayed by the visualizer, the format of the bounding box must be specified in the coords key in tensor meta information mentioned below.  Creating a 3d bbox tensor Note In order for 3D bounding boxes to be correctly displayed by the visualizer, the format of the bounding box must be specified in the coords key in tensor meta information mentioned below.  In addition, for projecting 3D bounding boxes onto 2D data (such as an image), the intrinsics tensor must exist in the dataset, or the intrinsics matrix must be specified in the ds.img_tensor.info dictionary, where the key is 'intrinsics' and the value is the matrix.  A 3d bbox tensor can be created using >>> ds.create_tensor('3d_boxes', htype='bbox.3d', coords={'mode': 'center'}) Copy to clipboard Optional args: coords: A dictionary with key 'mode'.  mode: Specifies the convention for the bbox coordinates


Htypes

Creating a segment_mask tensor A segment_mask tensor can be created using >>> classes = ['background', 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle'] >>> ds.create_tensor('masks', htype='segment_mask', class_names=classes, sample_compression='lz4') Copy to clipboard Optional args: class_names: This must be a list of strings .  tensor.info.class_names will be set to this list.  sample_compression or chunk_compression dtype: Defaults to uint32.  Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set the class names after tensor creation.  >>> ds.labels.info.update(class_names = ['background', 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle']) Copy to clipboard Note Since segmentation masks often contain large amounts of data, it is recommended to compress them using lz4.  Appending segmentation masks Segmentation masks can be appended as np.ndarray.  Examples >>> ds.masks.append(np.zeros((512, 512))) Copy to clipboard Note Since each pixel can only be labeled once, segmentation masks are not appropriate for datasets where objects might overlap, or where multiple objects within the same class must be distinguished.  For these use cases, please use htype = 'binary_mask'.  Binary Mask Htype Sample dimensions: (height, width, # objects in a sample) Binary masks are similar to segmentation masks, except that each object is represented by a channel in the mask.  Each channel in the mask encodes values for a single object.  A pixel in a mask channel should have a value of 1 if the pixel of the image belongs to this object and 0 otherwise.  The labels corresponding to the channels should be stored in an adjacent tensor of htype class_label, in which the number of labels at a given index is equal to the number of objects (number of channels) in the binary mask.  Creating a binary_mask tensor A binary_mask tensor can be created using >>> ds.create_tensor('masks', htype='binary_mask', sample_compression='lz4') Copy to clipboard Optional args: ref:sample_compression <sample_compression> or chunk_compression dtype: Defaults to bool.  Supported compressions: >>> ['lz4'] Copy to clipboard Note Since segmentation masks often contain large amounts of data, it is recommended to compress them using lz4.  Appending binary masks Binary masks can be appended as np.ndarray.  Examples Appending a binary mask with 5 objects >>> ds.masks.append(np.zeros((512, 512, 5), dtype='bool')) >>> ds.labels.append(['aeroplane', 'aeroplane', 'bottle', 'bottle', 'bird']) Copy to clipboard COCO Keypoints Htype Sample dimensions: (3 x # keypoints, # objects in a sample) COCO keypoints are a convention for storing points of interest in an image.  Each keypoint consists of 3 values: x - coordinate, y - coordinate and v - visibility.  A set of K keypoints of an object is represented as: [x1, y1, v1, x2, y2, v2, …, xk, yk, vk] The visibility v can be one of three values: 0 keypoint not in image.  1 keypoint in image but not visible.  2 keypoint in image and visible.  Creating a keypoints_coco tensor A keypoints_coco tensor can be created using >>> ds.create_tensor('keypoints', htype='keypoints_coco', keypoints=['knee', 'elbow', 'head'], connections=[[0, 1], [1, 2]]) Copy to clipboard Optional args: keypoints: List of strings describing the i th keypoint


Htypes

image.rgb and image.gray tensors can be created using >>> ds.create_tensor('rgb_images', htype='image.rgb', sample_compression='...') Copy to clipboard >>> ds.create_tensor('gray_images', htype='image.gray', sample_compression='...') Copy to clipboard Video Htype Sample dimensions: (# frames, height, width, # channels) or (# frames, height, width) Creating a video tensor A video tensor can be created using >>> ds.create_tensor('videos', htype='video', sample_compression='mp4') Copy to clipboard Optional args: dtype: Defaults to uint8 .  Supported compressions: >>> [None, 'mp4', 'mkv', 'avi'] Copy to clipboard Appending video samples Video samples can be of type np.ndarray or Sample which is returned by deeplake.read().  Deep Lake does not support compression of raw video frames.  Therefore, array of raw frames can only be appended to tensors with None compression.  Recompression of samples read with deeplake.read is also not supported.  Examples Appending Deep Lake video sample >>> ds.videos.append(deeplake.read('videos/0012.mp4')) Copy to clipboard Extending with multiple videos >>> ds.videos.extend([deeplake.read(f'videos/00{i}.mp4') for i in range(10)]) Copy to clipboard Audio Htype Sample dimensions: (# samples in audio, # channels) or (# samples in audio,) Creating an audio tensor An audio tensor can be created using >>> ds.create_tensor('audios', htype='audio', sample_compression='mp3') Copy to clipboard Optional args: dtype: Defaults to float64.  Supported compressions: >>> [None, 'mp3', 'wav', 'flac'] Copy to clipboard Appending audio samples Audio samples can be of type np.ndarray or Sample which is returned by deeplake.read().  Like videos, Deep Lake does not support compression or recompression of input audio samples.  Thus, samples of type np.ndarray can only be appended to tensors with None compression.  Examples Appending Deep Lake audio sample >>> ds.audios.append(deeplake.read('audios/001.mp3')) Copy to clipboard Extending with Deep Lake audio samples >>> ds.audios.extend([deeplake.read(f'videos/00{i}.mp3') for i in range(10)]) Copy to clipboard Class Label Htype Sample dimensions: (# labels,) Class labels are stored as numerical values in tensors, which are indices of the list tensor.info.class_names.  Creating a class label tensor A class label tensor can be created using >>> classes = ['airplanes', 'cars', 'birds', 'cats', 'deer', 'dogs', 'frogs', 'horses', 'ships', 'trucks'] >>> ds.create_tensor('labels', htype='class_label', class_names=classes, chunk_compression='lz4') Copy to clipboard Optional args: class_names: This must be a list of strings.  tensor.info.class_names will be set to this list.  sample_compression or chunk_compression.  dtype: Defaults to uint32.  Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set the class names after tensor creation.  >>> ds.labels.info.update(class_names = ['airplanes', 'cars', 'birds', 'cats', 'deer', 'dogs', 'frogs', 'horses', 'ships', 'trucks']) Copy to clipboard Note If specifying compression, since the number of labels in one sample will be too low, chunk_compression would be the better option to use.  Appending class labels Class labels can be appended as int, str, np.ndarray or list of int or str.  In case of strings, tensor.info.class_names is updated automatically


Htypes

tensor.info.keypoints will be set to this list .  connections: List of strings describing which points should be connected by lines in the visualizer.  sample_compression or chunk_compression dtype: Defaults to int32.  Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set keypoints and / or connections after tensor creation.  >>> ds.keypoints.info.update(keypoints = ['knee', 'elbow',...]) >>> ds.keypoints.info.update(connections = [[0,1], [2,3], ...]) Copy to clipboard Appending keypoints Keypoints can be appended as np.ndarray or list.  Examples Appending keypoints sample with 3 keypoints and 4 objects >>> ds.keypoints.update(keypoints = ['left ear', 'right ear', 'nose']) >>> ds.keypoints.update(connections = [[0, 2], [1, 2]]) >>> kp_arr array([[465, 398, 684, 469], [178, 363, 177, 177], [ 2, 2, 2, 1], [454, 387, 646, 478], [177, 322, 137, 161], [ 2, 2, 2, 2], [407, 379, 536, 492], [271, 335, 150, 143], [ 2, 1, 2, 2]]) >>> kp_arr.shape (9, 4) >>> ds.keypoints.append(kp_arr) Copy to clipboard Warning In order to correctly use the keypoints and connections metadata, it is critical that all objects in every sample have the same number of K keypoints in the same order.  For keypoints that are not present in an image, they can be stored with dummy coordinates of x = 0, y = 0, and v = 0, and the visibility will prevent them from being drawn in the visualizer.  Point Htype Sample dimensions: (# points, 2) in case of 2-D (X, Y) co-ordinates or (# points, 3) in case of 3-D (X, Y, Z) co-ordinates of the point.  Points does not contain a fixed mapping across samples between the point order and real-world objects (i.e., point 0 is an elbow, point 1 is a knee, etc.).  If you require such a mapping, use COCO Keypoints Htype.  Creating a point tensor A point tensor can be created using >>> ds.create_tensor('points', htype='point', sample_compression=None) Copy to clipboard Optional args: sample_compression or chunk_compression dtype: Defaults to int32.  Supported compressions: >>> ['lz4'] Copy to clipboard Appending point samples Points can be appended as np.ndarray or list.  Examples Appending 2 2-D points >>> ds.points.append([[0, 1], [1, 3]]) Copy to clipboard Appending 2 3-D points >>> ds.points.append(np.zeros((2, 3))) Copy to clipboard Polygon Htype Sample dimensions: (# polygons, # points per polygon, # co-ordinates per point) Each sample in a tensor of polygon htype is a list of polygons.  Each polygon is a list / array of points.  All points in a sample should have the same number of co-ordinates (eg., cannot mix 2-D points with 3-D points).  Different samples can have different number of polygons.  Different polygons can have different number of points.  Creating a polygon tensor A polygon tensor can be created using >>> ds.create_tensor('polygons', htype='polygon', sample_compression=None) Copy to clipboard Optional args: sample_compression or chunk_compression dtype: Defaults to float32.  Supported compressions: >>> ['lz4'] Copy to clipboard Appending polygons Polygons can be appended as a list of list of tuples or np.ndarray


Htypes

'center': [center_x, center_y, center_z, size_x, size_y, size_z, rot_x, rot_y, rot_z] Sample dimensions: (# bounding boxes, 9) size_x - is the length of the bounding box along x direction size_y - is the width of the bounding box along y direction size_z - is the height of the bounding box along z direction rot_x - rotation angle along x axis, given in degrees rot_y - rotation angle along y axis, given in degrees rot_z - rotation angle along z axis, given in degrees 'vertex': 8 3D vertices - [[x0, y0, z0], [x1, y1, z1], [x2, y2, z2], ….., [x7, y7, z7]] Sample dimensions: (# bounding boxes, 8, 3) The vertex order is of the following form: 4_____________________ 5 /| /| / | / | / | / | /___|_________________/ | 0| | | 1 | | | | | | | | | | | | | | |_________________|___| | / 7 | / 6 | / | / | / | / |/_____________________|/ 3 2 Copy to clipboard dtype: Defaults to float32 .  sample_compression or chunk_compression.  Supported compressions: >>> ['lz4'] Copy to clipboard Note rotation angles are specified in degrees, not radians Appending 3d bounding boxes Bounding boxes can be appended as np.ndarrays or list or lists of arrays.  Examples Appending one bounding box >>> box array([[462, 123, 238, 98, 22, 36, 44, 18, 0, 36, 0]]) >>> ds.3d_boxes.append(box) Copy to clipboard Appending sample with 3 bounding boxes >>> boxes array([[965, 110, 262, 77, 22, 36, 44, 18, 0, 28, 0], [462, 123, 238, 98, 26, 34, 24, 19, 0, -50, 0], [688, 108, 279, 116, 12, 32, 14, 38, 0, 30, 0]]) >>> boxes.shape (9, 4) >>> ds.3d_boxes.append(boxes) Copy to clipboard Intrinsics Htype Sample dimensions: (# intrinsics matrices, 3, 3) The intrinsic matrix represents a projective transformation from the 3-D camera's coordinates into the 2-D image coordinates.  The intrinsic parameters include the focal length, the optical center, also known as the principal point.  The camera intrinsic matrix, K, is defined as: [fx0cx0fycy001] [cx,cy] - Optical center (the principal point), in pixels.  [fx,fy] - Focal length in pixels.  fx=F/px fy=F/py F - Focal length in world units, typically expressed in millimeters.  (px,py) - Size of the pixel in world units.  Creating an intrinsics tensor An intrinsics tensor can be created using >>> ds.create_tensor('intrinsics', htype='intrinsics') Copy to clipboard Optional args: sample_compression or chunk_compression.  dtype: Defaults to float32.  Supported compressions: >>> ['lz4'] Copy to clipboard Appending intrinsics matrices >>> intrinsic_params = np.zeros((3, 3)) >>> ds.intrinsics.append(intrinsic_params) Copy to clipboard Segmentation Mask Htype Sample dimensions: (height, width) Segmentation masks are 2D representations of class labels where the numerical label data is encoded in an array of same shape as the image.  The numerical values are indices of the list tensor.info.class_names


Htypes

Different meshes can have different number of points .  Creating a mesh tensor A mesh tensor can be created using >>> ds.create_tensor('mesh', htype='mesh', sample_compression='ply') Copy to clipboard Optional args: sample_compression Supported compressions: >>> ['ply'] Copy to clipboard Appending meshes Examples Appending a ply file containing a mesh data to tensor >>> import deeplake as dp >>> sample = dp.read('example.ply') # mesh with 100 points and 200 faces >>> ds.mesh.append(sample) Copy to clipboard >>> ds.mesh.shape >>> (1, 100, 3) Copy to clipboard Embedding Htype Sample dimensions: (# elements in the embedding,) Creating an embedding tensor An embedding tensor can be created using >>> ds.create_tensor('embedding', htype='embedding') Copy to clipboard Supported compressions: >>> ['lz4', None] Copy to clipboard Appending embedding samples Embedding samples can be of type np.ndarray.  Examples Appending Deep Lake embedding sample >>> ds.embedding.append(np.random.uniform(low=-1, high=1, size=(1024))) Copy to clipboard Extending with Deep Lake embeddding samples >>> ds.embedding.extend([np.random.uniform(low=-1, high=1, size=(1024)) for i in range(10)]) Copy to clipboard Sequence htype A special meta htype for tensors where each sample is a sequence.  The items in the sequence are samples of another htype.  It is a wrapper htype that can wrap other htypes like sequence[image], sequence[video], sequence[text], etc.  Examples >>> ds.create_tensor('seq', htype='sequence') >>> ds.seq.append([1, 2, 3]) >>> ds.seq.append([4, 5, 6]) >>> ds.seq.numpy() array([[[1], [2], [3]], [[4], [5], [6]]]) Copy to clipboard >>> ds.create_tensor('image_seq', htype='sequence[image]', sample_compression='jpg') >>> ds.image_seq.append([deeplake.read('img01.jpg'), deeplake.read('img02.jpg')]) Copy to clipboard Link htype Link htype is a special meta htype that allows linking of external data (files) to the dataset, without storing the data in the dataset itself.  Moreover, there can be variations in this htype, such as link[image], link[video], link[audio], etc.  that would enable the activeloop visualizer to correctly display the data.  No data is actually loaded until you try to read the sample from a dataset.  There are a few exceptions to this:- If create_shape_tensor=True was specified during create_tensor of the tensor to which this is being added, the shape of the sample is read.  This is True by default.  If create_sample_info_tensor=True was specified during create_tensor of the tensor to which this is being added, the sample info is read.  This is True by default.  If verify=True was specified during create_tensor of the tensor to which this is being added, some metadata is read from them to verify the integrity of the link samples.  This is True by default.  If you do not want to verify your links, all three of verify, create_shape_tensor and create_sample_info_tensor have to be set to False