P

parent (deeplake.core.dataset.Dataset property) PartialSample (class in deeplake.core.partial_sample) path() (deeplake.core.tensor.Tensor method) PathNotEmptyException (class in deeplake.util.exceptions) pending_commit_id (deeplake.core.dataset.Dataset property) pil (deeplake.core.sample.Sample property) Pipeline (class in deeplake.core.transform) play() (deeplake.core.tensor.Tensor method) pop() (deeplake.api.info.Info method) (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) popitem() (deeplake.api.info.Info method) populate_creds() (deeplake.core.dataset.Dataset method) ProviderListEmptyError (class in deeplake.util.exceptions) ProviderSizeListMismatch (class in deeplake.util.exceptions) pytorch() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method)


S

S3DeletionError (class in deeplake.util.exceptions) S3Error (class in deeplake.util.exceptions) S3GetError (class in deeplake.util.exceptions) S3ListError (class in deeplake.util.exceptions) S3Provider (class in deeplake.core.storage) S3SetError (class in deeplake.util.exceptions) SamePathException (class in deeplake.util.exceptions) Sample (class in deeplake.core.sample) sample_by() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method) sample_indices (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) sample_info (deeplake.core.tensor.Tensor property) SampleCompressionError (class in deeplake.util.exceptions) SampleDecompressionError (class in deeplake.util.exceptions) SampleHtypeMismatchError (class in deeplake.util.exceptions) save_view() (deeplake.core.dataset.Dataset method) ServerException (class in deeplake.util.exceptions) set_bytes() (deeplake.core.storage.StorageProvider method) setdefault() (deeplake.api.info.Info method) shape (deeplake.core.tensor.Tensor property) shape_interval (deeplake.core.tensor.Tensor property) ShapeInterval (class in deeplake.util.shape_interval) shuffle() (deeplake.enterprise.DeepLakeDataLoader method) size_approx() (deeplake.core.dataset.Dataset method) slice_at_int() (in module deeplake.core.index) slice_length() (in module deeplake.core.index) StorageProvider (class in deeplake.core.storage) structure() (deeplake.auto.unstructured.image_classification.ImageClassification method) StructuredDataset (class in deeplake.auto.structured.base) subscriptable() (deeplake.core.index.IndexEntry method) summary() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) sync() (deeplake.core.storage.GDriveProvider method)


T

Tensor (class in deeplake.core.tensor) TensorAlreadyExistsError (class in deeplake.util.exceptions) TensorDoesNotExistError (class in deeplake.util.exceptions) TensorDtypeMismatchError (class in deeplake.util.exceptions) tensorflow() (deeplake.core.dataset.Dataset method) TensorGroupAlreadyExistsError (class in deeplake.util.exceptions) TensorGroupDoesNotExistError (class in deeplake.util.exceptions) TensorInvalidSampleShapeError (class in deeplake.util.exceptions) TensorMetaInvalidHtype (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteKey (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteValue (class in deeplake.util.exceptions) TensorMetaMissingKey (class in deeplake.util.exceptions) TensorMetaMissingRequiredValue (class in deeplake.util.exceptions) TensorMetaMutuallyExclusiveKeysError (class in deeplake.util.exceptions) TensorMismatchError (class in deeplake.util.exceptions) TensorModifiedError (class in deeplake.util.exceptions) tensors (deeplake.core.dataset.Dataset property) TensorUnsupportedSampleType (class in deeplake.util.exceptions) text() (deeplake.core.tensor.Tensor method) tiled() (in module deeplake) (in module deeplake.api.tiled) timestamps (deeplake.core.tensor.Tensor property) tobytes() (deeplake.core.tensor.Tensor method) token (deeplake.core.dataset.Dataset property) (deeplake.core.dataset.DeepLakeCloudDataset property) TokenPermissionError (class in deeplake.util.exceptions) transform() (deeplake.enterprise.DeepLakeDataLoader method) TransformError (class in deeplake.util.exceptions)


U

UnableToReadFromUrlError (class in deeplake.util.exceptions) uncompressed_bytes() (deeplake.core.sample.Sample method) UnexpectedStatusCodeException (class in deeplake.util.exceptions) UnstructuredDataset (class in deeplake.auto.unstructured.base) UnsupportedCompressionError (class in deeplake.util.exceptions) UnsupportedSchedulerError (class in deeplake.util.exceptions) UnsupportedTensorTypeError (class in deeplake.util.exceptions) update() (deeplake.api.info.Info method) update_creds_key() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) UserNotLoggedInException (class in deeplake.util.exceptions)


R

read() (in module deeplake) (in module deeplake.api.read) read_only (deeplake.core.dataset.Dataset property) ReadOnlyModeError (class in deeplake.util.exceptions) rechunk() (deeplake.core.dataset.Dataset method) register_deeplake_object() (deeplake.core.storage.LRUCache method) remove_deeplake_object() (deeplake.core.storage.LRUCache method) remove_memory_cache() (in module deeplake.util.remove_cache) rename() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.S3Provider method) (in module deeplake) rename_group() (deeplake.core.dataset.Dataset method) rename_tensor() (deeplake.core.dataset.Dataset method) RenameError (class in deeplake.util.exceptions) replace_with() (deeplake.api.info.Info method) reset() (deeplake.core.dataset.Dataset method) ResourceNotFoundException (class in deeplake.util.exceptions) root (deeplake.core.dataset.Dataset property)


Q

query() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method)


V

validate() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) values() (deeplake.api.info.Info method) verify (deeplake.core.tensor.Tensor property) VersionControlError (class in deeplake.util.exceptions) ViewEntry (class in deeplake.core.dataset) visualize() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method)


W

WaitTimeoutException (class in deeplake.util.exceptions)


Python Module Index

d d deeplake deeplake.api.info deeplake.integrations.wandb.wandb


Installation¶

Deep Lake can be installed with pip Copy to clipboard Deep Lake has the following extras that you can choose to install according to your needs. Htype configs¶ Install command Description Dependencies installed pip install 'deeplake[av]' Audio and video support via PyAV av pip install 'deeplake[visualizer]' Visualize Deep Lake datasets within notebooks. This is required for Dataset.visualize to work. IPython, flask pip install 'deeplake[gcp]' GCS support google-cloud-storage, google-auth, google-auth-oauthlib pip install 'deeplake[dicom]' DICOM data support pydicom pip install 'deeplake[gdrive]' Google Drive support google-api-python-client, oauth2client, google-auth, google-auth-oauthlib pip install 'deeplake[point_cloud]' Support for LiDAR point cloud data laspy pip install 'deeplake[all]' Installs all of the above


Datasets¶

Creating Datasets¶ deeplake.dataset Returns a Dataset object referencing either a new or existing dataset.  deeplake.empty Creates an empty dataset deeplake.like Creates a new dataset by copying the source dataset's structure to a new location.  deeplake.ingest Ingests a dataset from a source and stores it as a structured dataset to destination.  deeplake.ingest_kaggle Download and ingest a kaggle dataset and store it as a structured dataset to destination.  deeplake.ingest_dataframe Convert pandas dataframe to a Deep Lake Dataset.  deeplake.ingest_huggingface Converts Hugging Face datasets to Deep Lake format.  Loading Datasets¶ deeplake.load Loads an existing dataset Deleting and Renaming Datasets¶ deeplake.delete Deletes a dataset at a given path.  deeplake.rename Renames dataset at old_path to new_path.  Copying Datasets¶ deeplake.copy Copies dataset at src to dest.  deeplake.deepcopy Copies dataset at src to dest including version control history.  Dataset Operations¶ Dataset.summary Prints a summary of the dataset.  Dataset.append Append samples to mutliple tensors at once.  Dataset.extend Appends multiple rows of samples to mutliple tensors at once.  Dataset.query Returns a sliced Dataset with given query results.  Dataset.copy Copies this dataset or dataset view to dest.  Dataset.delete Deletes the entire dataset from the cache layers (if any) and the underlying storage.  Dataset.rename Renames the dataset to path.  Dataset.visualize Visualizes the dataset in the Jupyter notebook.  Dataset.pop Removes a sample from all the tensors of the dataset.  Dataset.rechunk Rewrites the underlying chunks to make their sizes optimal.  Dataset.flush Necessary operation after writes if caches are being used.  Dataset.clear_cache Flushes (see Dataset.flush()) the contents of the cache layers (if any) and then deletes contents of all the layers of it.  Dataset.size_approx Estimates the size in bytes of the dataset.  Dataset Visualization¶ Dataset.visualize Visualizes the dataset in the Jupyter notebook.  Dataset Credentials¶ Dataset.add_creds_key Adds a new creds key to the dataset.  Dataset.populate_creds Populates the creds key added in add_creds_key with the given creds.  Dataset.update_creds_key Replaces the old creds key with the new creds key.  Dataset.change_creds_management Changes the management status of the creds key.  Dataset.get_creds_keys Returns the list of creds keys added to the dataset.  Dataset Properties¶ Dataset.tensors All tensors belonging to this group, including those within sub groups.  Dataset.groups All sub groups in this group Dataset.num_samples Returns the length of the smallest tensor.  Dataset.read_only Returns True if dataset is in read-only mode and False otherwise.  Dataset.info Returns the information about the dataset.  Dataset Version Control¶ Dataset.commit Stores a snapshot of the current state of the dataset.  Dataset.diff Returns/displays the differences between commits/branches.  Dataset.checkout Checks out to a specific commit_id or branch.  Dataset.merge Merges the target_id into the current dataset.  Dataset.log Displays the details of all the past commits.  Dataset.reset Resets the uncommitted changes present in the branch.  Dataset.get_commit_details Get details of a particular commit.  Dataset.commit_id The lasted committed commit id of the dataset


Datasets¶

Dataset.branch The current branch of the dataset Dataset.pending_commit_id The commit_id of the next commit that will be made to the dataset .  Dataset.has_head_changes Returns True if currently at head node and uncommitted changes are present.  Dataset.commits Lists all the commits leading to the current dataset state.  Dataset.branches Lists all the branches of the dataset.  Dataset Views¶ A dataset view is a subset of a dataset that points to specific samples (indices) in an existing dataset.  Dataset views can be created by indexing a dataset, filtering a dataset with Dataset.filter() or querying a dataset with Dataset.query().  Filtering is done with user-defined functions or simplified expressions whereas query can perform SQL-like queries with our Tensor Query Language.  See the full TQL spec here.  Dataset views can only be saved when a dataset has been committed and has no changes on the HEAD node, in order to preserve data lineage and prevent the underlying data from changing after the query or filter conditions have been evaluated.  Example >>> import deeplake >>> # load dataset >>> ds = deeplake.load('hub://activeloop/mnist-train') >>> # filter dataset >>> zeros = ds.filter('labels == 0') >>> # save view >>> zeros.save_view(id='zeros') >>> # load_view >>> zeros = ds.load_view(id='zeros') >>> len(zeros) 5923 Copy to clipboard Dataset.query Returns a sliced Dataset with given query results.  Dataset.filter Filters the dataset in accordance of filter function f(x: sample) -> bool Dataset.save_view Saves a dataset view as a virtual dataset (VDS) Dataset.get_view Returns the dataset view corresponding to id.  Dataset.load_view Loads the view and returns the Dataset by id.  Dataset.delete_view Deletes the view with given view id.  Dataset.get_views Returns list of views stored in this Dataset.  Dataset.is_view Returns True if this dataset is a view and False otherwise.


Creating Datasets¶

deeplake.dataset Returns a Dataset object referencing either a new or existing dataset. deeplake.empty Creates an empty dataset deeplake.like Creates a new dataset by copying the source dataset's structure to a new location. deeplake.ingest Ingests a dataset from a source and stores it as a structured dataset to destination. deeplake.ingest_kaggle Download and ingest a kaggle dataset and store it as a structured dataset to destination. deeplake.ingest_dataframe Convert pandas dataframe to a Deep Lake Dataset. deeplake.ingest_huggingface Converts Hugging Face datasets to Deep Lake format.


Loading Datasets¶

deeplake.load Loads an existing dataset


Copying Datasets¶

deeplake.copy Copies dataset at src to dest. deeplake.deepcopy Copies dataset at src to dest including version control history.


Dataset Operations¶

Dataset.summary Prints a summary of the dataset. Dataset.append Append samples to mutliple tensors at once. Dataset.extend Appends multiple rows of samples to mutliple tensors at once. Dataset.query Returns a sliced Dataset with given query results. Dataset.copy Copies this dataset or dataset view to dest. Dataset.delete Deletes the entire dataset from the cache layers (if any) and the underlying storage. Dataset.rename Renames the dataset to path. Dataset.visualize Visualizes the dataset in the Jupyter notebook. Dataset.pop Removes a sample from all the tensors of the dataset. Dataset.rechunk Rewrites the underlying chunks to make their sizes optimal. Dataset.flush Necessary operation after writes if caches are being used. Dataset.clear_cache Flushes (see Dataset.flush()) the contents of the cache layers (if any) and then deletes contents of all the layers of it. Dataset.size_approx Estimates the size in bytes of the dataset.


Dataset Credentials¶

Dataset.add_creds_key Adds a new creds key to the dataset. Dataset.populate_creds Populates the creds key added in add_creds_key with the given creds. Dataset.update_creds_key Replaces the old creds key with the new creds key. Dataset.change_creds_management Changes the management status of the creds key. Dataset.get_creds_keys Returns the list of creds keys added to the dataset.


Dataset Version Control¶

Dataset.commit Stores a snapshot of the current state of the dataset. Dataset.diff Returns/displays the differences between commits/branches. Dataset.checkout Checks out to a specific commit_id or branch. Dataset.merge Merges the target_id into the current dataset. Dataset.log Displays the details of all the past commits. Dataset.reset Resets the uncommitted changes present in the branch. Dataset.get_commit_details Get details of a particular commit. Dataset.commit_id The lasted committed commit id of the dataset. Dataset.branch The current branch of the dataset Dataset.pending_commit_id The commit_id of the next commit that will be made to the dataset. Dataset.has_head_changes Returns True if currently at head node and uncommitted changes are present. Dataset.commits Lists all the commits leading to the current dataset state. Dataset.branches Lists all the branches of the dataset.


Dataset Visualization¶

Dataset.visualize Visualizes the dataset in the Jupyter notebook.


Dataset Properties¶

Dataset.tensors All tensors belonging to this group, including those within sub groups. Dataset.groups All sub groups in this group Dataset.num_samples Returns the length of the smallest tensor. Dataset.read_only Returns True if dataset is in read-only mode and False otherwise. Dataset.info Returns the information about the dataset.


Dataset Views¶

A dataset view is a subset of a dataset that points to specific samples (indices) in an existing dataset. Dataset views can be created by indexing a dataset, filtering a dataset with Dataset.filter() or querying a dataset with Dataset.query(). Filtering is done with user-defined functions or simplified expressions whereas query can perform SQL-like queries with our Tensor Query Language. See the full TQL spec here.Dataset views can only be saved when a dataset has been committed and has no changes on the HEAD node, in order to preserve data lineage and prevent the underlying data from changing after the query or filter conditions have been evaluated.Example Copy to clipboard Dataset.query Returns a sliced Dataset with given query results. Dataset.filter Filters the dataset in accordance of filter function f(x: sample) -> bool Dataset.save_view Saves a dataset view as a virtual dataset (VDS) Dataset.get_view Returns the dataset view corresponding to id. Dataset.load_view Loads the view and returns the Dataset by id. Dataset.delete_view Deletes the view with given view id. Dataset.get_views Returns list of views stored in this Dataset. Dataset.is_view Returns True if this dataset is a view and False otherwise.


Deleting and Renaming Datasets¶

deeplake.delete Deletes a dataset at a given path. deeplake.rename Renames dataset at old_path to new_path.


Tensors¶

Creating Tensors¶ Dataset.create_tensor Creates a new tensor in the dataset. Dataset.create_group Creates a tensor group. Dataset.create_tensor_like Copies the source tensor's meta information and creates a new tensor with it. Deleting and Renaming Tensors¶ Dataset.delete_tensor Delete a tensor from the dataset. Dataset.delete_group Delete a tensor group from the dataset. Dataset.rename_tensor Renames tensor with name name to new_name Dataset.rename_group Renames group with name name to new_name Adding and deleting samples¶ Tensor.append Appends a single sample to the end of the tensor. Tensor.extend Extends the end of the tensor by appending multiple elements from a sequence. Tensor.pop Removes an element at the given index. Tensor.clear Deletes all samples from the tensor Tensor.__setitem__ Update samples with new values. Retrieving samples¶ Tensor.numpy Computes the contents of the tensor in numpy format. Tensor.data Returns data in the tensor in a format based on the tensor's base htype. Tensor.tobytes Returns the bytes of the tensor. Tensor.text Return text data. Tensor.dict Return json data. Tensor.list Return list data. Tensor._linked_sample Returns the linked sample at the given index. Tensor Properties¶ Tensor.htype Htype of the tensor. Tensor.base_htype Base htype of the tensor. Tensor.dtype Dtype of the tensor. Tensor.shape Get the shape of this tensor. Tensor.shape_interval Returns a ShapeInterval object that describes this tensor's shape more accurately. Tensor.ndim Number of dimensions of the tensor. Tensor.num_samples Returns the length of the primary axis of the tensor. Tensor.__len__ Returns the length of the primary axis of the tensor. Tensor.is_dynamic Will return True if samples in this tensor have shapes that are unequal. Tensor.is_sequence Whether this tensor is a sequence tensor. Tensor.is_link Whether this tensor is a link tensor. Tensor.verify Whether linked data will be verified when samples are added. Info¶ Tensor.info Returns the information about the tensor. Tensor.sample_info Returns info about particular samples in a tensor. Video features¶ Tensor.play Play video sample. Tensor.timestamps Returns timestamps (in seconds) for video sample as numpy array.


Creating Tensors¶

Dataset.create_tensor Creates a new tensor in the dataset. Dataset.create_group Creates a tensor group. Dataset.create_tensor_like Copies the source tensor's meta information and creates a new tensor with it.


Adding and deleting samples¶

Tensor.append Appends a single sample to the end of the tensor. Tensor.extend Extends the end of the tensor by appending multiple elements from a sequence. Tensor.pop Removes an element at the given index. Tensor.clear Deletes all samples from the tensor Tensor.__setitem__ Update samples with new values.


Retrieving samples¶

Tensor.numpy Computes the contents of the tensor in numpy format. Tensor.data Returns data in the tensor in a format based on the tensor's base htype. Tensor.tobytes Returns the bytes of the tensor. Tensor.text Return text data. Tensor.dict Return json data. Tensor.list Return list data. Tensor._linked_sample Returns the linked sample at the given index.


Deleting and Renaming Tensors¶

Dataset.delete_tensor Delete a tensor from the dataset. Dataset.delete_group Delete a tensor group from the dataset. Dataset.rename_tensor Renames tensor with name name to new_name Dataset.rename_group Renames group with name name to new_name


Tensor Properties¶

Tensor.htype Htype of the tensor. Tensor.base_htype Base htype of the tensor. Tensor.dtype Dtype of the tensor. Tensor.shape Get the shape of this tensor. Tensor.shape_interval Returns a ShapeInterval object that describes this tensor's shape more accurately. Tensor.ndim Number of dimensions of the tensor. Tensor.num_samples Returns the length of the primary axis of the tensor. Tensor.__len__ Returns the length of the primary axis of the tensor. Tensor.is_dynamic Will return True if samples in this tensor have shapes that are unequal. Tensor.is_sequence Whether this tensor is a sequence tensor. Tensor.is_link Whether this tensor is a link tensor. Tensor.verify Whether linked data will be verified when samples are added.


Video features¶

Tensor.play Play video sample. Tensor.timestamps Returns timestamps (in seconds) for video sample as numpy array.


Info¶

Tensor.info Returns the information about the tensor. Tensor.sample_info Returns info about particular samples in a tensor.


Compressions¶

Deep Lake can read, compress, decompress and recompress data to different formats. The supported htype-compression configurations are given below. Sample Type Htype Compressions Image image bmp, dib, gif, ico, jpeg, jpeg2000, pcx, png, ppm, sgi, tga, tiff, webp, wmf, xbm, eps, fli, im, msp, mpo, apng Video video mp4, mkv, avi Audio audio flac, mp3, wav Dicom dicom dcm Point Cloud point_cloud las Other bbox, text, list, json, generic, etc. lz4 Sample Compression¶ If sample compression is specified when creating tensors, samples will be compressed to the given format if possible. If given data is already compressed and matches the provided sample_compression, it will be stored as is. If left as None, given samples are uncompressed. Note For audio and video, we don't support compressing raw frames but only reading compressed audio and video data. Examples: >>> ds.create_tensor('images', htype='image', sample_compression='jpg') >>> ds.create_tensor('videos', htype='video', sample_compression='mp4') >>> ds.create_tensor('point_clouds', htype='point_cloud', sample_compression='las') Copy to clipboard Structure of sample-wise compressed tensor.¶ Chunk Compression¶ If chunk compression is specified when creating tensors, addded samples will be clubbed together and compressed to the given format chunk-wise. If given data is already compressed, it will be uncompressed and then recompressed chunk-wise. Note Chunk-wise compression is not supported for audio, video and point_cloud htypes. Examples: >>> ds.create_tensor('images', htype='image', chunk_compression='jpg') >>> ds.create_tensor('boxes', htype='bbox', chunk_compression='lz4') Copy to clipboard Structure of chunk-wise compressed tensor.¶ Note See deeplake.read() to learn how to read data from files and populate these tensors.


Sample Compression¶

If sample compression is specified when creating tensors, samples will be compressed to the given format if possible. If given data is already compressed and matches the provided sample_compression, it will be stored as is. If left as None, given samples are uncompressed.NoteFor audio and video, we don't support compressing raw frames but only reading compressed audio and video data.Examples: Copy to clipboard Structure of sample-wise compressed tensor.¶


Chunk Compression¶

If chunk compression is specified when creating tensors, addded samples will be clubbed together and compressed to the given format chunk-wise. If given data is already compressed, it will be uncompressed and then recompressed chunk-wise.NoteChunk-wise compression is not supported for audio, video and point_cloud htypes.Examples: Copy to clipboard Structure of chunk-wise compressed tensor.¶ NoteSee deeplake.read() to learn how to read data from files and populate these tensors.


PyTorch and Tensorflow Support¶

Deep Lake datasets can be easily converted to Torch dataloaders or Tensorflow datasets for training. Dataset.pytorch Converts the dataset into a pytorch Dataloader. Dataset.tensorflow Converts the dataset into a tensorflow compatible format.


Checking if the site connection is secure

Enable JavaScript and cookies to continue


deeplake.client.log¶

Deep Lake does logging using the 'deeplake' logger. Logging level is logging.INFO by default. See example on how to change this. Copy to clipboard


Index


_

__contains__() (deeplake.core.storage.GCSProvider method) __delitem__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) __getitem__() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) __getstate__() (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) __init__() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) (deeplake.core.sample.Sample method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.util.shape_interval.ShapeInterval method) __iter__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) __len__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) (deeplake.core.tensor.Tensor method) __repr__() (deeplake.core.index.Index method) __setitem__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) (deeplake.core.tensor.Tensor method) __setstate__() (deeplake.core.storage.LRUCache method) __str__() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) __weakref__ (deeplake.core.index.Index attribute) (deeplake.core.index.IndexEntry attribute) (deeplake.core.storage.StorageProvider attribute) _all_keys() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) _check_is_file() (deeplake.core.storage.LocalProvider method) _check_update_creds() (deeplake.core.storage.S3Provider method) _config (deeplake.core.tensor.Tensor property) _flush_if_not_read_only() (deeplake.core.storage.LRUCache method) _forward() (deeplake.core.storage.LRUCache method) _forward_value() (deeplake.core.storage.LRUCache method) _free_up_space() (deeplake.core.storage.LRUCache method) _insert_in_cache() (deeplake.core.storage.LRUCache method) _is_hub_path (deeplake.core.storage.StorageProvider attribute) _linked_sample() (deeplake.core.tensor.Tensor method) _pop_from_cache() (deeplake.core.storage.LRUCache method) _set_hub_creds_info() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.S3Provider method) _state_keys() (deeplake.core.storage.S3Provider method)


A

add_creds_key() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) AgreementError (class in deeplake.util.exceptions) AgreementNotAcceptedError (class in deeplake.util.exceptions) append() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) apply() (deeplake.core.index.Index method) apply_squeeze() (deeplake.core.index.Index method) array (deeplake.core.sample.Sample property) AuthenticationException (class in deeplake.util.exceptions) AuthorizationException (class in deeplake.util.exceptions) AutoCompressionError (class in deeplake.util.exceptions)


B

BadGatewayException (class in deeplake.util.exceptions) BadRequestException (class in deeplake.util.exceptions) base_htype (deeplake.core.tensor.Tensor property) batch() (deeplake.experimental.DeepLakeDataLoader method) branch (deeplake.core.dataset.Dataset property) branches (deeplake.core.dataset.Dataset property) BufferError (class in deeplake.util.exceptions)


C

change_creds_management() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) check_readonly() (deeplake.core.storage.StorageProvider method) checkout() (deeplake.core.dataset.Dataset method) CheckoutError (class in deeplake.util.exceptions) ChunkEngineError (class in deeplake.util.exceptions) ChunkIdEncoderError (class in deeplake.util.exceptions) ChunkSizeTooSmallError (class in deeplake.util.exceptions) clear() (deeplake.api.info.Info method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) (deeplake.core.tensor.Tensor method) clear_cache() (deeplake.core.dataset.Dataset method) (deeplake.core.storage.LRUCache method) clear_deeplake_objects() (deeplake.core.storage.LRUCache method) client (deeplake.core.dataset.Dataset property) (deeplake.core.dataset.DeepLakeCloudDataset property) commit() (deeplake.core.dataset.Dataset method) commit_id (deeplake.core.dataset.Dataset property) CommitError (class in deeplake.util.exceptions) commits (deeplake.core.dataset.Dataset property) compose() (in module deeplake) compose_at() (deeplake.core.index.Index method) compressed_bytes() (deeplake.core.sample.Sample method) compute() (in module deeplake) connect() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (in module deeplake) copy() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.storage.StorageProvider method) (in module deeplake) CorruptedMetaError (class in deeplake.util.exceptions) CorruptedSampleError (class in deeplake.util.exceptions) CouldNotCreateNewDatasetException (class in deeplake.util.exceptions) create_group() (deeplake.core.dataset.Dataset method) create_read_copy_dataset() (in module deeplake.util.remove_cache) create_tensor() (deeplake.core.dataset.Dataset method) create_tensor_like() (deeplake.core.dataset.Dataset method)


D

data() (deeplake.core.tensor.Tensor method) DataFrame (class in deeplake.auto.structured.dataframe) dataloader() (in module deeplake.experimental) dataset (class in deeplake.api.dataset) Dataset (class in deeplake.core.dataset) dataset() (in module deeplake) DatasetHandlerError (class in deeplake.util.exceptions) DatasetUnsupportedPytorch (class in deeplake.util.exceptions) DatasetViewSavingError (class in deeplake.util.exceptions) deepcopy() (deeplake.api.dataset.dataset static method) (in module deeplake) deeplake module deeplake.api.info module deeplake.integrations.wandb.wandb module DeepLakeCloudDataset (class in deeplake.core.dataset) DeepLakeDataLoader (class in deeplake.experimental) delete() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (deeplake.core.dataset.ViewEntry method) (in module deeplake) delete_group() (deeplake.core.dataset.Dataset method) delete_tensor() (deeplake.core.dataset.Dataset method) delete_view() (deeplake.core.dataset.Dataset method) dict() (deeplake.core.tensor.Tensor method) diff() (deeplake.core.dataset.Dataset method) DirectoryAtPathException (class in deeplake.util.exceptions) disable_readonly() (deeplake.core.storage.StorageProvider method) download_kaggle_dataset() (in module deeplake.auto.unstructured.kaggle) dtype (deeplake.core.tensor.Tensor property) DynamicTensorNumpyError (class in deeplake.util.exceptions)


G

GatewayTimeoutException (class in deeplake.util.exceptions) GCSDefaultCredsNotFoundError (class in deeplake.util.exceptions) GCSProvider (class in deeplake.core.storage) GDriveProvider (class in deeplake.core.storage) get() (deeplake.api.info.Info method) get_base_storage() (in module deeplake.util.remove_cache) get_bytes() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) get_commit_details() (deeplake.core.dataset.Dataset method) get_creds_keys() (deeplake.core.dataset.Dataset method) get_dataset_with_zero_size_cache() (in module deeplake.util.remove_cache) get_deeplake_object() (deeplake.core.storage.LRUCache method) get_view() (deeplake.core.dataset.Dataset method) get_views() (deeplake.core.dataset.Dataset method) groups (deeplake.core.dataset.Dataset property)


F

FileAtPathException (class in deeplake.util.exceptions) fill_dataset() (deeplake.auto.structured.dataframe.DataFrame method) filter() (deeplake.core.dataset.Dataset method) FilterError (class in deeplake.util.exceptions) find_axis() (deeplake.core.index.Index method) flush() (deeplake.core.dataset.Dataset method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.StorageProvider method) FullChunkError (class in deeplake.util.exceptions)


E

empty() (deeplake.api.dataset.dataset static method) (in module deeplake) EmptyCommitError (class in deeplake.util.exceptions) EmptyTensorError (class in deeplake.util.exceptions) EmptyTokenException (class in deeplake.util.exceptions) enable_readonly() (deeplake.core.storage.StorageProvider method) exists() (deeplake.api.dataset.dataset static method) (in module deeplake) extend() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) ExternalCommandError (class in deeplake.util.exceptions)


I

id (deeplake.core.dataset.ViewEntry property) ImageClassification (class in deeplake.auto.unstructured.image_classification) Index (class in deeplake.core.index) IndexEntry (class in deeplake.core.index) indices() (deeplake.core.index.IndexEntry method) Info (class in deeplake.api.info) info (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) InfoError (class in deeplake.util.exceptions) ingest() (deeplake.api.dataset.dataset static method) (in module deeplake) ingest_dataframe() (deeplake.api.dataset.dataset static method) (in module deeplake) ingest_huggingface() (in module deeplake) ingest_kaggle() (deeplake.api.dataset.dataset static method) (in module deeplake) InvalidBytesRequestedError (class in deeplake.util.exceptions) InvalidFileExtension (class in deeplake.util.exceptions) InvalidHubPathException (class in deeplake.util.exceptions) InvalidImageDimensions (class in deeplake.util.exceptions) InvalidInputDataError (class in deeplake.util.exceptions) InvalidKeyTypeError (class in deeplake.util.exceptions) InvalidOperationError (class in deeplake.util.exceptions) InvalidOutputDatasetError (class in deeplake.util.exceptions) InvalidPasswordException (class in deeplake.util.exceptions) InvalidPathException (class in deeplake.util.exceptions) InvalidShapeIntervalError (class in deeplake.util.exceptions) InvalidTensorGroupNameError (class in deeplake.util.exceptions) InvalidTensorLinkError (class in deeplake.util.exceptions) InvalidTensorNameError (class in deeplake.util.exceptions) InvalidTokenException (class in deeplake.util.exceptions) InvalidTransformDataset (class in deeplake.util.exceptions) is_actually_cloud (deeplake.core.dataset.DeepLakeCloudDataset property) is_colab() (in module deeplake.util.notebook) is_dynamic (deeplake.core.tensor.Tensor property) is_jupyter() (in module deeplake.util.notebook) is_link (deeplake.core.tensor.Tensor property) is_notebook() (in module deeplake.util.notebook) is_sequence (deeplake.core.tensor.Tensor property) is_trivial() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) is_view (deeplake.core.dataset.Dataset property) items() (deeplake.api.info.Info method)


L

length() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) like() (deeplake.api.dataset.dataset static method) (in module deeplake) link() (in module deeplake) (in module deeplake.api.link) LinkedSample (class in deeplake.core.linked_sample) list() (deeplake.api.dataset.dataset static method) (deeplake.core.tensor.Tensor method) (in module deeplake) load() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.ViewEntry method) (in module deeplake) load_view() (deeplake.core.dataset.Dataset method) LocalProvider (class in deeplake.core.storage) LockedException (class in deeplake.util.exceptions) log() (deeplake.core.dataset.Dataset method) LoginException (class in deeplake.util.exceptions) LRUCache (class in deeplake.core.storage)


M

ManagedCredentialsNotFoundError (class in deeplake.util.exceptions) max_len (deeplake.core.dataset.Dataset property) maybe_flush() (deeplake.core.storage.StorageProvider method) MemoryDatasetCanNotBePickledError (class in deeplake.util.exceptions) MemoryProvider (class in deeplake.core.storage) merge() (deeplake.core.dataset.Dataset method) merge_slices() (in module deeplake.core.index) MergeConflictError (class in deeplake.util.exceptions) MergeError (class in deeplake.util.exceptions) MergeMismatchError (class in deeplake.util.exceptions) MergeNotSupportedError (class in deeplake.util.exceptions) message (deeplake.core.dataset.ViewEntry property) meta (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) MetaAlreadyExistsError (class in deeplake.util.exceptions) MetaDoesNotExistError (class in deeplake.util.exceptions) MetaError (class in deeplake.util.exceptions) MetaInvalidKey (class in deeplake.util.exceptions) MetaInvalidRequiredMetaKey (class in deeplake.util.exceptions) min_len (deeplake.core.dataset.Dataset property) modified_samples() (deeplake.core.tensor.Tensor method) module deeplake deeplake.api.info deeplake.integrations.wandb.wandb ModuleNotInstalledException (class in deeplake.util.exceptions)


H

has_head_changes (deeplake.core.dataset.Dataset property) hidden (deeplake.core.tensor.Tensor property) htype (deeplake.core.tensor.Tensor property) HubComposeEmptyListError (class in deeplake.util.exceptions) HubComposeIncompatibleFunction (class in deeplake.util.exceptions)


N

nbytes (deeplake.api.info.Info property) ndim (deeplake.core.tensor.Tensor property) need_to_reload_creds() (deeplake.core.storage.S3Provider method) NotLoggedInError (class in deeplake.util.exceptions) num_samples (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) numpy() (deeplake.core.tensor.Tensor method) (deeplake.experimental.DeepLakeDataLoader method)


O

optimize() (deeplake.core.dataset.ViewEntry method) OutOfChunkCountError (class in deeplake.util.exceptions) OutOfSampleCountError (class in deeplake.util.exceptions) OverLimitException (class in deeplake.util.exceptions)


P

parent (deeplake.core.dataset.Dataset property) PartialSample (class in deeplake.core.partial_sample) path() (deeplake.core.tensor.Tensor method) PathNotEmptyException (class in deeplake.util.exceptions) pending_commit_id (deeplake.core.dataset.Dataset property) play() (deeplake.core.tensor.Tensor method) pop() (deeplake.api.info.Info method) (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) popitem() (deeplake.api.info.Info method) populate_creds() (deeplake.core.dataset.Dataset method) ProviderListEmptyError (class in deeplake.util.exceptions) ProviderSizeListMismatch (class in deeplake.util.exceptions) pytorch() (deeplake.core.dataset.Dataset method) (deeplake.experimental.DeepLakeDataLoader method)


Q

query() (deeplake.core.dataset.Dataset method) (deeplake.experimental.DeepLakeDataLoader method)


K

KaggleDatasetAlreadyDownloadedError (class in deeplake.util.exceptions) KaggleError (class in deeplake.util.exceptions) KaggleMissingCredentialsError (class in deeplake.util.exceptions) keys() (deeplake.api.info.Info method)


R

read() (in module deeplake) (in module deeplake.api.read) read_only (deeplake.core.dataset.Dataset property) ReadOnlyModeError (class in deeplake.util.exceptions) rechunk() (deeplake.core.dataset.Dataset method) register_deeplake_object() (deeplake.core.storage.LRUCache method) remove_deeplake_object() (deeplake.core.storage.LRUCache method) remove_memory_cache() (in module deeplake.util.remove_cache) rename() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.S3Provider method) (in module deeplake) rename_group() (deeplake.core.dataset.Dataset method) rename_tensor() (deeplake.core.dataset.Dataset method) RenameError (class in deeplake.util.exceptions) replace_with() (deeplake.api.info.Info method) reset() (deeplake.core.dataset.Dataset method) ResourceNotFoundException (class in deeplake.util.exceptions) root (deeplake.core.dataset.Dataset property)


S

S3DeletionError (class in deeplake.util.exceptions) S3Error (class in deeplake.util.exceptions) S3GetError (class in deeplake.util.exceptions) S3ListError (class in deeplake.util.exceptions) S3Provider (class in deeplake.core.storage) S3SetError (class in deeplake.util.exceptions) SamePathException (class in deeplake.util.exceptions) Sample (class in deeplake.core.sample) sample_by() (deeplake.core.dataset.Dataset method) (deeplake.experimental.DeepLakeDataLoader method) sample_indices (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) sample_info (deeplake.core.tensor.Tensor property) SampleCompressionError (class in deeplake.util.exceptions) SampleDecompressionError (class in deeplake.util.exceptions) SampleHtypeMismatchError (class in deeplake.util.exceptions) save_view() (deeplake.core.dataset.Dataset method) ServerException (class in deeplake.util.exceptions) set_bytes() (deeplake.core.storage.StorageProvider method) setdefault() (deeplake.api.info.Info method) shape (deeplake.core.tensor.Tensor property) shape_interval (deeplake.core.tensor.Tensor property) ShapeInterval (class in deeplake.util.shape_interval) shuffle() (deeplake.experimental.DeepLakeDataLoader method) size_approx() (deeplake.core.dataset.Dataset method) slice_at_int() (in module deeplake.core.index) slice_length() (in module deeplake.core.index) StorageProvider (class in deeplake.core.storage) structure() (deeplake.auto.unstructured.image_classification.ImageClassification method) StructuredDataset (class in deeplake.auto.structured.base) subscriptable() (deeplake.core.index.IndexEntry method) summary() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) sync() (deeplake.core.storage.GDriveProvider method)


U

UnableToReadFromUrlError (class in deeplake.util.exceptions) uncompressed_bytes() (deeplake.core.sample.Sample method) UnexpectedStatusCodeException (class in deeplake.util.exceptions) UnstructuredDataset (class in deeplake.auto.unstructured.base) UnsupportedCompressionError (class in deeplake.util.exceptions) UnsupportedSchedulerError (class in deeplake.util.exceptions) UnsupportedTensorTypeError (class in deeplake.util.exceptions) update() (deeplake.api.info.Info method) update_creds_key() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) UserNotLoggedInException (class in deeplake.util.exceptions)


T

Tensor (class in deeplake.core.tensor) TensorAlreadyExistsError (class in deeplake.util.exceptions) TensorDoesNotExistError (class in deeplake.util.exceptions) TensorDtypeMismatchError (class in deeplake.util.exceptions) tensorflow() (deeplake.core.dataset.Dataset method) TensorGroupAlreadyExistsError (class in deeplake.util.exceptions) TensorGroupDoesNotExistError (class in deeplake.util.exceptions) TensorInvalidSampleShapeError (class in deeplake.util.exceptions) TensorMetaInvalidHtype (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteKey (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteValue (class in deeplake.util.exceptions) TensorMetaMissingKey (class in deeplake.util.exceptions) TensorMetaMissingRequiredValue (class in deeplake.util.exceptions) TensorMetaMutuallyExclusiveKeysError (class in deeplake.util.exceptions) TensorMismatchError (class in deeplake.util.exceptions) TensorModifiedError (class in deeplake.util.exceptions) tensors (deeplake.core.dataset.Dataset property) TensorUnsupportedSampleType (class in deeplake.util.exceptions) text() (deeplake.core.tensor.Tensor method) tiled() (in module deeplake) (in module deeplake.api.tiled) timestamps (deeplake.core.tensor.Tensor property) tobytes() (deeplake.core.tensor.Tensor method) token (deeplake.core.dataset.Dataset property) (deeplake.core.dataset.DeepLakeCloudDataset property) TokenPermissionError (class in deeplake.util.exceptions) transform() (deeplake.experimental.DeepLakeDataLoader method) TransformError (class in deeplake.util.exceptions)


V

validate() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) values() (deeplake.api.info.Info method) verify (deeplake.core.tensor.Tensor property) VersionControlError (class in deeplake.util.exceptions) ViewEntry (class in deeplake.core.dataset) visualize() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method)


W

WaitTimeoutException (class in deeplake.util.exceptions)


Python Module Index

d d deeplake deeplake.api.info deeplake.integrations.wandb.wandb


How to Install Deep Lake and Get Started

Deep Lake can be installed through pip. By default, Deep Lake does not install dependencies for audio, video, google-cloud, and other features. Details on all installation options are available here. ! pip install deeplake


How to Create a Deep Lake Vector Store

Let's create a Vector Store in LangChain for storing and searching information about the Twitter OSS recommendation algorithm.


Downloading and Preprocessing the Data

First, let's import necessary packages and make sure the Activeloop and OpenAI keys are in the environmental variables ACTIVELOOP_TOKEN, OPENAI_API_KEY.from deeplake.core.vectorstore import VectorStoreimport openaiimport osNext, let's clone the Twitter OSS recommendation algorithm and define paths for for source data and the Vector Store.!git clone https://github.com/twitter/the-algorithmvector_store_path = '/vector_store_getting_started'repo_path = '/the-algorithm'Next, let's load all the files from the repo into list of data that will be added to the Vector Store (chunked_text and metadata). We use simple text chunking based on a constant number of characters. CHUNK_SIZE = 1000chunked_text = []metadata = []for dirpath, dirnames, filenames in os.walk(repo_path): for file in filenames: try: full_path = os.path.join(dirpath,file) with open(full_path, 'r') as f: text = f.read() new_chunkned_text = [text[i:i+1000] for i in range(0,len(text), CHUNK_SIZE)] chunked_text += new_chunkned_text metadata += [{'filepath': full_path} for i in range(len(new_chunkned_text))] except Exception as e: print(e) passNext, let's define an embedding function using OpenAI. It must work for a single string and a list of strings, so that it can both be used to embed a prompt and a batch of texts. def embedding_function(texts, model='text-embedding-ada-002'): if isinstance(texts, str): texts = [texts] texts = [t.replace('\n', ' ') for t in texts] return [data['embedding']for data in openai.Embedding.create(input = texts, model=model)['data']]Finally, let's create the Deep Lake Vector Store and populate it with data. We use a default tensor configuration, which creates tensors with text (str), metadata(json), id (str, auto-populated), embedding (float32). Learn more about tensor customizability here. vector_store = VectorStore( path = vector_store_path,)vector_store.add(text = chunked_text, embedding_function = embedding_function, embedding_data = chunked_text, metadata = metadata)The Vector Store's data structure can be summarized using vector_store.summary(), which shows 4 tensors with 21055 samples: tensor htype shape dtype compression ------- ------- ------- ------- ------- embedding embedding (21055, 1536) float32 None id text (21055, 1) str None metadata json (21055, 1) str None text text (21055, 1) str None To create a vector store using pre-compute embeddings, instead of embedding_data and embedding_function, you may run:# vector_store.add(text = chunked_text, # embedding = <list_of_embeddings>, # metadata = [{'source': source_text}]*len(chunked_text))


How to Search the Deep Lake Vector Store

Deep Lake offers highly-flexible vector search and hybrid search options discussed in detail in this tutorial.


Performing Vector Search

First, let's show a simple example of vector search using default options, which performs simple cosine similarity search in Python on the client (your machine). prompt = 'What do trust and safety models do?'search_results = vector_store.search(embedding_data=prompt, embedding_function=embedding_function)The search_results is a dictionary with keys for the text, score, id, and metadata, with data ordered by score. By default, the search returns the top 4 results which can be verified using: len(search_results['text']) # Returns 4If we examine the first returned text, it appears to contain the text about trust and safety models that is relevant to the prompt.search_results['text'][0]Returns:Trust and Safety Models=======================We decided to open source the training code of the following models:- pNSFWMedia: Model to detect tweets with NSFW images. This includes adult and porn content.- pNSFWText: Model to detect tweets with NSFW text, adult/sexual topics.- pToxicity: Model to detect toxic tweets. Toxicity includes marginal content like insults and certain types of harassment. Toxic content does not violate Twitter's terms of service.- pAbuse: Model to detect abusive content. This includes violations of Twitter's terms of service, including hate speech, targeted harassment and abusive behavior.We have several more models and rules that we are not going to open source at this time because of the adversarial nature of this area. The team is considering open sourcing more models going forward and will keep the community posted accordingly.We can also retrieve the corresponding filename from the metadata, which shows the top result came from the README.search_results['metadata'][0]# Returns: {'filepath': '/the-algorithm/trust_and_safety_models/README.md'}


Customization of Vector Search

You can customize your vector search with simple parameters, such as selecting the distance_metric and top k results:search_results = vector_store.search(embedding_data=prompt, embedding_function=embedding_functiondding, k=10, distance_metric='l2')The search now returns 10 search results:len(search_results['text']) # Returns: 10The first search result with the L2 distance metric returns the same text as the previous Cos search:search_results['text'][0]Returns:Trust and Safety Models=======================We decided to open source the training code of the following models:- pNSFWMedia: Model to detect tweets with NSFW images. This includes adult and porn content.- pNSFWText: Model to detect tweets with NSFW text, adult/sexual topics.- pToxicity: Model to detect toxic tweets. Toxicity includes marginal content like insults and certain types of harassment. Toxic content does not violate Twitter's terms of service.- pAbuse: Model to detect abusive content. This includes violations of Twitter's terms of service, including hate speech, targeted harassment and abusive behavior.We have several more models and rules that we are not going to open source at this time because of the adversarial nature of this area. The team is considering open sourcing more models going forward and will keep the community posted accordingly.


Full Customization of Vector Search

Deep Lake's Compute Engine can be used to rapidly execute a variety of different search logic. It is available with !pip install 'deeplake[enterprise]' (Make sure to restart your kernel after installation), and it is only available for data stored in or connected to Deep Lake. Let's load a representative Vector Store that is already stored in Deep Lake Tensor Database. If data is not being written, is advisable to use read_only = True.vector_store = VectorStore( path = 'hub://activeloop/twitter-algorithm', read_only=True)The query should be constructed using the Tensor Query Language (TQL) syntax.prompt = 'What do trust and safety models do?'embedding = embedding_function(prompt)[0]# Format the embedding array or list as a string, so it can be passed in the REST API request.embedding_string = ','.join([str(item) for item in embedding])tql_query = f'select * from (select text, cosine_similarity(embedding, ARRAY[{embedding_string}]) as score) order by score desc limit 5'Let's run the query, noting that the query execution happens in the Managed Tensor Database, and not on the client.search_results = vector_store.search(query=tql_query)If we examine the first returned text, it appears to contain the same text about trust and safety models that is relevant to the prompt.search_results['text'][0]Returns:Trust and Safety Models=======================We decided to open source the training code of the following models:- pNSFWMedia: Model to detect tweets with NSFW images. This includes adult and porn content.- pNSFWText: Model to detect tweets with NSFW text, adult/sexual topics.- pToxicity: Model to detect toxic tweets. Toxicity includes marginal content like insults and certain types of harassment. Toxic content does not violate Twitter's terms of service.- pAbuse: Model to detect abusive content. This includes violations of Twitter's terms of service, including hate speech, targeted harassment and abusive behavior.We have several more models and rules that we are not going to open source at this time because of the adversarial nature of this area. The team is considering open sourcing more models going forward and will keep the community posted accordingly.We can also retrieve the corresponding filename from the metadata, which shows the top result came from the README.print(search_results['metadata'][0])# Returns {'filepath': '/Users/istranic/ActiveloopCode/the-algorithm/trust_and_safety_models/README.md', 'extension': '.md'}


Search Options for Deep Lake Vector Stores in the Deep Lake API

This tutorial requires installation of:!pip3 install 'deeplake[enterprise]' langchain openai tiktoken


Vector Search on the Client

Let's load the same vector store used in the Quickstart and run embeddings search based on a user prompt using the Deep Lake Vector Store module. (Note: DeepLakeVectorStore class is deprecated, but you can still use it. The new API for calling Deep Lake's Vector Store is: VectorStore)from deeplake.core.vectorstore import VectorStoreimport openaiimport osos.environ['OPENAI_API_KEY'] = <OPENAI_API_KEY>vector_store_path = 'hub://activeloop/paul_graham_essay'vector_store = VectorStore( path = vector_store_path, read_only = True)Next, let's define an embedding function using OpenAI. It must work for a single string and a list of strings so that it can be used to embed a prompt and a batch of texts. def embedding_function(texts, model = 'text-embedding-ada-002'): if isinstance(texts, str): texts = [texts] texts = [t.replace('\n', ' ') for t in texts] return [data['embedding']for data in openai.Embedding.create(input = texts, model=model)['data']]


Simple Vector Search

Let's run a simple vector search using default options, which performs a simple cosine similarity search in Python on the client. prompt = 'What are the first programs he tried writing?'search_results = vector_store.search(embedding_data=prompt, embedding_function=embedding_function)The search_results is a dictionary with keys for the text, score, id, and metadata, with data ordered by score. By default, it returns 4 samples ordered by similarity score, and if we examine the first returned text, it appears to contain the text about trust and safety models that is relevant to the prompt.search_results['text'][0]Returns:What I Worked OnFebruary 2021Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.The first programs I tried writing were on the IBM 1401 that our school district used for what was then called 'data processing.' This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.


Hybrid Search Using UDFs

Vector search can be combined with other search logic for performing more advanced queries. Let's define a function compatible with deeplake.filter for filtering data before the vector search. The function below will filter samples that contain the word 'program' in the text tensor.def filter_fn(x): # x is a single row in Deep Lake, 'text' is the tensor name, .data()['value'] is the method for fetching the data return 'program' in x['text'].data()['value'].lower()Let's run the vector search with the filter above, and return more samples (k = 10), and perform similarity search using L2 metric (distance_metric = 'l2'):prompt = 'What are the first programs he tried writing?'search_results_filter = vector_store.search(embedding_data = prompt, embedding_function = embedding_function, filter = filter_fn, k = 10, distance_metric = 'l2', exec_option = 'python')We can verity that the word 'program' is present in all of the results:all(['program' in result for result in search_results_filter['text']])# Returns TrueUDFs are only supported with query execution using the Python engine, so in the search above, exec_option = 'python' should be specified.


Hybrid Search Using Metadata Filters

Instead of using UDFs, a filter can be specified using dictionary syntax. For json tensors, the syntax is filter = {'tensor_name': {'key': 'value'}}. For text tensors, it is filter = {'tensor': 'value'}. In all cases, an exact match is performed.search_results_filter = vector_store.search(embedding_data = prompt, embedding_function = embedding_function, filter = {'metadata': {'source': 'paul_graham_essay.txt'}})


Hybrid Search using TQL Vector Search

Deep Lake offers advanced search that executes queries with higher performance in C++, and offers querying using Deep Lake's Tensor Query Language (TQL). In order to use Compute Engine, Deep Lake data must be stored in Deep Lake Storage, or in the user's cloud while being connected to Deep Lake using Managed Credentials. Let's load a larger Vector Store for running more interesting queries:vector_store_path = 'hub://activeloop/twitter-algorithm'vector_store = VectorStore( path = vector_store_path, read_only = True)NOTE: this Vector Store is stored in us-east, and query performance may vary significantly depending on your location. In real-world use-cases, users would store their Vector Stores in regions optimized for their use case.Now let's run a search that includes filtering of text, metadata, and embedding tensors. We do this using TQL by combining embedding search syntax (cosine_similarity(embedding, ...)) and filtering syntax (where ....). We are interested in answering a prompt based on the question:prompt = 'What does the python code do?'Therefore, we apply a filter to only search for text that contains the word 'python' and metadata where the source key contains '.py'.embedding = embedding_function(prompt)[0]# Format the embedding array or list as a string, so it can be passed in the REST API request.embedding_string = ','.join([str(item) for item in embedding])tql_query = f'select * from (select text, metadata, cosine_similarity(embedding, ARRAY[{embedding_string}]) as score where contains(text, 'python') or contains(metadata['source'], '.py')) order by score desc limit 5'search_results = vector_store.search(query = tql_query)


Vector Search Using the Managed Tensor Database (Server-Side)

For Vector Stored in the Managed Tensor Database, queries will automatically execute on the database (instead of the client). Vector Stores are created in the Managed Tensor Database by specifying vector_store_path = hub://org_id/dataset_name and runtime = {'tensor_db': True} during Vector Store creation.# vector_store = VectorStore(# path = 'hub://<org_id>/<dataset_name>',# runtime = {'tensor_db': True}# )search_results = vector_store.search(embedding_data=prompt, embedding_function=embedding_function)If Vector Stores are not in the Managed Tensor Database, they can be migrated using these steps:


How to Run Vector Search in the Deep Lake Tensor Database using the REST API

The REST API is currently in Alpha, and the syntax may change without announcement.To use the REST API, Deep Lake data must be stored in the Managed Tensor Database by specifying the deeplake_path = hub://org_id/dataset_name and runtime = {'tensor_db': True}. Full details on path and storage management are available here.


Performing Vector Search Using the REST API

Let's query this Vector Store stored in the Managed Tensor Database using the REST API. The steps are:1.Define the authentication tokens and search terms2.Embed the search search term using OpenAI3.Reformat the embedding to an embedding_search string that can be passed to the REST API request.4.Create the query string using Deep Lake TQL. The dataset_path and embedding_search are a part of the query string. 5.Submit the request and print the response data dataimport requestsimport openaiimport os# Tokens should be set in environmental variables.ACTIVELOOP_TOKEN = os.environ['ACTIVELOOP_TOKEN']DATASET_PATH = 'hub://activeloop/twitter-algorithm'ENDPOINT_URL = 'https://app.activeloop.ai/api/query/v1'SEARCH_TERM = 'What do the trust and safety models do?'# os.environ['OPENAI_API_KEY'] OPEN AI TOKEN should also exist in env variables# The headers contains the user tokenheaders = { 'Authorization': f'Bearer {ACTIVELOOP_TOKEN}',}# Embed the search termembedding = openai.Embedding.create(input=SEARCH_TERM, model='text-embedding-ada-002')['data'][0]['embedding']# Format the embedding array or list as a string, so it can be passed in the REST API request.embedding_string = ','.join([str(item) for item in embedding])# Create the query using TQLquery = f'select * from (select text, cosine_similarity(embedding, ARRAY[{embedding_string}]) as score from \'{dataset_path}\') order by score desc limit 5' # Submit the request response = requests.post(ENDPOINT_URL, json={'query': query}, headers=headers)data = response.json()print(data)const axios = require('axios');OPENAI_API_KEY = process.env.OPENAI_API_KEY;ACTIVELOOP_TOKEN = process.env.ACTIVELOOP_TOKEN;const QUERY = 'What do the trust and safety models do?';const DATASET_PATH = 'hub://activeloop/twitter-algorithm';const ENDPOINT_URL = 'https://app.activeloop.ai/api/query/v1';// Function to get the embeddings of a text from Open AI APIasync function getEmbedding(text) { const response = await axios.post('https://api.openai.com/v1/embeddings', { input: text, model: 'text-embedding-ada-002' }, { headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${OPENAI_API_KEY}` } }); return response.data;}// Function to search the dataset using the given query on Activeloopasync function searchDataset(query) { const response = await axios.post(${ENDPOINT_URL}, { query: query, }, { headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${ACTIVELOOP_TOKEN}` } }); return response.data;}// Main function to search for similar texts in the dataset based on the query_termasync function searchSimilarTexts(query, dataset_path) { // Get the embedding of the query_term const embedding = await getEmbedding(query); const embedding_search = embedding.data[0].embedding.join(','); // Construct the search query const TQL = `SELECT * FROM ( SELECT text, l2_norm(embedding - ARRAY[${embedding_search}]) AS score from '${dataset_path}' ) ORDER BY score DESC LIMIT 5`; // Search the dataset using the constructed query const response = await searchDataset(TQL); // Log the search results console.log(response);}searchSimilarTexts(QUERY, DATASET_PATH)Congrats! You performed a vector search using the Deep Lake Managed Database! 🎉


How to Execute Vector Search Using Deep Lake in LangChain

This tutorial requires installation of:!pip3 install langchain deeplake openai tiktokenLet's load the same vector store used in the Quickstart and run embeddings search based on a user prompt using the LangChain API. from langchain.vectorstores import DeepLakefrom langchain.chains import RetrievalQAfrom langchain.llms import OpenAIChatfrom langchain.embeddings.openai import OpenAIEmbeddingsimport osos.environ['OPENAI_API_KEY'] = <OPENAI_API_KEY>vector_store_path = 'hub://activeloop/paul_graham_essay'embedding_function = OpenAIEmbeddings(model = 'text-embedding-ada-002')# Re-load the vector storedb = DeepLake(dataset_path = vector_store_path, embedding = embedding_function, read_only = True)qa = RetrievalQA.from_chain_type(llm=OpenAIChat(model = 'gpt-3.5-turbo'), chain_type = 'stuff', retriever = db.as_retriever())


Vector Similarity Search

Let's run a similarity search on Paul Graham's essay based on a query we want to answer. The query is embedded and a similarity search is performed against the stored embeddings, with execution taking place on the client.prompt = 'What are the first programs he tried writing?'query_docs = db.similarity_search(query = prompt)If we print the first document using query_docs[0].page_content, it appears to be relevant to the query:What I Worked OnFebruary 2021Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.The first programs I tried writing were on the IBM 1401 that our school district used for what was then called 'data processing.' This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.


Vector Search Using the Managed Tensor Database

For Vector Stores in the Managed Tensor Database, queries will automatically execute on the database (instead of the client). Vector Stores are created in the Managed Tensor Database by specifying vector_store_path = hub://org_id/dataset_name and runtime = {'tensor_db': True} during Vector Store creation.# db = DeepLake(dataset_path = 'hub://<org_id>/<dataset_name>', # runtime = {'tensor_db': True},# embedding = embedding_function# )If Vector Stores are not in the Managed Tensor Database, they can be migrated using these steps:


Vector Search in an LLM Context

We can directly use LangChain to run a Q&A using an LLM and answer the question about Paul Graham's essay. Internally, this API performs an embedding search to find the most relevant data to feeds them into the LLM context.qa = RetrievalQA.from_chain_type(llm = OpenAIChat(model = 'gpt-3.5-turbo'), chain_type = 'stuff', retriever = db.as_retriever())qa.run(prompt)


This tutorial is also available as a Colab Notebook

Step 8 in the Getting Started Guide highlights how deeplake.compute can be used to rapidly upload datasets. This tutorial expands further and highlights the power of parallel computing for dataset processing.


Transformations on New Datasets

Computer vision applications often require users to process and transform their data. For example, you may perform perspective transforms, resize images, adjust their coloring, or many others. In this example, a flipped version of the MNIST dataset is created, which may be useful for training a model that identifies text in scenes where the camera orientation is unknown. First, let's define a function that will flip the dataset images.import deeplakefrom PIL import Imageimport numpy as np@deeplake.computedef flip_vertical(sample_in, sample_out): ## First two arguments are always default arguments containing: # 1st argument is an element of the input iterable (list, dataset, array,...) # 2nd argument is a dataset sample # Append the label and image to the output sample sample_out.append({'labels': sample_in.labels.numpy(), 'images': np.flip(sample_in.images.numpy(), axis = 0)}) return sample_outNext, the existing MNIST dataset is loaded, and deeplake.like is used to create an empty dataset with the same tensor structure.ds_mnist = deeplake.load('deeplake://activeloop/mnist-train')#We use the overwrite=True to make this code re-runnableds_mnist_flipped = deeplake.like('./mnist_flipped', ds_mnist, overwrite = True)Finally, the flipping operation is evaluated for the 1st 100 elements in the input dataset ds_in, and the result is automatically stored in ds_out.flip_vertical().eval(ds_mnist[0:100], ds_mnist_flipped, num_workers = 2)Let's check out the flipped images:Image.fromarray(ds_mnist.images[0].numpy())Image.fromarray(ds_mnist_flipped.images[0].numpy())


Dataset Processing Pipelines

In order to modularize your dataset processing, it is helpful to create functions for specific data processing tasks and combine them in pipelines. In this example, you can create a pipeline using the flip_vertical function from the first example and the resize function below.@deeplake.computedef resize(sample_in, sample_out, new_size): ## First two arguments are always default arguments containing: # 1st argument is an element of the input iterable (list, dataset, array,...) # 2nd argument is a dataset sample ## Third argument is the required size for the output images # Append the label and image to the output sample sample_out.labels.append(sample_in.labels.numpy()) sample_out.images.append(np.array(Image.fromarray(sample_in.images.numpy()).resize(new_size))) return sample_outFunctions decorated using deeplake.compute can be combined into pipelines using deeplake.compose. Required arguments for the functions must be passed into the pipeline in this step:pipeline = deeplake.compose([flip_vertical(), resize(new_size = (64,64))])Just like for the single-function example above, the input and output datasets are created first, and the pipeline is evaluated for the 1st 100 elements in the input dataset ds_in. The result is automatically stored in ds_out.#We use the overwrite=True to make this code re-runnableds_mnist_pipe = deeplake.like('./mnist_pipeline', ds_mnist, overwrite = True)pipeline.eval(ds_mnist[0:100], ds_mnist_pipe, num_workers = 2)


Transformations on Existing Datasets

In the previous example, a new dataset was created while performing a transformation. In this example, a transformation is used to modify an existing dataset. First, download and unzip the small classification dataset below called animals. Next, use deeplake.ingest_classification to automatically convert this image classification dataset into Deep Lake format and save it in ./animals_deeplake.ds = deeplake.ingest_classification('./animals', './animals_deeplake') # Creates the datasetThe first image in the dataset is a picture of a cat:Image.fromarray(ds.images[0].numpy())The images in the dataset can now be flipped by evaluating the flip_vertical() transformation function from the previous example. If a second dataset is not specified as an input to .eval(), the transformation is applied to the input dataset. flip_vertical().eval(ds, num_workers = 2)The picture of the cat is now flipped:Image.fromarray(ds.images[0].numpy())


Recovering From Errors

If an error occurs related to a specific sample_in, deplake.compute will throw a TransformError and the error-causing index or sample can be caught using:# from deeplake.util.exceptions import TransformError# try:# compute_fn.eval(...)# except TransformError as e:# failed_idx = e.index# failed_sample = e.sampleThe traceback also typically shows information such as the filename of the data that was causing issues. One the problematic sample has been identified, it should be removed from the list of input samples and the deeplake.compute function should be re-executed. Congrats! You just learned how to make parallelize your computations using Deep Lake! 🎉


Installing Deep Lake

Deep Lake can be installed through pip. By default, Deep Lake does not install dependencies for audio, video, google-cloud, and other features. Details on all installation options are available here. ! pip install deeplake


Fetching Your First Deep Lake Dataset

Let's load MNIST, the hello world dataset of machine learning. First, instantiate a Dataset by pointing to its storage location. Datasets hosted on Activeloop Platform are typically identified by the namespace of the organization followed by the dataset name: activeloop/mnist-train.import deeplakedataset_path = 'hub://activeloop/mnist-train'ds = deeplake.load(dataset_path) # Returns a Deep Lake Dataset but does not download data locally


Reading Samples From a Deep Lake Dataset

Data is not immediately read into memory because Deep Lake operates lazily. You can fetch data by calling the .numpy() method, which reads data into a NumPy array.# Indexingimg = ds.images[0].numpy() # Fetch the 1st image and return a NumPy arraylabel = ds.labels[0].numpy(aslist=True) # Fetch the 1st label and store it as a # as a list text_labels = ds.labels[0].data()['text'] # Fetch the first labels and return them as text# Slicingimgs = ds.images[0:100].numpy() # Fetch 100 images and return a NumPy array # The method above produces an exception if # the images are not all the same sizelabels = ds.labels[0:100].numpy(aslist=True) # Fetch 100 labels and store # them as a list of NumPy arraysCongratulations, you've got Deep Lake working on your local machine🤓


Data in Deep Lake can be stored in raw uncompressed format. However, compression is highly recommended for achieving optimal performance in terms of speed and storage.

Compression is specified separately for each tensor, and it can occur at the sample or chunk level. For example, when creating a tensor for storing images, you can choose the compression technique for the image samples using the sample_compression input:ds.create_tensor('images', htype = 'image', sample_compression = 'jpeg')In this example, every image added in subsequent .append(...) calls is compressed using the specified sample_compression method. The full list of available compressions is shown in the API Reference.


Choosing the Right Compression

There is no single answer for choosing the right compression, and the tradeoffs are described in detail in the next section. However, good rules of thumb are:1.For data that has application-specific compressors (image, audio, video,...), choose the sample_compression technique that is native to the application such as jpg, mp3, mp4,...2.For other data containing large samples (i.e. large arrays with >100 values), lz4 is a generic compressor that works well in most applications.1.lz4 can be used as a sample_compression or chunk_compression . In most cases, sample_compression is sufficient, but in theory, chunk_compression produces slightly smaller data.3.For other data containing small samples (i.e. labels with <100 values), it is not necessary to use compression.


Compression Tradeoffs

Lossiness - Certain compression techniques are lossy, meaning that there is irreversible information loss when compressing the data. Lossless compression is less important for data such as images and videos, but it is critical for label data such as numerical labels, binary masks, and segmentation data.Memory - Different compression techniques have substantially different memory footprints. For instance, png vs jpeg compression may result in a 10X difference in the size of a Hub dataset. Runtime - The primary variables affecting download and upload speeds for generating usable data are the network speed and available compute power for processing the data. In most cases, the network speed is the limiting factor. Therefore, the highest end-to-end throughput for non-local applications is achieved by maximizing compression and utilizing compute power to decompress/convert the data to formats that are consumed by deep learning models (i.e. arrays). Upload Considerations - When applicable, the highest uploads speeds can be achieved when the sample_compression input matches the compression of the source data, such as:# sample_compression and my_image are 'jpeg'ds.create_tensor('images', htype = 'image', sample_compression = 'jpeg')ds.images.append(deeplake.read('my_image.jpeg'))In this case, the input data is a .jpg, and the Deep Lake sample_compression is jpg. However, a mismatch between the compression of the source data and sample_compression in Deep Lake results in significantly slower upload speeds, because Deep Lake must decompress the source data and recompress it using the specified sample_compression before saving.Therefore, due to the computational costs associated with decompressing and recompressing data, it is important that you consider the runtime implications of uploading source data that is compressed differently than the specified sample_compression.


Loading Datasets

Deep Lake Datasets can be loaded from a variety of storage locations using:import deeplake# Local Filepathds = deeplake.load('./my_dataset_path') # Similar functionality to deeplake.dataset(path)# S3ds = deeplake.load('s3://my_dataset_bucket', creds={...})# Public Dataset hosted by Activeloop## Activeloop Storage - See Step 6ds = deeplake.load('hub://activeloop/public_dataset_name')# Dataset in another organization on Activeloop Platformds = deeplake.load('hub://org_name/dataset_name')Since ds = deeplake.dataset(path)can be used to both create and load datasets, you may accidentally create a new dataset if there is a typo in the path you provided while intending to load a dataset. If that occurs, simply use ds.delete() to remove the unintended dataset permanently.


Updating Data

Existing data in a Deep Lake dataset can be updated using:ds.images[1] = deeplake.read('https://i.postimg.cc/Yq2SNz9J/photo-1534567110243-8875d64ca8ff.jpg') # If the URI is not public, credentials should be specified using deeplake.read(URI, creds = {...})ds.labels[1] = 'giraffe' # Tensors of htype = 'class_label' can be updated with either numeric values or textImage.fromarray(ds.images[1].numpy())


Accessing Data

Data within the tensors is loaded and accessed using the .numpy() , .data() , and .tobytes() commands. When the underlying data can be converted to a numpy array, .data() and .numpy() return equivalent objects.# Indexingimg = ds.images[0].numpy() # Fetch the 1st image and return a NumPy arraylabel = ds.labels[0].numpy(aslist=True) # Fetch the 1st label and store it as a # as a list # frame = ds.videos[0][4].numpy() # Fetch the 5th frame in the 1st video # and return a NumPy array text_labels = ds.labels[0].data()['value'] # Fetch the first labels and return them as text# Slicingimgs = ds.images[0:100].numpy() # Fetch 100 images and return a NumPy array # The method above produces an exception if # the images are not all the same sizelabels = ds.labels[0:100].numpy(aslist=True) # Fetch 100 labels and store # them as a list of NumPy arraysThe .numpy()method produces an exception if all samples in the requested tensor do not have a uniform shape. If that's the case, running .numpy(aslist=True)returns a list of NumPy arrays, where the indices of the list correspond to different samples.


Referencing Tensors

Deep Lake allows you to reference specific tensors using keys or via the '.' notation outlined below. Note: data is still not loaded by these commands.### NO HIERARCHY ###ds.images # is equivalent tods['images']ds.labels # is equivalent tods['labels']### WITH HIERARCHY ###ds.localization.boxes # is equivalent tods['localization/boxes']ds.localization.labels # is equivalent tods['localization/labels']


Register

You can store your Deep Lake Datasets with Activeloop by first creating an account in the Deep Lake App or in the CLI using:activeloop register


Login

In order for the Python API to authenticate with your account, you can use API tokens (see below), or log in from the CLI using:!activeloop login# Alternatively, you can directly input your username and password in the same line:# activeloop login -u <your_username> -p <your_password>You can then access or create Deep Lake Datasets by passing the Deep Lake path to deeplake.dataset()import deeplakedeeplake_path = 'hub://organization_name/dataset_name' #'hub://jane_smith/my_awesome_dataset' ds = deeplake.dataset(deeplake_path)When you create an account in Deep Lake, a default organization is created that has the same name as your username. You can also create other organizations that represent companies, teams, or other collections of multiple users. Public datasets such as 'hub://activeloop/mnist-train' can be accessed without logging in.


API Tokens

Once you have an Activeloop account, you can create tokens in the Deep Lake App (Organization Details -> API Tokens) and authenticate by setting the environmental variable: os.environ['ACTIVELOOP_TOKEN'] = <your_token>Or login in the CLI using the token:!activeloop login --token <your_token>If you are not logged in through the CLI, you may also pass the token to python commands that require authentication:ds = deeplake.load(deeplake_path, token = 'xyz')


How to Accelerate Deep Lake Workflows with Parallel Computing

Deep Lake enables you to easily run computations in parallel and significantly accelerate your data processing workflows. This example primarily focuses on parallel dataset uploading.Parallel computing use cases such as dataset transformations can be found in this tutorial.Parallel compute using Deep Lake has two core steps: 1.Define a function or pipeline that will run in parallel and2.Evaluate the function using the appropriate inputs and outputs.


Defining the parallel computing function

The first step is to define a function that will run in parallel by decorating it using @deeplake.compute. In the example below, file_to_deeplake converts data from files into Deep Lake format, just like in Step 2: Creating Hub Datasets Manually. If you have not completed Step 2, please download and unzip the example image classification dataset below:animals datasetimport deeplakefrom PIL import Imageimport numpy as npimport os@deeplake.computedef file_to_deeplake(file_name, sample_out, class_names): ## First two arguments are always default arguments containing: # 1st argument is an element of the input iterable (list, dataset, array,...) # 2nd argument is a dataset sample # Other arguments are optional # Find the label number corresponding to the file label_text = os.path.basename(os.path.dirname(file_name)) label_num = class_names.index(label_text) # Append the label and image to the output sample sample_out.append({'labels': np.uint32(label_num), 'images': deeplake.read(file_name)}) return sample_outIn all functions decorated using @deeplake.compute, the first argument must be a single element of any input iterable that is being processed in parallel. In this case, that is a filename file_name, because file_to_deeplake reads image files and populates data in the dataset's tensors. The second argument is a dataset sample sample_out, which can be operated on using similar syntax to dataset objects, such as sample_out.append(...), sample_out.extend(...), etc.The function decorated using @deeplake.compute must return sample_out, which represents the data that is added or modified by that function.


Executing the parallel computation

To execute the parallel computation, you must define the dataset that will be modified.ds = deeplake.empty('./animals_deeplake_transform') # Creates the datasetNext, you define the input iterable that describes the information that will be operated on in parallel. In this case, that is a list of files files_list:# Find the class_names and list of files that need to be uploadeddataset_folder = './animals'class_names = os.listdir(dataset_folder)files_list = []for dirpath, dirnames, filenames in os.walk(dataset_folder): for filename in filenames: files_list.append(os.path.join(dirpath, filename))You can now create the tensors for the dataset and run the parallel computation using the .eval syntax. Pass the optional input arguments to file_to_deeplake and skip the first two default arguments file_name and sample_out. The input iterable files_list and output dataset ds is passed to the .eval method as the first and second argument respectively.with ds: ds.create_tensor('images', htype = 'image', sample_compression = 'jpeg') ds.create_tensor('labels', htype = 'class_label', class_names = class_names) file_to_deeplake(class_names=class_names).eval(files_list, ds, num_workers = 2)Additional parallel computing use cases such as dataset transformations can be found in this tutorial.Image.fromarray(ds.images[0].numpy())Congrats! You just created a dataset using parallel computing! 🎈


Deep Lake enables users to manage their data more easily so they can train better ML models. This tutorial shows you how to train a simple image classification model while streaming data from a Deep Lake dataset stored in the cloud.


Data Preprocessing

The first step is to select a dataset for training. This tutorial uses the Fashion MNIST dataset that has already been converted into Deep Lake format. It is a simple image classification dataset that categorizes images by clothing type (trouser, shirt, etc.)import deeplakefrom PIL import Imageimport numpy as npimport os, timeimport torchfrom torchvision import transforms, models# Connect to the training and testing datasetsds_train = deeplake.load('hub://activeloop/fashion-mnist-train')ds_test = deeplake.load('hub://activeloop/fashion-mnist-test')The next step is to define a transformation function that will process the data and convert it into a format that can be passed into a deep learning model. In this particular example, torchvision.transforms is used as a part of the transformation pipeline that performs operations such as normalization and image augmentation (rotation).tform = transforms.Compose([ transforms.RandomRotation(20), # Image augmentation transforms.ToTensor(), # Must convert to pytorch tensor for subsequent operations to run transforms.Normalize([0.5], [0.5]),])You can now create a pytorch dataloader that connects the Deep Lake dataset to the PyTorch model using the provided method ds.pytorch(). This method automatically applies the transformation function, takes care of random shuffling (if desired), and converts Deep Lake data to PyTorch tensors. The num_workers parameter can be used to parallelize data preprocessing, which is critical for ensuring that preprocessing does not bottleneck the overall training workflow.The transform input is a dictionary where the key is the tensor name and the value is the transformation function that should be applied to that tensor. If a specific tensor's data does not need to be returned, it should be omitted from the keys. If the transformation function is set as None, the input tensor is converted to a torch tensor without additional modification.# Since torchvision transforms expect PIL images, we use the 'pil' decode_method for the 'images' tensor. This is much faster than running ToPILImage inside the transformtrain_loader = ds_train.pytorch(num_workers = 0, shuffle = True, transform = {'images': tform, 'labels': None}, batch_size = batch_size, decode_method = {'images': 'pil'})test_loader = ds_test.pytorch(num_workers = 0, transform = {'images': tform, 'labels': None}, batch_size = batch_size, decode_method = {'images': 'pil'})


Model Definition

This tutorial uses a pre-trained ResNet18 neural network from the torchvision.models module, converted to a single-channel network for grayscale images.Training is run on a GPU if possible. Otherwise, run on a CPU.device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')print(device)# Use a pre-trained ResNet18model = models.resnet18(pretrained=True)# Convert model to grayscalemodel.conv1 = torch.nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)# Update the fully connected layer based on the number of classes in the datasetmodel.fc = torch.nn.Linear(model.fc.in_features, len(ds_train.labels.info.class_names))model.to(device)# Specity the loss function and optimizercriterion = torch.nn.CrossEntropyLoss()optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.1)


Training the Model

Helper functions for training and testing the model are defined. Note that the output from Deep Lake's PyTorch dataloader is fed into the model just like data from ordinary PyTorch dataloaders.def train_one_epoch(model, optimizer, data_loader, device): model.train() # Zero the performance stats for each epoch running_loss = 0.0 start_time = time.time() total = 0 correct = 0 for i, data in enumerate(data_loader): # get the inputs; data is a list of [inputs, labels] inputs = data['images'] labels = torch.squeeze(data['labels']) inputs = inputs.to(device) labels = labels.to(device) # zero the parameter gradients optimizer.zero_grad() # forward + backward + optimize outputs = model(inputs.float()) loss = criterion(outputs, labels) loss.backward() optimizer.step() _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() accuracy = 100 * correct / total # Print performance statistics running_loss += loss.item() if i % 10 == 0: # print every 10 batches batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] loss: %.3f, speed: %.2f, accuracy: %.2f %%' % (i, running_loss, speed, accuracy)) running_loss = 0.0 total = 0 correct = 0 def test_model(model, data_loader): model.eval() start_time = time.time() total = 0 correct = 0 with torch.no_grad(): for i, data in enumerate(data_loader): # get the inputs; data is a list of [inputs, labels] inputs = data['images'] labels = torch.squeeze(data['labels']) inputs = inputs.to(device) labels = labels.to(device) # zero the parameter gradients optimizer.zero_grad() # forward + backward + optimize outputs = model(inputs.float()) _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() accuracy = 100 * correct / total print('Finished Testing') print('Testing accuracy: %.1f %%' %(accuracy))The model and data are ready for training🚀!num_epochs = 3for epoch in range(num_epochs): # loop over the dataset multiple times print('------------------ Training Epoch {} ------------------'.format(epoch+1)) train_one_epoch(model, optimizer, train_loader, device) test_model(model, test_loader)print('Finished Training')Congrats! You successfully trained an image classification model while streaming data directly from the cloud! 🎉


How to Train Deep Learning models using Deep Lake and MMDetection

This tutorial assumes the reader has experience training models using MMDET and has installed it successfully. At the bottom of the page, we provide a high-level overview of MMDetection fundamentals.Deep Lake offers an integration with MMDetection, a popular open-source object detection toolbox based on PyTorch. The integration enables users to train models while streaming Deep Lake dataset using the transformation, training, and evaluation tools built by MMDet.


Integration Interface

Training using MMDET is typically executed using wrapper scripts like the one provided here in their repo. In the example below, we write a similar simplified wrapper script for training using a Deep Lake dataset.The integrations with MMDET occurs in the deeplake.integrations.mmdet module. At a high-level, Deep Lake is responsible for the pytorch dataloader that streams data to the training framework, while MMDET is used for the training, transformation, and evaluation logic.In the example script below, the user should apply the build_detector and train_detector provided by Deep Lake. The build_detector is mostly boilerplate. and the Deep Lake-related features primarily exist in train_detector.import osfrom mmcv import Configimport mmcvfrom deeplake.integrations import mmdet as mmdet_deeplakeimport argparsedef parse_args(): parser = argparse.ArgumentParser(description='Deep Lake Training Using MMDET') parser.add_argument( '--cfg_file', type=str, required=True, help='Path for loading the config file', ) parser.add_argument( '--validate', action='store_true', default=True, help='Whether to run dataset validation', ) parser.add_argument( '--distributed', action='store_true', default=False, help='Whether to run distributed training', ) parser.add_argument( '--num_classes', type=int, default=None, help='Number of classes in the model', ) args = parser.parse_args() return argsif __name__ == '__main__': args = parse_args() # Read the config file cfg = Config.fromfile(args.cfg_file) cfg.model.bbox_head.num_classes = args.num_classes # Build the detector model = mmdet_deeplake.build_detector(cfg.model) # Create work_dir mmcv.mkdir_or_exist(os.path.abspath(cfg.work_dir)) # Run the training mmdet_deeplake.train_detector(model, cfg, distributed=args.distributed, validate=args.validate)


Inputs to train_detector

Inputs to the Deep Lake train_detector are a modified MMDET config file, optional dataset objects (see below), and flags for specifying whether to perform distributed training and validation.


Modifications to the cfg file


Passing Deep Lake dataset objects to the train_detector (Optional)

The Deep Lake dataset object or dataset view can be passed to the train_detector directly, thus overwriting any dataset information in the config file. Below are the respective modifications that should be made to the training script above:ds_train = deeplake.load(dataset_path, token, ...)ds_train.checkout(commit_id)ds_train_view = ds_train.query('Add query string')mmdet_deeplake.train_detector(model, cfg, ds_train = ds_train_view, ds_val = ..., distributed = args.distributed, validate = args.validate)Congrats! You're now able to train models using MMDET while streaming Deep Lake Datasets! 🎉


What is MMDetection?

MMDetection is a powerful open-source object detection toolbox that provides a flexible and extensible platform for computer vision tasks. Developed by the Multimedia Laboratory (MMLab) as part of the OpenMMLab project, MMDetection is built upon the PyTorch framework and offers a composable and modular API design. This unique feature enables developers to easily construct custom object detection and segmentation pipelines. This article will delve deeper into how to use MMDetection with Activeloop Deep Lake.


MMDetection Features

MMDetection's Modular and Composable API Design MMDetection's API design follows a modular approach, enabling seamless integration with frameworks like Deep Lake and easy component customization. This flexibility allows users to adapt the object detection pipeline to meet specific project requirements.


Custom Object Detection and Segmentation Pipelines

MMDetection streamlines custom pipeline creation, allowing users to construct tailored models by selecting and combining different backbones, necks, and heads for more accurate and efficient computer vision pipelines.


Comprehensive Training & Inference Support

MMDetection's toolbox supports various data augmentation techniques, distributed training, mixed-precision training, and detailed evaluation metrics to help users assess their model's performance and identify areas for improvement.


Extensive Model Zoo & Configurations

MMDetection offers a vast model zoo with numerous pre-trained models and configuration files for diverse computer vision tasks, such as object detection, instance segmentation, and panoptic segmentation.


MMDetection Backbone

Backbones pre-trained convolutional neural networks (CNNs) to extract feature maps. Popular backbones include ResNet, VGG, and MobileNet.


MMDetection Head

These components are meant for specific tasks, e.g. to generate the final predictions, such as bounding boxes, class labels, or masks. Examples include RPN (Region Proposal Network), FCOS (Fully Convolutional One-Stage Object Detector), and Mask Head. Neck: Components, like FPN (Feature Pyramid Network) and PAN (Path Aggregation Network), refine and consolidate features extracted by backbones, connecting them to the head.


MMDetection ROI Extractor

Region of Interest Extractor is a critical MMDetection component extracting RoI features from the feature maps generated by the backbone and neck components, improving the accuracy of final predictions (e.g., bounding boxes and class labels). One of the most popular methods for RoI feature extraction is RoI Align (a technique that addresses the issue of misalignment between RoI features and the input image due to quantization in RoI Pooling).