R

random_split() (deeplake.core.dataset.Dataset method) read() (in module deeplake) (in module deeplake.api.read) read_only (deeplake.core.dataset.Dataset property) ReadOnlyModeError (class in deeplake.util.exceptions) rechunk() (deeplake.core.dataset.Dataset method) register_deeplake_object() (deeplake.core.storage.LRUCache method) remove_deeplake_object() (deeplake.core.storage.LRUCache method) remove_memory_cache() (in module deeplake.util.remove_cache) rename() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.S3Provider method) (in module deeplake) rename_group() (deeplake.core.dataset.Dataset method) rename_tensor() (deeplake.core.dataset.Dataset method) RenameError (class in deeplake.util.exceptions) replace_with() (deeplake.api.info.Info method) reset() (deeplake.core.dataset.Dataset method) ResourceNotFoundException (class in deeplake.util.exceptions) root (deeplake.core.dataset.Dataset property)


S

S3DeletionError (class in deeplake.util.exceptions) S3Error (class in deeplake.util.exceptions) S3GetError (class in deeplake.util.exceptions) S3ListError (class in deeplake.util.exceptions) S3Provider (class in deeplake.core.storage) S3SetError (class in deeplake.util.exceptions) SamePathException (class in deeplake.util.exceptions) Sample (class in deeplake.core.sample) sample_by() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method) sample_indices (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) sample_info (deeplake.core.tensor.Tensor property) SampleCompressionError (class in deeplake.util.exceptions) SampleDecompressionError (class in deeplake.util.exceptions) SampleHtypeMismatchError (class in deeplake.util.exceptions) save_view() (deeplake.core.dataset.Dataset method) ServerException (class in deeplake.util.exceptions) set_bytes() (deeplake.core.storage.StorageProvider method) setdefault() (deeplake.api.info.Info method) shape (deeplake.core.tensor.Tensor property) shape_interval (deeplake.core.tensor.Tensor property) ShapeInterval (class in deeplake.util.shape_interval) shuffle() (deeplake.enterprise.DeepLakeDataLoader method) size_approx() (deeplake.core.dataset.Dataset method) slice_at_int() (in module deeplake.core.index) slice_length() (in module deeplake.core.index) StorageProvider (class in deeplake.core.storage) structure() (deeplake.auto.unstructured.image_classification.ImageClassification method) StructuredDataset (class in deeplake.auto.structured.base) subscriptable() (deeplake.core.index.IndexEntry method) summary() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) sync() (deeplake.core.storage.GDriveProvider method)


U

UnableToReadFromUrlError (class in deeplake.util.exceptions) uncompressed_bytes() (deeplake.core.sample.Sample method) UnexpectedStatusCodeException (class in deeplake.util.exceptions) UnstructuredDataset (class in deeplake.auto.unstructured.base) UnsupportedCompressionError (class in deeplake.util.exceptions) UnsupportedSchedulerError (class in deeplake.util.exceptions) UnsupportedTensorTypeError (class in deeplake.util.exceptions) update() (deeplake.api.info.Info method) update_creds_key() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) UserNotLoggedInException (class in deeplake.util.exceptions)


T

Tensor (class in deeplake.core.tensor) TensorAlreadyExistsError (class in deeplake.util.exceptions) TensorDoesNotExistError (class in deeplake.util.exceptions) TensorDtypeMismatchError (class in deeplake.util.exceptions) tensorflow() (deeplake.core.dataset.Dataset method) TensorGroupAlreadyExistsError (class in deeplake.util.exceptions) TensorGroupDoesNotExistError (class in deeplake.util.exceptions) TensorInvalidSampleShapeError (class in deeplake.util.exceptions) TensorMetaInvalidHtype (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteKey (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteValue (class in deeplake.util.exceptions) TensorMetaMissingKey (class in deeplake.util.exceptions) TensorMetaMissingRequiredValue (class in deeplake.util.exceptions) TensorMetaMutuallyExclusiveKeysError (class in deeplake.util.exceptions) TensorMismatchError (class in deeplake.util.exceptions) TensorModifiedError (class in deeplake.util.exceptions) tensors (deeplake.core.dataset.Dataset property) TensorUnsupportedSampleType (class in deeplake.util.exceptions) text() (deeplake.core.tensor.Tensor method) tiled() (in module deeplake) (in module deeplake.api.tiled) timestamps (deeplake.core.tensor.Tensor property) tobytes() (deeplake.core.tensor.Tensor method) token (deeplake.core.dataset.Dataset property) (deeplake.core.dataset.DeepLakeCloudDataset property) TokenPermissionError (class in deeplake.util.exceptions) transform() (deeplake.enterprise.DeepLakeDataLoader method) TransformError (class in deeplake.util.exceptions)


W

WaitTimeoutException (class in deeplake.util.exceptions)


V

validate() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) values() (deeplake.api.info.Info method) verify (deeplake.core.tensor.Tensor property) VersionControlError (class in deeplake.util.exceptions) ViewEntry (class in deeplake.core.dataset) visualize() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method)


O

optimize() (deeplake.core.dataset.ViewEntry method) OutOfChunkCountError (class in deeplake.util.exceptions) OutOfSampleCountError (class in deeplake.util.exceptions) OverLimitException (class in deeplake.util.exceptions)


Python Module Index

d d deeplake deeplake.api.info deeplake.integrations.wandb.wandb


Installation¶

Deep Lake can be installed with pip Copy to clipboard Deep Lake has the following extras that you can choose to install according to your needs. Installation commands¶ Install command Description Dependencies installed pip install 'deeplake[av]' Audio and video support via PyAV av pip install 'deeplake[visualizer]' Visualize Deep Lake datasets within notebooks. This is required for Dataset.visualize to work. IPython, flask pip install 'deeplake[gcp]' GCS support google-cloud-storage, google-auth, google-auth-oauthlib pip install 'deeplake[dicom]' DICOM data support pydicom pip install 'deeplake[gdrive]' Google Drive support google-api-python-client, oauth2client, google-auth, google-auth-oauthlib pip install 'deeplake[point_cloud]' Support for LiDAR point cloud data laspy pip install 'deeplake[enterprise]' Installs Deep Lake's enterprise features libdeeplake pip install 'deeplake[all]' Installs all of the above


Datasets¶

Dataset.commit_id The lasted committed commit id of the dataset .  Dataset.branch The current branch of the dataset Dataset.pending_commit_id The commit_id of the next commit that will be made to the dataset.  Dataset.has_head_changes Returns True if currently at head node and uncommitted changes are present.  Dataset.commits Lists all the commits leading to the current dataset state.  Dataset.branches Lists all the branches of the dataset.  Dataset Views¶ A dataset view is a subset of a dataset that points to specific samples (indices) in an existing dataset.  Dataset views can be created by indexing a dataset, filtering a dataset with Dataset.filter(), querying a dataset with Dataset.query() or by sampling a dataset with Dataset.sample_by().  Filtering is done with user-defined functions or simplified expressions whereas query can perform SQL-like queries with our Tensor Query Language.  See the full TQL spec here.  Dataset views can only be saved when a dataset has been committed and has no changes on the HEAD node, in order to preserve data lineage and prevent the underlying data from changing after the query or filter conditions have been evaluated.  Example >>> import deeplake >>> # load dataset >>> ds = deeplake.load('hub://activeloop/mnist-train') >>> # filter dataset >>> zeros = ds.filter('labels == 0') >>> # save view >>> zeros.save_view(id='zeros') >>> # load_view >>> zeros = ds.load_view(id='zeros') >>> len(zeros) 5923 Copy to clipboard Dataset.query Returns a sliced Dataset with given query results.  Dataset.sample_by Returns a sliced Dataset with given weighted sampler applied.  Dataset.filter Filters the dataset in accordance of filter function f(x: sample) -> bool Dataset.save_view Saves a dataset view as a virtual dataset (VDS) Dataset.get_view Returns the dataset view corresponding to id.  Dataset.load_view Loads the view and returns the Dataset by id.  Dataset.delete_view Deletes the view with given view id.  Dataset.get_views Returns list of views stored in this Dataset.  Dataset.is_view Returns True if this dataset is a view and False otherwise.  Dataset.min_view Returns a view of the dataset in which all tensors are sliced to have the same length as the shortest tensor.  Dataset.max_view Returns a view of the dataset in which shorter tensors are padded with None s to have the same length as the longest tensor.


Datasets¶

Creating Datasets¶ deeplake.dataset Returns a Dataset object referencing either a new or existing dataset.  deeplake.empty Creates an empty dataset deeplake.like Creates a new dataset by copying the source dataset's structure to a new location.  deeplake.ingest Ingests a dataset from a source and stores it as a structured dataset to destination.  deeplake.ingest_kaggle Download and ingest a kaggle dataset and store it as a structured dataset to destination.  deeplake.ingest_dataframe Convert pandas dataframe to a Deep Lake Dataset.  deeplake.ingest_huggingface Converts Hugging Face datasets to Deep Lake format.  Loading Datasets¶ deeplake.load Loads an existing dataset Deleting and Renaming Datasets¶ deeplake.delete Deletes a dataset at a given path.  deeplake.rename Renames dataset at old_path to new_path.  Copying Datasets¶ deeplake.copy Copies dataset at src to dest.  deeplake.deepcopy Copies dataset at src to dest including version control history.  Dataset Operations¶ Dataset.summary Prints a summary of the dataset.  Dataset.append Append samples to mutliple tensors at once.  Dataset.extend Appends multiple rows of samples to mutliple tensors at once.  Dataset.query Returns a sliced Dataset with given query results.  Dataset.copy Copies this dataset or dataset view to dest.  Dataset.delete Deletes the entire dataset from the cache layers (if any) and the underlying storage.  Dataset.rename Renames the dataset to path.  Dataset.visualize Visualizes the dataset in the Jupyter notebook.  Dataset.pop Removes a sample from all the tensors of the dataset.  Dataset.rechunk Rewrites the underlying chunks to make their sizes optimal.  Dataset.flush Necessary operation after writes if caches are being used.  Dataset.clear_cache Flushes (see Dataset.flush()) the contents of the cache layers (if any) and then deletes contents of all the layers of it.  Dataset.size_approx Estimates the size in bytes of the dataset.  Dataset Visualization¶ Dataset.visualize Visualizes the dataset in the Jupyter notebook.  Dataset Credentials¶ Dataset.add_creds_key Adds a new creds key to the dataset.  Dataset.populate_creds Populates the creds key added in add_creds_key with the given creds.  Dataset.update_creds_key Replaces the old creds key with the new creds key.  Dataset.change_creds_management Changes the management status of the creds key.  Dataset.get_creds_keys Returns the list of creds keys added to the dataset.  Dataset Properties¶ Dataset.tensors All tensors belonging to this group, including those within sub groups.  Dataset.groups All sub groups in this group Dataset.num_samples Returns the length of the smallest tensor.  Dataset.read_only Returns True if dataset is in read-only mode and False otherwise.  Dataset.info Returns the information about the dataset.  Dataset.max_len Return the maximum length of the tensor.  Dataset.min_len Return the minimum length of the tensor.  Dataset Version Control¶ Dataset.commit Stores a snapshot of the current state of the dataset.  Dataset.diff Returns/displays the differences between commits/branches.  Dataset.checkout Checks out to a specific commit_id or branch.  Dataset.merge Merges the target_id into the current dataset.  Dataset.log Displays the details of all the past commits.  Dataset.reset Resets the uncommitted changes present in the branch.  Dataset.get_commit_details Get details of a particular commit


Deleting and Renaming Datasets¶

deeplake.delete Deletes a dataset at a given path. deeplake.rename Renames dataset at old_path to new_path.


Creating Datasets¶

deeplake.dataset Returns a Dataset object referencing either a new or existing dataset. deeplake.empty Creates an empty dataset deeplake.like Creates a new dataset by copying the source dataset's structure to a new location. deeplake.ingest Ingests a dataset from a source and stores it as a structured dataset to destination. deeplake.ingest_kaggle Download and ingest a kaggle dataset and store it as a structured dataset to destination. deeplake.ingest_dataframe Convert pandas dataframe to a Deep Lake Dataset. deeplake.ingest_huggingface Converts Hugging Face datasets to Deep Lake format.


Copying Datasets¶

deeplake.copy Copies dataset at src to dest. deeplake.deepcopy Copies dataset at src to dest including version control history.


Dataset Properties¶

Dataset.tensors All tensors belonging to this group, including those within sub groups. Dataset.groups All sub groups in this group Dataset.num_samples Returns the length of the smallest tensor. Dataset.read_only Returns True if dataset is in read-only mode and False otherwise. Dataset.info Returns the information about the dataset. Dataset.max_len Return the maximum length of the tensor. Dataset.min_len Return the minimum length of the tensor.


Loading Datasets¶

deeplake.load Loads an existing dataset


Dataset Visualization¶

Dataset.visualize Visualizes the dataset in the Jupyter notebook.


Dataset Operations¶

Dataset.summary Prints a summary of the dataset. Dataset.append Append samples to mutliple tensors at once. Dataset.extend Appends multiple rows of samples to mutliple tensors at once. Dataset.query Returns a sliced Dataset with given query results. Dataset.copy Copies this dataset or dataset view to dest. Dataset.delete Deletes the entire dataset from the cache layers (if any) and the underlying storage. Dataset.rename Renames the dataset to path. Dataset.visualize Visualizes the dataset in the Jupyter notebook. Dataset.pop Removes a sample from all the tensors of the dataset. Dataset.rechunk Rewrites the underlying chunks to make their sizes optimal. Dataset.flush Necessary operation after writes if caches are being used. Dataset.clear_cache Flushes (see Dataset.flush()) the contents of the cache layers (if any) and then deletes contents of all the layers of it. Dataset.size_approx Estimates the size in bytes of the dataset.


Dataset Views¶

A dataset view is a subset of a dataset that points to specific samples (indices) in an existing dataset. Dataset views can be created by indexing a dataset, filtering a dataset with Dataset.filter(), querying a dataset with Dataset.query() or by sampling a dataset with Dataset.sample_by(). Filtering is done with user-defined functions or simplified expressions whereas query can perform SQL-like queries with our Tensor Query Language. See the full TQL spec here.Dataset views can only be saved when a dataset has been committed and has no changes on the HEAD node, in order to preserve data lineage and prevent the underlying data from changing after the query or filter conditions have been evaluated.Example Copy to clipboard Dataset.query Returns a sliced Dataset with given query results. Dataset.sample_by Returns a sliced Dataset with given weighted sampler applied. Dataset.filter Filters the dataset in accordance of filter function f(x: sample) -> bool Dataset.save_view Saves a dataset view as a virtual dataset (VDS) Dataset.get_view Returns the dataset view corresponding to id. Dataset.load_view Loads the view and returns the Dataset by id. Dataset.delete_view Deletes the view with given view id. Dataset.get_views Returns list of views stored in this Dataset. Dataset.is_view Returns True if this dataset is a view and False otherwise. Dataset.min_view Returns a view of the dataset in which all tensors are sliced to have the same length as the shortest tensor. Dataset.max_view Returns a view of the dataset in which shorter tensors are padded with None s to have the same length as the longest tensor.


Dataset Credentials¶

Dataset.add_creds_key Adds a new creds key to the dataset. Dataset.populate_creds Populates the creds key added in add_creds_key with the given creds. Dataset.update_creds_key Replaces the old creds key with the new creds key. Dataset.change_creds_management Changes the management status of the creds key. Dataset.get_creds_keys Returns the list of creds keys added to the dataset.


Dataset Version Control¶

Dataset.commit Stores a snapshot of the current state of the dataset. Dataset.diff Returns/displays the differences between commits/branches. Dataset.checkout Checks out to a specific commit_id or branch. Dataset.merge Merges the target_id into the current dataset. Dataset.log Displays the details of all the past commits. Dataset.reset Resets the uncommitted changes present in the branch. Dataset.get_commit_details Get details of a particular commit. Dataset.commit_id The lasted committed commit id of the dataset. Dataset.branch The current branch of the dataset Dataset.pending_commit_id The commit_id of the next commit that will be made to the dataset. Dataset.has_head_changes Returns True if currently at head node and uncommitted changes are present. Dataset.commits Lists all the commits leading to the current dataset state. Dataset.branches Lists all the branches of the dataset.


Tensors¶

Creating Tensors¶ Dataset.create_tensor Creates a new tensor in the dataset. Dataset.create_group Creates a tensor group. Dataset.create_tensor_like Copies the source tensor's meta information and creates a new tensor with it. Deleting and Renaming Tensors¶ Dataset.delete_tensor Delete a tensor from the dataset. Dataset.delete_group Delete a tensor group from the dataset. Dataset.rename_tensor Renames tensor with name name to new_name Dataset.rename_group Renames group with name name to new_name Adding and deleting samples¶ Tensor.append Appends a single sample to the end of the tensor. Tensor.extend Extends the end of the tensor by appending multiple elements from a sequence. Tensor.pop Removes an element at the given index. Tensor.clear Deletes all samples from the tensor Tensor.__setitem__ Update samples with new values. Retrieving samples¶ Tensor.numpy Computes the contents of the tensor in numpy format. Tensor.data Returns data in the tensor in a format based on the tensor's base htype. Tensor.tobytes Returns the bytes of the tensor. Tensor.text Return text data. Tensor.dict Return json data. Tensor.list Return list data. Tensor._linked_sample Returns the linked sample at the given index. Tensor Properties¶ Tensor.htype Htype of the tensor. Tensor.base_htype Base htype of the tensor. Tensor.dtype Dtype of the tensor. Tensor.shape Get the shape of this tensor. Tensor.shape_interval Returns a ShapeInterval object that describes this tensor's shape more accurately. Tensor.ndim Number of dimensions of the tensor. Tensor.num_samples Returns the length of the primary axis of the tensor. Tensor.__len__ Returns the length of the primary axis of the tensor. Tensor.is_dynamic Will return True if samples in this tensor have shapes that are unequal. Tensor.is_sequence Whether this tensor is a sequence tensor. Tensor.is_link Whether this tensor is a link tensor. Tensor.verify Whether linked data will be verified when samples are added. Info¶ Tensor.info Returns the information about the tensor. Tensor.sample_info Returns info about particular samples in a tensor. Video features¶ Tensor.play Play video sample. Tensor.timestamps Returns timestamps (in seconds) for video sample as numpy array.


Adding and deleting samples¶

Tensor.append Appends a single sample to the end of the tensor. Tensor.extend Extends the end of the tensor by appending multiple elements from a sequence. Tensor.pop Removes an element at the given index. Tensor.clear Deletes all samples from the tensor Tensor.__setitem__ Update samples with new values.


Retrieving samples¶

Tensor.numpy Computes the contents of the tensor in numpy format. Tensor.data Returns data in the tensor in a format based on the tensor's base htype. Tensor.tobytes Returns the bytes of the tensor. Tensor.text Return text data. Tensor.dict Return json data. Tensor.list Return list data. Tensor._linked_sample Returns the linked sample at the given index.


Tensor Properties¶

Tensor.htype Htype of the tensor. Tensor.base_htype Base htype of the tensor. Tensor.dtype Dtype of the tensor. Tensor.shape Get the shape of this tensor. Tensor.shape_interval Returns a ShapeInterval object that describes this tensor's shape more accurately. Tensor.ndim Number of dimensions of the tensor. Tensor.num_samples Returns the length of the primary axis of the tensor. Tensor.__len__ Returns the length of the primary axis of the tensor. Tensor.is_dynamic Will return True if samples in this tensor have shapes that are unequal. Tensor.is_sequence Whether this tensor is a sequence tensor. Tensor.is_link Whether this tensor is a link tensor. Tensor.verify Whether linked data will be verified when samples are added.


Deleting and Renaming Tensors¶

Dataset.delete_tensor Delete a tensor from the dataset. Dataset.delete_group Delete a tensor group from the dataset. Dataset.rename_tensor Renames tensor with name name to new_name Dataset.rename_group Renames group with name name to new_name


Video features¶

Tensor.play Play video sample. Tensor.timestamps Returns timestamps (in seconds) for video sample as numpy array.


Info¶

Tensor.info Returns the information about the tensor. Tensor.sample_info Returns info about particular samples in a tensor.


Creating Tensors¶

Dataset.create_tensor Creates a new tensor in the dataset. Dataset.create_group Creates a tensor group. Dataset.create_tensor_like Copies the source tensor's meta information and creates a new tensor with it.


Compressions¶

Deep Lake can read, compress, decompress and recompress data to different formats. The supported htype-compression configurations are given below. Sample Type Htype Compressions Image image bmp, dib, gif, ico, jpeg, jpeg2000, pcx, png, ppm, sgi, tga, tiff, webp, wmf, xbm, eps, fli, im, msp, mpo, apng Video video mp4, mkv, avi Audio audio flac, mp3, wav Dicom dicom dcm Point Cloud point_cloud las Other bbox, text, list, json, generic, etc. lz4 Sample Compression¶ If sample compression is specified when creating tensors, samples will be compressed to the given format if possible. If given data is already compressed and matches the provided sample_compression, it will be stored as is. If left as None, given samples are uncompressed. Note For audio and video, we don't support compressing raw frames but only reading compressed audio and video data. Examples: >>> ds.create_tensor('images', htype='image', sample_compression='jpg') >>> ds.create_tensor('videos', htype='video', sample_compression='mp4') >>> ds.create_tensor('point_clouds', htype='point_cloud', sample_compression='las') Copy to clipboard Structure of sample-wise compressed tensor.¶ Chunk Compression¶ If chunk compression is specified when creating tensors, addded samples will be clubbed together and compressed to the given format chunk-wise. If given data is already compressed, it will be uncompressed and then recompressed chunk-wise. Note Chunk-wise compression is not supported for audio, video and point_cloud htypes. Examples: >>> ds.create_tensor('images', htype='image', chunk_compression='jpg') >>> ds.create_tensor('boxes', htype='bbox', chunk_compression='lz4') Copy to clipboard Structure of chunk-wise compressed tensor.¶ Note See deeplake.read() to learn how to read data from files and populate these tensors.


Chunk Compression¶

If chunk compression is specified when creating tensors, addded samples will be clubbed together and compressed to the given format chunk-wise. If given data is already compressed, it will be uncompressed and then recompressed chunk-wise.NoteChunk-wise compression is not supported for audio, video and point_cloud htypes.Examples: Copy to clipboard Structure of chunk-wise compressed tensor.¶ NoteSee deeplake.read() to learn how to read data from files and populate these tensors.


Sample Compression¶

If sample compression is specified when creating tensors, samples will be compressed to the given format if possible. If given data is already compressed and matches the provided sample_compression, it will be stored as is. If left as None, given samples are uncompressed.NoteFor audio and video, we don't support compressing raw frames but only reading compressed audio and video data.Examples: Copy to clipboard Structure of sample-wise compressed tensor.¶


PyTorch and Tensorflow Support¶

Deep Lake datasets can be easily converted to Torch dataloaders or Tensorflow datasets for training. Dataset.pytorch Converts the dataset into a pytorch Dataloader. Dataset.tensorflow Converts the dataset into a tensorflow compatible format.


Making Deep Lake Samples¶

read Utility that reads raw data from supported files into Deep Lake format. link Utility that stores a link to raw data.


General Functions¶

list List all available Deep Lake cloud datasets. exists Checks if a dataset exists at the given path.


Utility Functions¶

General Functions¶ list List all available Deep Lake cloud datasets. exists Checks if a dataset exists at the given path. Making Deep Lake Samples¶ read Utility that reads raw data from supported files into Deep Lake format. link Utility that stores a link to raw data. Parallelism¶ compute Compute is a decorator for functions. compose Takes a list of functions decorated using deeplake.compute() and creates a pipeline that can be evaluated using .eval Transform pipelines returned by compute() and compose() are evaluated using eval: eval Evaluates the pipeline on data_in to produce an output dataset ds_out.


Parallelism¶

compute Compute is a decorator for functions. compose Takes a list of functions decorated using deeplake.compute() and creates a pipeline that can be evaluated using .eval Transform pipelines returned by compute() and compose() are evaluated using eval: eval Evaluates the pipeline on data_in to produce an output dataset ds_out.


Weights and Biases¶

Deep Lake's Weights and Biases integration allows you to track and improve reproducibility of your machine learning experiments. Deep Lake will automatically push all information required to reproduce the snapshot of the data like your dataset's URI, commit ID, and view IDs of any views that you have used in your training workflow.Learn more about Weights and Biases here. Logging Dataset Creation¶ If you create a Deep Lake dataset using any of the functions mentioned in Creating Datasets, just perform a commit on the dataset to log its creation on W&B. >>> run = wandb.init(project='deeplake_wandb', job_type='dataset_upload') >>> ds = deeplake.empty('hub://fayazrahman4u/my_dataset') # create dataset >>> ds.create_tensor('images', htype='image', sample_compression='jpg') # create a tensor >>> ds.images.append(deeplake.read('files/images/dog.jpg')) # add a sample >>> ds.commit('creation') # commit -> trigger logging >>> run.finish() Copy to clipboard Note If you created your dataset using deeplake.deepcopy(), perform the commit only if you have head changes. Note If you make changes to an existing dataset, commit the changes with an active Weights and Biases run to log it's state. Logging Dataset Read¶ A dataset read will be logged if you iterate over a dataset or call Dataset.pytorch() or Tensor.numpy() on its tensors. >>> run = wandb.init(project='deeplake_wandb', job_type='torch dataloader') >>> train_loader = ds.pytorch() >>> run.finish() Copy to clipboard >>> run = wandb.init(project='deeplake_wandb', job_type='iteration') >>> for sample in ds: >>> print(sample['images'].shape) >>> run.finish() Copy to clipboard


Logging Dataset Creation¶

If you create a Deep Lake dataset using any of the functions mentioned in Creating Datasets, just perform a commit on the dataset to log its creation on W&B. Copy to clipboard NoteIf you created your dataset using deeplake.deepcopy(), perform the commit only if you have head changes.NoteIf you make changes to an existing dataset, commit the changes with an active Weights and Biases run to log it's state.


Logging Dataset Read¶

A dataset read will be logged if you iterate over a dataset or call Dataset.pytorch() or Tensor.numpy() on its tensors. Copy to clipboard Copy to clipboard


Dataloader¶

Train your models using the new high performance C++ dataloader.  See the dataloader method on how to create dataloaders from your datasets: Dataset.dataloader Returns a DeepLakeDataLoader object.  DeepLakeDataLoader¶ class deeplake.enterprise.DeepLakeDataLoader¶ batch(batch_size: int, drop_last: bool = False)¶ Returns a batched DeepLakeDataLoader object.  Parameters: batch_size (int) – Number of samples in each batch.  drop_last (bool) – If True, the last batch will be dropped if its size is less than batch_size.  Defaults to False.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .batch() has already been called.  numpy(num_workers: int = 0, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, decode_method: Optional[Dict[str, str]] = None)¶ Returns a DeepLakeDataLoader object.  Parameters: num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are:- ‘numpy': Default behaviour.  Returns samples as numpy arrays.  'tobytes': Returns raw bytes of the samples.  'pil': Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or ‘png'.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .pytorch() or .numpy() has already been called.  pytorch(num_workers: int = 0, collate_fn: Optional[Callable] = None, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, distributed: bool = False, return_index: bool = True, decode_method: Optional[Dict[str, str]] = None)¶ Returns a DeepLakeDataLoader object.  Parameters: num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  collate_fn (Callable, Optional) – merges a list of samples to form a mini-batch of Tensor(s).  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  distributed (bool) – Used for DDP training.  Distributes different sections of the dataset to different ranks.  Defaults to False.  return_index (bool) – Used to idnetify where loader needs to retur sample index or not.  Defaults to True


Dataloader¶

decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor .  Defaults to None.  Supported decode methods are: 'numpy': Default behaviour.  Returns samples as numpy arrays.  'tobytes': Returns raw bytes of the samples.  'pil': Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or ‘png'.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .pytorch() or .numpy() has already been called.  query(query_string: str)¶ Returns a sliced DeepLakeDataLoader object with given query results.  It allows to run SQL like queries on dataset and extract results.  See supported keywords and the Tensor Query Language documentation here.  Parameters: query_string (str) – An SQL string adjusted with new functionalities to run on the dataset object Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Examples >>> import deeplake >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> query_ds_train = ds_train.dataloader().query('select * where labels != 5') Copy to clipboard >>> import deeplake >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> query_ds_train = ds_train.dataloader().query('(select * where contains(categories, 'car') limit 1000) union (select * where contains(categories, 'motorcycle') limit 1000)') Copy to clipboard sample_by(weights: Union[str, list, tuple, ndarray], replace: Optional[bool] = True, size: Optional[int] = None)¶ Returns a sliced DeepLakeDataLoader with given weighted sampler applied Parameters: weights – (Union[str, list, tuple, np.ndarray]): If it's string then tql will be run to calculate the weights based on the expression.  list, tuple and ndarray will be treated as the list of the weights per sample replace – Optional[bool] If true the samples can be repeated in the result view.  (default: True).  size – Optional[int] The length of the result view.  (default: len(dataset)) Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Examples Sample the dataloader with labels == 5 twice more than labels == 6 >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('max_weight(labels == 5: 10, labels == 6: 5)') Copy to clipboard Sample the dataloader treating labels tensor as weights.  >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('labels') Copy to clipboard Sample the dataloader with the given weights; >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> weights = list() >>> for i in range(0, len(ds_train)): ...  weights.append(i % 5) ...  >>> sampled_ds = ds.dataloader().sample_by(weights, replace=False) Copy to clipboard shuffle(shuffle: bool = True, buffer_size: int = 2048)¶ Returns a shuffled DeepLakeDataLoader object.  Parameters: shuffle (bool) – shows wheter we need to shuffle elements or not.  Defaults to True.  buffer_size (int) – The size of the buffer used to shuffle the data in MBs.  Defaults to 2048 MB.  Increasing the buffer_size will increase the extent of shuffling.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .shuffle() has already been called.  ValueError – If dataset is view and shuffle is True transform(transform: Union[Callable, Dict[str, Optional[Callable]]], **kwargs: Dict)¶ Returns a transformed DeepLakeDataLoader object


Dataloader¶

Parameters: transform (Callable or Dict[Callable]) – A function or dictionary of functions to apply to the data .  kwargs – Additional arguments to be passed to transform.  Only applicable if transform is a callable.  Ignored if transform is a dictionary.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .transform() has already been called.


DeepLakeDataLoader¶

class deeplake.enterprise.DeepLakeDataLoader¶ batch(batch_size: int, drop_last: bool = False)¶ Returns a batched DeepLakeDataLoader object.  Parameters: batch_size (int) – Number of samples in each batch.  drop_last (bool) – If True, the last batch will be dropped if its size is less than batch_size.  Defaults to False.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .batch() has already been called.  numpy(num_workers: int = 0, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, decode_method: Optional[Dict[str, str]] = None)¶ Returns a DeepLakeDataLoader object.  Parameters: num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are:- ‘numpy': Default behaviour.  Returns samples as numpy arrays.  'tobytes': Returns raw bytes of the samples.  'pil': Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or ‘png'.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .pytorch() or .numpy() has already been called.  pytorch(num_workers: int = 0, collate_fn: Optional[Callable] = None, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, distributed: bool = False, return_index: bool = True, decode_method: Optional[Dict[str, str]] = None)¶ Returns a DeepLakeDataLoader object.  Parameters: num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  collate_fn (Callable, Optional) – merges a list of samples to form a mini-batch of Tensor(s).  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  distributed (bool) – Used for DDP training.  Distributes different sections of the dataset to different ranks.  Defaults to False.  return_index (bool) – Used to idnetify where loader needs to retur sample index or not.  Defaults to True.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are: 'numpy': Default behaviour.  Returns samples as numpy arrays.  'tobytes': Returns raw bytes of the samples


DeepLakeDataLoader¶

'pil': Returns samples as PIL images .  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or ‘png'.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .pytorch() or .numpy() has already been called.  query(query_string: str)¶ Returns a sliced DeepLakeDataLoader object with given query results.  It allows to run SQL like queries on dataset and extract results.  See supported keywords and the Tensor Query Language documentation here.  Parameters: query_string (str) – An SQL string adjusted with new functionalities to run on the dataset object Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Examples >>> import deeplake >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> query_ds_train = ds_train.dataloader().query('select * where labels != 5') Copy to clipboard >>> import deeplake >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> query_ds_train = ds_train.dataloader().query('(select * where contains(categories, 'car') limit 1000) union (select * where contains(categories, 'motorcycle') limit 1000)') Copy to clipboard sample_by(weights: Union[str, list, tuple, ndarray], replace: Optional[bool] = True, size: Optional[int] = None)¶ Returns a sliced DeepLakeDataLoader with given weighted sampler applied Parameters: weights – (Union[str, list, tuple, np.ndarray]): If it's string then tql will be run to calculate the weights based on the expression.  list, tuple and ndarray will be treated as the list of the weights per sample replace – Optional[bool] If true the samples can be repeated in the result view.  (default: True).  size – Optional[int] The length of the result view.  (default: len(dataset)) Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Examples Sample the dataloader with labels == 5 twice more than labels == 6 >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('max_weight(labels == 5: 10, labels == 6: 5)') Copy to clipboard Sample the dataloader treating labels tensor as weights.  >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('labels') Copy to clipboard Sample the dataloader with the given weights; >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> weights = list() >>> for i in range(0, len(ds_train)): ...  weights.append(i % 5) ...  >>> sampled_ds = ds.dataloader().sample_by(weights, replace=False) Copy to clipboard shuffle(shuffle: bool = True, buffer_size: int = 2048)¶ Returns a shuffled DeepLakeDataLoader object.  Parameters: shuffle (bool) – shows wheter we need to shuffle elements or not.  Defaults to True.  buffer_size (int) – The size of the buffer used to shuffle the data in MBs.  Defaults to 2048 MB.  Increasing the buffer_size will increase the extent of shuffling.  Returns: A DeepLakeDataLoader object.  Return type: DeepLakeDataLoader Raises: ValueError – If .shuffle() has already been called.  ValueError – If dataset is view and shuffle is True transform(transform: Union[Callable, Dict[str, Optional[Callable]]], **kwargs: Dict)¶ Returns a transformed DeepLakeDataLoader object.  Parameters: transform (Callable or Dict[Callable]) – A function or dictionary of functions to apply to the data.  kwargs – Additional arguments to be passed to transform.  Only applicable if transform is a callable.  Ignored if transform is a dictionary.  Returns: A DeepLakeDataLoader object


DeepLakeDataLoader¶

Return type: DeepLakeDataLoader Raises: ValueError – If .transform() has already been called.


Sampler¶

The sampler applies weighted sampling on the dataset and returns the sampled view. It creates a discrete distribution with given weights and randomly picks samples based on it. The resulting view is generated in such a way that when creating a dataloader from the view and training on it, the performance impact is minimal. See the sample_by method on how to use this feature: Dataset.sample_by Returns a sliced Dataset with given weighted sampler applied.


Tensor Query Language¶

This page describes the Tensor Query Language (TQL), an SQL-like language used for Querying in Activeloop Platform as well as in ds.query in our Python API.  To use queries, install deeplake with pip install deeplake[enterprise].  Querying datasets is part of our Growth and Enterprise Plan.  Users of our Community plan can only perform queries on Activeloop datasets ('hub://activeloop/…' datasets).  To run queries on your own datasets, upgrade your organization's plan.  Language¶ SELECT¶ TQL supports only SELECT statement.  Every TQL expression starts with SELECT *.  TQL supports only * which means to select all tensors.  The common syntax for select statement is the following: SELECT * [FROM string] [WHERE expression] [LIMIT number [OFFSET number]] [ORDER BY expression [ASC/DESC]] Copy to clipboard Each part of the SELECT statement can be omitted.  FROM expression is allowed, but it does not have any effect on the query, because for now TQL queries are run on a specific dataset, so the FROM is known from the context WHERE¶ WHERE expression is used to filter the samples in the dataset by conditions.  The conditions should be convertible to boolean.  Any expression which outputs a number will be converted to boolean with non-zero values taken as True.  If the expression is not convertible to boolean, such as strings, json objects and arrays, the query will print the corresponding error.  ORDER BY¶ ORDER BY expression orders the output of the query by the given criteria.  The criteria can be any expression output of which can be ordered.  The ordered outputs are either scalar numbers or strings.  In addition it can also be json, which contains number or string.  ORDER BY statement optionally accepts ASC/DESC keywords specifying whether the ordering should be ascending or descending.  It is ascending by default.  LIMIT OFFSET¶ LIMIT and OFFSET expressions are used to limit the output of the query by index, as in SQL.  Expressions¶ TQL supports any comparison operator (==, !=, <, <=, >=) where the left side is a tensor and the right side is a known value.  The value can be numeric scalar or array as well as string value.  String literal should be provided within single quotes (') and can be used on class_label, json and text tensors.  For class labels it will get corresponding numeric value from the class_names list and do numeric comparison.  For json and text it will do string comparison.  The left side of the expression can be indexed (subscripted) if the tensor is multidimensional array or json.  Jsons support indexing by string, e.g.  index_meta['id'] == 'some_id'.  Jsons can also be indexed by number if the underlying data is array.  Numeric multidimensional tensors can be indexed by numbers, e.g.  categories[0] == 1 as well as Python style slicing and multidimensional indexing, such as boxes[:2]


Tensor Query Language¶

This last expression returns array containing the third elements of the initial two dimensional array boxes .  TQL supports logical operators - AND, OR and NOT.  These operators can be used to combine boolean expressions.  For example, labels == 0 OR labels == 1 Copy to clipboard From SQL we also support the following two keywords: BETWEEN labels BETWEEN 0 and 5 Copy to clipboard IN labels in ARRAY[0, 2, 4, 6, 8] Copy to clipboard Functions¶ There are predefined functions which can be used in WHERE expression as well as in ORDER BY expressions: CONTAINS - checks if the given tensor contains given value - CONTAINS(categories, 'person') RANDOM - returns random number.  May be used in ORDER BY to shuffle the output - ORDER BY RANDOM() SHAPE - returns the shape array of the given tensor - SHAPE(boxes) ALL - takes an array of booleans and returns single boolean, True if all elements of the input array are True ALL_STRICT - same as ALL with one difference.  ALL returns True on empty array, while ALL_STRICT return False ANY - takes an array of booleans and returns single boolean, True if any of the elements int the input array is True LOGICAL_AND - takes two boolean arrays, does element wise logical and, returns the result array.  This will return False if the input arrays have different sizes.  LOGICAL_OR - takes two boolean arrays, does element wise logical or, returns the result array.  This will return False if the input arrays have different sizes.  UNION, INTERSECT, EXCEPT¶ Query can contain multiple SELECT statements, combined by one of the set operations - UNION, INTERSECT and EXCEPT.  Examples¶ Querying for images containing 0 in MNIST Train Dataset with ds.query.  >>> import deeplake >>> ds = deeplake.load('hub://activeloop/mnist-train') >>> result = ds.query('select * where labels == 0') >>> len(result) 5923 Copy to clipboard Querying for samples with car or motorcycle in categories of COCO Train Dataset.  >>> import deeplake >>> ds = deeplake.load('hub://activeloop/coco-train') >>> result = ds.query('(select * where contains(categories, 'car')) union (select * where contains(categories, 'motorcycle'))') >>> len(result) 14376 Copy to clipboard


Language¶

SELECT¶ TQL supports only SELECT statement.  Every TQL expression starts with SELECT *.  TQL supports only * which means to select all tensors.  The common syntax for select statement is the following: SELECT * [FROM string] [WHERE expression] [LIMIT number [OFFSET number]] [ORDER BY expression [ASC/DESC]] Copy to clipboard Each part of the SELECT statement can be omitted.  FROM expression is allowed, but it does not have any effect on the query, because for now TQL queries are run on a specific dataset, so the FROM is known from the context WHERE¶ WHERE expression is used to filter the samples in the dataset by conditions.  The conditions should be convertible to boolean.  Any expression which outputs a number will be converted to boolean with non-zero values taken as True.  If the expression is not convertible to boolean, such as strings, json objects and arrays, the query will print the corresponding error.  ORDER BY¶ ORDER BY expression orders the output of the query by the given criteria.  The criteria can be any expression output of which can be ordered.  The ordered outputs are either scalar numbers or strings.  In addition it can also be json, which contains number or string.  ORDER BY statement optionally accepts ASC/DESC keywords specifying whether the ordering should be ascending or descending.  It is ascending by default.  LIMIT OFFSET¶ LIMIT and OFFSET expressions are used to limit the output of the query by index, as in SQL.  Expressions¶ TQL supports any comparison operator (==, !=, <, <=, >=) where the left side is a tensor and the right side is a known value.  The value can be numeric scalar or array as well as string value.  String literal should be provided within single quotes (') and can be used on class_label, json and text tensors.  For class labels it will get corresponding numeric value from the class_names list and do numeric comparison.  For json and text it will do string comparison.  The left side of the expression can be indexed (subscripted) if the tensor is multidimensional array or json.  Jsons support indexing by string, e.g.  index_meta['id'] == 'some_id'.  Jsons can also be indexed by number if the underlying data is array.  Numeric multidimensional tensors can be indexed by numbers, e.g.  categories[0] == 1 as well as Python style slicing and multidimensional indexing, such as boxes[:2].  This last expression returns array containing the third elements of the initial two dimensional array boxes.  TQL supports logical operators - AND, OR and NOT.  These operators can be used to combine boolean expressions


ORDER BY¶

ORDER BY expression orders the output of the query by the given criteria. The criteria can be any expression output of which can be ordered. The ordered outputs are either scalar numbers or strings. In addition it can also be json, which contains number or string.ORDER BY statement optionally accepts ASC/DESC keywords specifying whether the ordering should be ascending or descending. It is ascending by default.


LIMIT OFFSET¶

LIMIT and OFFSET expressions are used to limit the output of the query by index, as in SQL.


SELECT¶

TQL supports only SELECT statement. Every TQL expression starts with SELECT *. TQL supports only * which means to select all tensors. The common syntax for select statement is the following: Copy to clipboard Each part of the SELECT statement can be omitted.FROM expression is allowed, but it does not have any effect on the query, because for now TQL queries are run on a specific dataset, so the FROM is known from the context


Expressions¶

TQL supports any comparison operator (==, !=, <, <=, >=) where the left side is a tensor and the right side is a known value.The value can be numeric scalar or array as well as string value.String literal should be provided within single quotes (') and can be used on class_label, json and text tensors.For class labels it will get corresponding numeric value from the class_names list and do numeric comparison.For json and text it will do string comparison. The left side of the expression can be indexed (subscripted) if the tensor is multidimensional array or json. Jsons support indexing by string, e.g. index_meta['id'] == 'some_id'. Jsons can also be indexed by number if the underlying data is array.Numeric multidimensional tensors can be indexed by numbers, e.g. categories[0] == 1 as well as Python style slicing and multidimensional indexing, such as boxes[:2]. This last expression returns array containing the third elements of the initial two dimensional array boxes.TQL supports logical operators - AND, OR and NOT. These operators can be used to combine boolean expressions. For example, Copy to clipboard From SQL we also support the following two keywords: BETWEEN Copy to clipboard IN Copy to clipboard


Functions¶

There are predefined functions which can be used in WHERE expression as well as in ORDER BY expressions: CONTAINS - checks if the given tensor contains given value - CONTAINS(categories, 'person') RANDOM - returns random number. May be used in ORDER BY to shuffle the output - ORDER BY RANDOM() SHAPE - returns the shape array of the given tensor - SHAPE(boxes) ALL - takes an array of booleans and returns single boolean, True if all elements of the input array are True ALL_STRICT - same as ALL with one difference. ALL returns True on empty array, while ALL_STRICT return False ANY - takes an array of booleans and returns single boolean, True if any of the elements int the input array is True LOGICAL_AND - takes two boolean arrays, does element wise logical and, returns the result array. This will return False if the input arrays have different sizes. LOGICAL_OR - takes two boolean arrays, does element wise logical or, returns the result array. This will return False if the input arrays have different sizes.


UNION, INTERSECT, EXCEPT¶

Query can contain multiple SELECT statements, combined by one of the set operations - UNION, INTERSECT and EXCEPT.


Examples¶

Querying for images containing 0 in MNIST Train Dataset with ds.query. Copy to clipboard Querying for samples with car or motorcycle in categories of COCO Train Dataset. Copy to clipboard


Language¶

For example, labels == 0 OR labels == 1 Copy to clipboard From SQL we also support the following two keywords: BETWEEN labels BETWEEN 0 and 5 Copy to clipboard IN labels in ARRAY[0, 2, 4, 6, 8] Copy to clipboard Functions¶ There are predefined functions which can be used in WHERE expression as well as in ORDER BY expressions: CONTAINS - checks if the given tensor contains given value - CONTAINS(categories, 'person') RANDOM - returns random number .  May be used in ORDER BY to shuffle the output - ORDER BY RANDOM() SHAPE - returns the shape array of the given tensor - SHAPE(boxes) ALL - takes an array of booleans and returns single boolean, True if all elements of the input array are True ALL_STRICT - same as ALL with one difference.  ALL returns True on empty array, while ALL_STRICT return False ANY - takes an array of booleans and returns single boolean, True if any of the elements int the input array is True LOGICAL_AND - takes two boolean arrays, does element wise logical and, returns the result array.  This will return False if the input arrays have different sizes.  LOGICAL_OR - takes two boolean arrays, does element wise logical or, returns the result array.  This will return False if the input arrays have different sizes.  UNION, INTERSECT, EXCEPT¶ Query can contain multiple SELECT statements, combined by one of the set operations - UNION, INTERSECT and EXCEPT.


WHERE¶

WHERE expression is used to filter the samples in the dataset by conditions. The conditions should be convertible to boolean. Any expression which outputs a number will be converted to boolean with non-zero values taken as True. If the expression is not convertible to boolean, such as strings, json objects and arrays, the query will print the corresponding error.


deeplake¶

The deeplake package provides a database which stores data as compressed chunked arrays that can be stored anywhere and later streamed to deep learning models.  deeplake.dataset(path: Union[str, Path], read_only: Optional[bool] = None, overwrite: bool = False, public: bool = False, memory_cache_size: int = 256, local_cache_size: int = 0, creds: Optional[Union[str, Dict]] = None, token: Optional[str] = None, verbose: bool = True, access_method: str = 'stream')¶ Returns a Dataset object referencing either a new or existing dataset.  Examples >>> ds = deeplake.dataset('hub://username/dataset') >>> ds = deeplake.dataset('s3://mybucket/my_dataset') >>> ds = deeplake.dataset('./datasets/my_dataset', overwrite=True) Copy to clipboard Parameters: path (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  read_only (bool, optional) – Opens dataset in read only mode if this is passed as True.  Defaults to False.  Datasets stored on Deep Lake cloud that your account does not have write access to will automatically open in read mode.  overwrite (bool) – If set to True this overwrites the dataset if it already exists.  Defaults to False.  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to True.  memory_cache_size (int) – The size of the memory cache to be used in MB.  local_cache_size (int) – The size of the local filesystem cache to be used in MB.  creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  verbose (bool) – If True, logs will be printed.  Defaults to True.  access_method (str) – The access method to use for the dataset.  Can be: 'stream' Streams the data from the dataset i.e.  only fetches data when required.  This is the default value.  'download' Downloads the data to the local filesystem to the path specified in environment variable DEEPLAKE_DOWNLOAD_PATH.  This will overwrite DEEPLAKE_DOWNLOAD_PATH


deeplake¶

Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set or if the dataset does not exist .  The ‘download' access method can be modified to specify num_workers and/or scheduler.  For example: ‘download:2:processed' will use 2 workers and use processed scheduler, while ‘download:3' will use 3 workers and default scheduler (threaded), and ‘download:processed' will use a single worker and use processed scheduler.  'local' Downloads the dataset if it doesn't already exist, otherwise loads from local storage.  Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set.  The ‘local' access method can be modified to specify num_workers and/or scheduler to be used in case dataset needs to be downloaded.  If dataset needs to be downloaded, ‘local:2:processed' will use 2 workers and use processed scheduler, while ‘local:3' will use 3 workers and default scheduler (threaded), and ‘local:processed' will use a single worker and use processed scheduler.  Returns: Dataset created using the arguments provided.  Return type: Dataset Raises: AgreementError – When agreement is rejected UserNotLoggedInException – When user is not logged in InvalidTokenException – If the specified token is invalid TokenPermissionError – When there are permission or other errors related to token Danger Setting overwrite to True will delete all of your data if it exists! Be very careful when setting this parameter.  Warning Setting access_method to download will overwrite the local copy of the dataset if it was previously downloaded.  Note Any changes made to the dataset in download / local mode will only be made to the local copy and will not be reflected in the original dataset.  deeplake.empty(path: Union[str, Path], overwrite: bool = False, public: bool = False, memory_cache_size: int = 256, local_cache_size: int = 0, creds: Optional[dict] = None, token: Optional[str] = None, verbose: bool = True) → Dataset¶ Creates an empty dataset Parameters: path (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  overwrite (bool) – If set to True this overwrites the dataset if it already exists.  Defaults to False.  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to False.  memory_cache_size (int) – The size of the memory cache to be used in MB.  local_cache_size (int) – The size of the local filesystem cache to be used in MB


deeplake¶

creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path .  If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  verbose (bool) – If True, logs will be printed.  Defaults to True.  Returns: Dataset created using the arguments provided.  Return type: Dataset Raises: DatasetHandlerError – If a Dataset already exists at the given path and overwrite is False.  UserNotLoggedInException – When user is not logged in InvalidTokenException – If the specified toke is invalid TokenPermissionError – When there are permission or other errors related to token Danger Setting overwrite to True will delete all of your data if it exists! Be very careful when setting this parameter.  deeplake.like(dest: Union[str, Path], src: Union[str, Dataset, Path], tensors: Optional[List[str]] = None, overwrite: bool = False, creds: Optional[dict] = None, token: Optional[str] = None, public: bool = False) → Dataset¶ Creates a new dataset by copying the source dataset's structure to a new location.  No samples are copied, only the meta/info for the dataset and it's tensors.  Parameters: dest – Empty Dataset or Path where the new dataset will be created.  src (Union[str, Dataset]) – Path or dataset object that will be used as the template for the new dataset.  tensors (List[str], optional) – Names of tensors (and groups) to be replicated.  If not specified all tensors in source dataset are considered.  overwrite (bool) – If True and a dataset exists at destination, it will be overwritten.  Defaults to False.  creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to False.  Returns: New dataset object.  Return type: Dataset deeplake.ingest(src: Union[str, Path], dest: Union[str, Path], images_compression: str = 'auto', dest_creds: Optional[Dict] = None, progressbar: bool = True, summary: bool = True, **dataset_kwargs) → Dataset¶ Ingests a dataset from a source and stores it as a structured dataset to destination


deeplake¶

Parameters: src (str, pathlib.Path) – Local path to where the unstructured dataset is stored or path to csv file .  dest (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  images_compression (str) – For image classification datasets, this compression will be used for the images tensor.  If images_compression is 'auto', compression will be automatically determined by the most common extension in the directory.  dest_creds (Optional[Dict]) – A dictionary containing credentials used to access the destination path of the dataset.  progressbar (bool) – Enables or disables ingestion progress bar.  Defaults to True.  summary (bool) – If True, a summary of skipped files will be printed after completion.  Defaults to True.  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function.  Returns: New dataset object with structured dataset.  Return type: Dataset Raises: InvalidPathException – If the source directory does not exist.  SamePathException – If the source and destination path are same.  AutoCompressionError – If the source director is empty or does not contain a valid extension.  InvalidFileExtension – If the most frequent file extension is found to be ‘None' during auto-compression.  Note Currently only local source paths and image classification datasets / csv files are supported for automatic ingestion.  Supported filetypes: png/jpeg/jpg/csv.  All files and sub-directories with unsupported filetypes are ignored.  Valid source directory structures for image classification look like: data/ img0.jpg img1.jpg ...  Copy to clipboard or: data/ class0/ cat0.jpg ...  class1/ dog0.jpg ...  ...  Copy to clipboard or: data/ train/ class0/ img0.jpg ...  ...  val/ class0/ img0.jpg ...  ...  ...  Copy to clipboard Classes defined as sub-directories can be accessed at ds['test/labels'].info.class_names.  Support for train and test sub directories is present under ds['train/images'], ds['train/labels'] and ds['test/images'], ds['test/labels'].  Mapping filenames to classes from an external file is currently not supported.  deeplake.ingest_kaggle(tag: str, src: Union[str, Path], dest: Union[str, Path], exist_ok: bool = False, images_compression: str = 'auto', dest_creds: Optional[Dict] = None, kaggle_credentials: Optional[dict] = None, progressbar: bool = True, summary: bool = True, **dataset_kwargs) → Dataset¶ Download and ingest a kaggle dataset and store it as a structured dataset to destination.  Parameters: tag (str) – Kaggle dataset tag


deeplake¶

Example: 'coloradokb/dandelionimages' points to https://www.kaggle.com/coloradokb/dandelionimages src (str, pathlib.Path) – Local path to where the raw kaggle dataset will be downlaoded to .  dest (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  exist_ok (bool) – If the kaggle dataset was already downloaded and exist_ok is True, ingestion will proceed without error.  images_compression (str) – For image classification datasets, this compression will be used for the images tensor.  If images_compression is 'auto', compression will be automatically determined by the most common extension in the directory.  dest_creds (Optional[Dict]) – A dictionary containing credentials used to access the destination path of the dataset.  kaggle_credentials (dict) – A dictionary containing kaggle credentials {'username':'YOUR_USERNAME', 'key': 'YOUR_KEY'}.  If None, environment variables/the kaggle.json file will be used if available.  progressbar (bool) – Enables or disables ingestion progress bar.  Set to True by default.  summary (bool) – Generates ingestion summary.  Set to True by default.  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function.  See deeplake.dataset().  Returns: New dataset object with structured dataset.  Return type: Dataset Raises: SamePathException – If the source and destination path are same.  Note Currently only local source paths and image classification datasets are supported for automatic ingestion.  deeplake.ingest_dataframe(src, dest: Union[str, Path, Dataset], dest_creds: Optional[Dict] = None, progressbar: bool = True, **dataset_kwargs)¶ Convert pandas dataframe to a Deep Lake Dataset.  Parameters: src (pd.DataFrame) – The pandas dataframe to be converted.  dest (str, pathlib.Path, Dataset) – A Dataset or The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  dest_creds (Optional[Dict]) – A dictionary containing credentials used to access the destination path of the dataset.  progressbar (bool) – Enables or disables ingestion progress bar.  Set to True by default


deeplake¶

**dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function .  See deeplake.dataset().  Returns: New dataset created from the dataframe.  Return type: Dataset Raises: Exception – If src is not a valid pandas dataframe object.  deeplake.ingest_huggingface(src, dest, use_progressbar=True) → Dataset¶ Converts Hugging Face datasets to Deep Lake format.  Parameters: src (hfDataset, DatasetDict) – Hugging Face Dataset or DatasetDict to be converted.  Data in different splits of a DatasetDict will be stored under respective tensor groups.  dest (Dataset, str, pathlib.Path) – Destination dataset or path to it.  use_progressbar (bool) – Defines if progress bar should be used to show conversion progress.  Returns: The destination Deep Lake dataset.  Return type: Dataset Note if DatasetDict looks like: >>> { ...  train: Dataset({ ...  features: ['data'] ...  }), ...  validation: Dataset({ ...  features: ['data'] ...  }), ...  test: Dataset({ ...  features: ['data'] ...  }), ...  } Copy to clipboard it will be converted to a Deep Lake Dataset with tensors ['train/data', 'validation/data', 'test/data'].  Features of the type Sequence(feature=Value(dtype='string')) are not supported.  Columns of such type are skipped.  deeplake.load(path: Union[str, Path], read_only: Optional[bool] = None, memory_cache_size: int = 256, local_cache_size: int = 0, creds: Optional[dict] = None, token: Optional[str] = None, verbose: bool = True, access_method: str = 'stream') → Dataset¶ Loads an existing dataset Parameters: path (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  read_only (bool, optional) – Opens dataset in read only mode if this is passed as True.  Defaults to False.  Datasets stored on Deep Lake cloud that your account does not have write access to will automatically open in read mode.  memory_cache_size (int) – The size of the memory cache to be used in MB.  local_cache_size (int) – The size of the local filesystem cache to be used in MB.  creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset


deeplake¶

This is optional, tokens are normally autogenerated .  verbose (bool) – If True, logs will be printed.  Defaults to True.  access_method (str) – The access method to use for the dataset.  Can be: 'stream' Streams the data from the dataset i.e.  only fetches data when required.  This is the default value.  'download' Downloads the data to the local filesystem to the path specified in environment variable DEEPLAKE_DOWNLOAD_PATH.  This will overwrite DEEPLAKE_DOWNLOAD_PATH.  Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set or if the dataset does not exist.  The ‘download' access method can be modified to specify num_workers and/or scheduler.  For example: ‘download:2:processed' will use 2 workers and use processed scheduler, while ‘download:3' will use 3 workers and default scheduler (threaded), and ‘download:processed' will use a single worker and use processed scheduler.  'local' Downloads the dataset if it doesn't already exist, otherwise loads from local storage.  Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set.  The ‘local' access method can be modified to specify num_workers and/or scheduler to be used in case dataset needs to be downloaded.  If dataset needs to be downloaded, ‘local:2:processed' will use 2 workers and use processed scheduler, while ‘local:3' will use 3 workers and default scheduler (threaded), and ‘local:processed' will use a single worker and use processed scheduler.  Returns: Dataset loaded using the arguments provided.  Return type: Dataset Raises: DatasetHandlerError – If a Dataset does not exist at the given path.  AgreementError – When agreement is rejected UserNotLoggedInException – When user is not logged in InvalidTokenException – If the specified toke is invalid TokenPermissionError – When there are permission or other errors related to token Warning Setting access_method to download will overwrite the local copy of the dataset if it was previously downloaded.  Note Any changes made to the dataset in download / local mode will only be made to the local copy and will not be reflected in the original dataset.  deeplake.delete(path: Union[str, Path], force: bool = False, large_ok: bool = False, creds: Optional[dict] = None, token: Optional[str] = None, verbose: bool = False) → None¶ Deletes a dataset at a given path.  Parameters: path (str, pathlib.Path) – The path to the dataset to be deleted.  force (bool) – Delete data regardless of whether it looks like a deeplake dataset.  All data at the path will be removed if set to True.  large_ok (bool) – Delete datasets larger than 1GB.  Disabled by default.  creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys


deeplake¶

When set to 0, it will always use serial processing, irrespective of the scheduler .  scheduler (str) – The scheduler to be used for copying.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool) – Displays a progress bar if True (default).  Returns: New dataset object.  Return type: Dataset Raises: DatasetHandlerError – If a dataset already exists at destination path and overwrite is False.  deeplake.deepcopy(src: Union[str, Path], dest: Union[str, Path], tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, src_token=None, dest_creds=None, dest_token=None, num_workers: int = 0, scheduler='threaded', progressbar=True, public: bool = False, verbose: bool = True)¶ Copies dataset at src to dest including version control history.  Parameters: src (str, pathlib.Path) – Path to the dataset to be copied.  dest (str, pathlib.Path) – Destination path to copy to.  tensors (List[str], optional) – Names of tensors (and groups) to be copied.  If not specified all tensors are copied.  overwrite (bool) – If True and a dataset exists at destination, it will be overwritten.  Defaults to False.  src_creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  src_token (str, optional) – Activeloop token, used for fetching credentials to the dataset at src if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  dest_creds (dict, optional) – creds required to create / overwrite datasets at dest.  dest_token (str, optional) – token used to for fetching credentials to dest.  num_workers (int) – The number of workers to use for copying.  Defaults to 0.  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str) – The scheduler to be used for copying.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool) – Displays a progress bar if True (default).  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to False.  verbose (bool) – If True, logs will be printed.  Defaults to True.  Returns: New dataset object.  Return type: Dataset Raises: DatasetHandlerError – If a dataset already exists at destination path and overwrite is False.  deeplake.connect(src_path: str, creds_key: str, dest_path: Optional[str] = None, org_id: Optional[str] = None, ds_name: Optional[str] = None, token: Optional[str] = None) → Dataset¶ Connects dataset at src_path to Deep Lake via the provided path.  Examples >>> # Connect an s3 dataset >>> ds = deeplake.connect(src_path='s3://bucket/dataset', dest_path='hub://my_org/dataset', creds_key='my_managed_credentials_key') >>> # or >>> ds = deeplake.connect(src_path='s3://bucket/dataset', org_id='my_org', creds_key='my_managed_credentials_key') Copy to clipboard Parameters: src_path (str) – Cloud path to the source dataset


deeplake¶

token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset .  This is optional, tokens are normally autogenerated.  verbose (bool) – If True, logs will be printed.  Defaults to True.  Raises: DatasetHandlerError – If a Dataset does not exist at the given path and force = False.  NotImplementedError – When attempting to delete a managed view.  Warning This is an irreversible operation.  Data once deleted cannot be recovered.  deeplake.rename(old_path: Union[str, Path], new_path: Union[str, Path], creds: Optional[dict] = None, token: Optional[str] = None) → Dataset¶ Renames dataset at old_path to new_path.  Examples >>> deeplake.rename('hub://username/image_ds', 'hub://username/new_ds') >>> deeplake.rename('s3://mybucket/my_ds', 's3://mybucket/renamed_ds') Copy to clipboard Parameters: old_path (str, pathlib.Path) – The path to the dataset to be renamed.  new_path (str, pathlib.Path) – Path to the dataset after renaming.  creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  This takes precedence over credentials present in the environment.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url' and ‘aws_region' as keys.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  Returns: The renamed Dataset.  Return type: Dataset Raises: DatasetHandlerError – If a Dataset does not exist at the given path or if new path is to a different directory.  deeplake.copy(src: Union[str, Path, Dataset], dest: Union[str, Path], tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, src_token=None, dest_creds=None, dest_token=None, num_workers: int = 0, scheduler='threaded', progressbar=True)¶ Copies dataset at src to dest.  Version control history is not included.  Parameters: src (Union[str, Dataset, pathlib.Path]) – The Dataset or the path to the dataset to be copied.  dest (str, pathlib.Path) – Destination path to copy to.  tensors (List[str], optional) – Names of tensors (and groups) to be copied.  If not specified all tensors are copied.  overwrite (bool) – If True and a dataset exists at dest, it will be overwritten.  Defaults to False.  src_creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  src_token (str, optional) – Activeloop token, used for fetching credentials to the dataset at src if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  dest_creds (dict, optional) – creds required to create / overwrite datasets at dest.  dest_token (str, optional) – token used to for fetching credentials to dest.  num_workers (int) – The number of workers to use for copying.  Defaults to 0


deeplake¶

Can be: an s3 path like s3://bucket/path/to/dataset .  a gcs path like gcs://bucket/path/to/dataset.  creds_key (str) – The managed credentials to be used for accessing the source path.  dest_path (str, optional) – The full path to where the connected Deep Lake dataset will reside.  Can be: a Deep Lake path like hub://organization/dataset org_id (str, optional) – The organization to where the connected Deep Lake dataset will be added.  ds_name (str, optional) – The name of the connected Deep Lake dataset.  Will be infered from dest_path or src_path if not provided.  token (str, optional) – Activeloop token used to fetch the managed credentials.  Returns: The connected Deep Lake dataset.  Return type: Dataset Raises: InvalidSourcePathError – If the src_path is not a valid s3 or gcs path.  InvalidDestinationPathError – If dest_path, or org_id and ds_name do not form a valid Deep Lake path.  deeplake.list(workspace: str = '', token: Optional[str] = None) → None¶ List all available Deep Lake cloud datasets.  Parameters: workspace (str) – Specify user/organization name.  If not given, returns a list of all datasets that can be accessed, regardless of what workspace they are in.  Otherwise, lists all datasets in the given workspace.  token (str, optional) – Activeloop token, used for fetching credentials for Deep Lake datasets.  This is optional, tokens are normally autogenerated.  Returns: List of dataset names.  Return type: List deeplake.exists(path: Union[str, Path], creds: Optional[dict] = None, token: Optional[str] = None) → bool¶ Checks if a dataset exists at the given path.  Parameters: path (str, pathlib.Path) – the path which needs to be checked.  creds (dict, optional) – A dictionary containing credentials used to access the dataset at the path.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  Returns: A boolean confirming whether the dataset exists or not at the given path.  deeplake.read(path: Union[str, Path], verify: bool = False, creds: Optional[Dict] = None, compression: Optional[str] = None, storage: Optional[StorageProvider] = None) → Sample¶ Utility that reads raw data from supported files into Deep Lake format.  Recompresses data into format required by the tensor if permitted by the tensor htype.  Simply copies the data in the file if file format matches sample_compression of the tensor, thus maximizing upload speeds.  Examples >>> ds.create_tensor('images', htype='image', sample_compression='jpeg') >>> ds.images.append(deeplake.read('path/to/cat.jpg')) >>> ds.images.shape (1, 399, 640, 3) Copy to clipboard >>> ds.create_tensor('videos', htype='video', sample_compression='mp4') >>> ds.videos.append(deeplake.read('path/to/video.mp4')) >>> ds.videos.shape (1, 136, 720, 1080, 3) Copy to clipboard >>> ds.create_tensor('images', htype='image', sample_compression='jpeg') >>> ds.images.append(deeplake.read('https://picsum.photos/200/300')) >>> ds.images[0].shape (300, 200, 3) Copy to clipboard Supported file types: Image: 'bmp', 'dib', 'gif', 'ico', 'jpeg', 'jpeg2000', 'pcx', 'png', 'ppm', 'sgi', 'tga', 'tiff', 'webp', 'wmf', 'xbm' Audio: 'flac', 'mp3', 'wav' Video: 'mp4', 'mkv', 'avi' Dicom: 'dcm' Copy to clipboard Parameters: path (str) – Path to a supported file


deeplake¶

Example: @deeplake.compute def my_fn(sample_in: Any, samples_out, my_arg0, my_arg1=0): samples_out.my_tensor.append(my_arg0 * my_arg1) # This transform can be used using the eval method in one of these 2 ways:- # Directly evaluating the method # here arg0 and arg1 correspond to the 3rd and 4th argument in my_fn my_fn(arg0, arg1).eval(data_in, ds_out, scheduler='threaded', num_workers=5) # As a part of a Transform pipeline containing other functions pipeline = deeplake.compose([my_fn(a, b), another_function(x=2)]) pipeline.eval(data_in, ds_out, scheduler='processed', num_workers=2) Copy to clipboard The eval method evaluates the pipeline/transform function .  It has the following arguments: data_in: Input passed to the transform to generate output dataset.  It should support __getitem__ and __len__.  This can be a Deep Lake dataset.  ds_out (Dataset, optional): The dataset object to which the transform will get written.  If this is not provided, data_in will be overwritten if it is a Deep Lake dataset, otherwise error will be raised.  It should have all keys being generated in output already present as tensors.  It's initial state should be either: Empty i.e.  all tensors have no samples.  In this case all samples are added to the dataset.  All tensors are populated and have same length.  In this case new samples are appended to the dataset.  num_workers (int): The number of workers to use for performing the transform.  Defaults to 0.  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str): The scheduler to be used to compute the transformation.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool): Displays a progress bar if True (default).  skip_ok (bool): If True, skips the check for output tensors generated.  This allows the user to skip certain tensors in the function definition.  This is especially useful for inplace transformations in which certain tensors are not modified.  Defaults to False.  check_lengths (bool): If True, checks whether ds_out has tensors of same lengths initially.  pad_data_in (bool): If True, pads tensors of data_in to match the length of the largest tensor in data_in.  Defaults to False.  Note pad_data_in is only applicable if data_in is a Deep Lake dataset.  It raises the following errors: InvalidInputDataError: If data_in passed to transform is invalid.  It should support __getitem__ and __len__ operations.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as data_in will also raise this.  InvalidOutputDatasetError: If all the tensors of ds_out passed to transform don't have the same length.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as ds_out will also raise this.  TensorMismatchError: If one or more of the outputs generated during transform contain different tensors than the ones present in ds_out provided to transform.  UnsupportedSchedulerError: If the scheduler passed is not recognized.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'


deeplake¶

verify (bool) – If True, contents of the file are verified .  creds (optional, Dict) – Credentials for s3, gcp and http urls.  compression (optional, str) – Format of the file.  Only required if path does not have an extension.  storage (optional, StorageProvider) – Storage provider to use to retrieve remote files.  Useful if multiple files are being read from same storage to minimize overhead of creating a new provider.  Returns: Sample object.  Call sample.array to get the np.ndarray.  Return type: Sample Note No data is actually loaded until you try to get a property of the returned Sample.  This is useful for passing along to Tensor.append and Tensor.extend.  deeplake.link(path: str, creds_key: Optional[str] = None) → LinkedSample¶ Utility that stores a link to raw data.  Used to add data to a Deep Lake Dataset without copying it.  See Link htype.  Supported file types: Image: 'bmp', 'dib', 'gif', 'ico', 'jpeg', 'jpeg2000', 'pcx', 'png', 'ppm', 'sgi', 'tga', 'tiff', 'webp', 'wmf', 'xbm' Audio: 'flac', 'mp3', 'wav' Video: 'mp4', 'mkv', 'avi' Dicom: 'dcm' Copy to clipboard Parameters: path (str) – Path to a supported file.  creds_key (optional, str) – The credential key to use to read data for this sample.  The actual credentials are fetched from the dataset.  Returns: LinkedSample object that stores path and creds.  Return type: LinkedSample Examples >>> ds = deeplake.dataset('test/test_ds') >>> ds.create_tensor('images', htype='link[image]') >>> ds.images.append(deeplake.link('https://picsum.photos/200/300')) Copy to clipboard See more examples here.  deeplake.tiled(sample_shape: Tuple[int, ...], tile_shape: Optional[Tuple[int, ...]] = None, dtype: Union[str, dtype] = dtype('uint8'))¶ Allocates an empty sample of shape sample_shape, broken into tiles of shape tile_shape (except for edge tiles).  Example >>> with ds: ...  ds.create_tensor('image', htype='image', sample_compression='png') ...  ds.image.append(deeplake.tiled(sample_shape=(1003, 1103, 3), tile_shape=(10, 10, 3))) ...  ds.image[0][-217:, :212, 1:] = np.random.randint(0, 256, (217, 212, 2), dtype=np.uint8) Copy to clipboard Parameters: sample_shape (Tuple[int, ...]) – Full shape of the sample.  tile_shape (Optional, Tuple[int, ...]) – The sample will be will stored as tiles where each tile will have this shape (except edge tiles).  If not specified, it will be computed such that each tile is close to half of the tensor's max_chunk_size (after compression).  dtype (Union[str, np.dtype]) – Dtype for the sample array.  Default uint8.  Returns: A PartialSample instance which can be appended to a Tensor.  Return type: PartialSample deeplake.compute(fn, name: Optional[str] = None) → Callable[[...], ComputeFunction]¶ Compute is a decorator for functions.  The functions should have atleast 2 argument, the first two will correspond to sample_in and samples_out.  There can be as many other arguments as required.  The output should be appended/extended to the second argument in a deeplake like syntax.  Any value returned by the fn will be ignored


deeplake¶

TransformError: All other exceptions raised if there are problems while running the pipeline .  deeplake.compose(functions: List[ComputeFunction])¶ Takes a list of functions decorated using deeplake.compute() and creates a pipeline that can be evaluated using .eval Example: pipeline = deeplake.compose([my_fn(a=3), another_function(b=2)]) pipeline.eval(data_in, ds_out, scheduler='processed', num_workers=2) Copy to clipboard The eval method evaluates the pipeline/transform function.  It has the following arguments: data_in: Input passed to the transform to generate output dataset.  It should support __getitem__ and __len__.  This can be a Deep Lake dataset.  ds_out (Dataset, optional): The dataset object to which the transform will get written.  If this is not provided, data_in will be overwritten if it is a Deep Lake dataset, otherwise error will be raised.  It should have all keys being generated in output already present as tensors.  It's initial state should be either: Empty i.e.  all tensors have no samples.  In this case all samples are added to the dataset.  All tensors are populated and have same length.  In this case new samples are appended to the dataset.  num_workers (int): The number of workers to use for performing the transform.  Defaults to 0.  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str): The scheduler to be used to compute the transformation.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool): Displays a progress bar if True (default).  skip_ok (bool): If True, skips the check for output tensors generated.  This allows the user to skip certain tensors in the function definition.  This is especially useful for inplace transformations in which certain tensors are not modified.  Defaults to False.  It raises the following errors: InvalidInputDataError: If data_in passed to transform is invalid.  It should support __getitem__ and __len__ operations.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as data_in will also raise this.  InvalidOutputDatasetError: If all the tensors of ds_out passed to transform don't have the same length.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as ds_out will also raise this.  TensorMismatchError: If one or more of the outputs generated during transform contain different tensors than the ones present in ‘ds_out' provided to transform.  UnsupportedSchedulerError: If the scheduler passed is not recognized.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  TransformError: All other exceptions raised if there are problems while running the pipeline.


deeplake.core¶

deeplake.core.sample Sample deeplake.core.linked_sample LinkedSample deeplake.core.partial_sample PartialSample deeplake.core.storage Base Storage Provider LRU Cache S3 Storage Provider Google Cloud Storage Provider Google Drive Storage Provider Local Storage Provider Memory Provider deeplake.core.index IndexEntry Index merge_slices() slice_at_int() slice_length()