deeplake.api

deeplake.api.dataset dataset deeplake.api.info Info deeplake.api.link link() deeplake.api.read read() deeplake.api.tiled tiled() deeplake.api.link_tiled link_tiled()


deeplake.auto

deeplake.auto.structured deeplake.auto.structured.base deeplake.auto.structured.dataframe deeplake.auto.unstructured deeplake.auto.unstructured.base deeplake.auto.unstructured.image_classification deeplake.auto.unstructured.kaggle


deeplake.util

deeplake.util.shape_interval ShapeInterval deeplake.util.remove_cache remove_memory_cache() get_base_storage() get_dataset_with_zero_size_cache() create_read_copy_dataset() deeplake.util.notebook is_notebook() is_jupyter() is_colab() deeplake.util.exceptions ExternalCommandError KaggleError KaggleMissingCredentialsError KaggleDatasetAlreadyDownloadedError InvalidPathException AutoCompressionError InvalidFileExtension SamePathException TensorInvalidSampleShapeError TensorMetaMissingKey TensorDoesNotExistError TensorAlreadyExistsError TensorGroupDoesNotExistError TensorGroupAlreadyExistsError InvalidTensorNameError InvalidTensorGroupNameError DynamicTensorNumpyError InvalidShapeIntervalError InvalidKeyTypeError UnsupportedTensorTypeError InvalidBytesRequestedError ProviderListEmptyError DirectoryAtPathException FileAtPathException ProviderSizeListMismatch ModuleNotInstalledException LoginException UserNotLoggedInException InvalidHubPathException PathNotEmptyException AuthenticationException AuthorizationException InvalidPasswordException CouldNotCreateNewDatasetException ResourceNotFoundException BadRequestException OverLimitException ServerException BadGatewayException GatewayTimeoutException WaitTimeoutException LockedException UnexpectedStatusCodeException EmptyTokenException S3Error S3GetError S3SetError S3DeletionError S3ListError UnsupportedCompressionError SampleCompressionError SampleDecompressionError InvalidImageDimensions TensorUnsupportedSampleType MetaError MetaDoesNotExistError MetaAlreadyExistsError MetaInvalidKey MetaInvalidRequiredMetaKey TensorMetaInvalidHtype TensorMetaInvalidHtypeOverwriteValue TensorMetaMissingRequiredValue TensorMetaInvalidHtypeOverwriteKey TensorDtypeMismatchError InvalidTensorLinkError TensorMetaMutuallyExclusiveKeysError ReadOnlyModeError TransformError FilterError InvalidInputDataError UnsupportedSchedulerError TensorMismatchError InvalidOutputDatasetError InvalidTransformDataset HubComposeEmptyListError HubComposeIncompatibleFunction DatasetUnsupportedPytorch CorruptedMetaError ChunkEngineError FullChunkError ChunkIdEncoderError ChunkSizeTooSmallError DatasetHandlerError MemoryDatasetCanNotBePickledError CorruptedSampleError VersionControlError MergeError MergeNotSupportedError MergeMismatchError MergeConflictError CheckoutError CommitError EmptyCommitError TensorModifiedError GCSDefaultCredsNotFoundError InvalidOperationError AgreementError AgreementNotAcceptedError NotLoggedInError RenameError BufferError InfoError OutOfChunkCountError OutOfSampleCountError SampleHtypeMismatchError EmptyTensorError DatasetViewSavingError ManagedCredentialsNotFoundError UnableToReadFromUrlError InvalidTokenException TokenPermissionError


deeplake.client.log

Deep Lake does logging using the 'deeplake' logger. Logging level is logging.INFO by default. See example on how to change this. Copy to clipboard


deeplake.core.transform

Raises InvalidInputDataError – If data_in passed to transform is invalid .  It should support __getitem__ and __len__ operations.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as data_in will also raise this.  InvalidOutputDatasetError – If all the tensors of ds_out passed to transform don't have the same length.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as ds_out will also raise this.  TensorMismatchError – If one or more of the outputs generated during transform contain different tensors than the ones present in ‘ds_out' provided to transform.  UnsupportedSchedulerError – If the scheduler passed is not recognized.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  TransformError – All other exceptions raised if there are problems while running the pipeline.  ValueError – If num_workers > 0 and checkpoint_interval is not a multiple of num_workers or if checkpoint_interval > 0 and ds_out is None.  # noqa: DAR401 Example: @deeplake.compute def my_fn(sample_in: Any, samples_out, my_arg0, my_arg1=0): samples_out.my_tensor.append(my_arg0 * my_arg1) # This transform can be used using the eval method in one of these 2 ways:- # Directly evaluating the method # here arg0 and arg1 correspond to the 3rd and 4th argument in my_fn my_fn(arg0, arg1).eval(data_in, ds_out, scheduler='threaded', num_workers=5) # As a part of a Transform pipeline containing other functions pipeline = deeplake.compose([my_fn(a, b), another_function(x=2)]) pipeline.eval(data_in, ds_out, scheduler='processed', num_workers=2) Copy to clipboard Note pad_data_in is only applicable if data_in is a Deep Lake dataset.


deeplake.core.transform

class deeplake.core.transform.Pipeline(functions: List[ComputeFunction]) eval(data_in, ds_out: Optional[Dataset] = None, num_workers: int = 0, scheduler: str = 'threaded', progressbar: bool = True, skip_ok: bool = False, check_lengths: bool = True, pad_data_in: bool = False, read_only_ok: bool = False, cache_size: int = 16, checkpoint_interval: int = 0, ignore_errors: bool = False, verbose: bool = True, **kwargs) Evaluates the pipeline on data_in to produce an output dataset ds_out.  Parameters data_in – Input passed to the transform to generate output dataset.  Should support __getitem__ and __len__.  Can be a Deep Lake dataset.  ds_out (Dataset, optional) – The dataset object to which the transform will get written.  If this is not provided, data_in will be overwritten if it is a Deep Lake dataset, otherwise error will be raised.  It should have all keys being generated in output already present as tensors.  It's initial state should be either: Empty, i.e., all tensors have no samples.  In this case all samples are added to the dataset.  All tensors are populated and have same length.  In this case new samples are appended to the dataset.  num_workers (int) – The number of workers to use for performing the transform.  Defaults to 0.  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str) – The scheduler to be used to compute the transformation.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool) – Displays a progress bar if True (default).  skip_ok (bool) – If True, skips the check for output tensors generated.  This allows the user to skip certain tensors in the function definition.  This is especially useful for inplace transformations in which certain tensors are not modified.  Defaults to False.  check_lengths (bool) – If True, checks whether ds_out has tensors of same lengths initially.  pad_data_in (bool) – If True, pads tensors of data_in to match the length of the largest tensor in data_in.  Defaults to False.  read_only_ok (bool) – If True and output dataset is same as input dataset, the read-only check is skipped.  Defaults to False.  cache_size (int) – Cache size to be used by transform per worker.  checkpoint_interval (int) – If > 0, the transform will be checkpointed with a commit every checkpoint_interval input samples to avoid restarting full transform due to intermitten failures.  If the transform is interrupted, the intermediate data is deleted and the dataset is reset to the last commit.  If <= 0, no checkpointing is done.  Checkpoint interval should be a multiple of num_workers if num_workers > 0.  Defaults to 0.  ignore_errors (bool) – If True, input samples that causes transform to fail will be skipped and the errors will be ignored if possible.  verbose (bool) – If True, prints additional information about the transform.  **kwargs – Additional arguments


deeplake.core.vectorstore.deep_memory

VectorStore class deeplake.core.vectorstore.deep_memory.DeepMemory __init__(dataset: Dataset, client: DeepMemoryBackendClient, logger: Logger, embedding_function: Optional[Any] = None, token: Optional[str] = None, creds: Optional[Dict[str, Any]] = None) Based Deep Memory class to train and evaluate models on DeepMemory managed service.  Parameters dataset (Dataset) – deeplake dataset object.  client (DeepMemoryBackendClient) – Client to interact with the DeepMemory managed service.  Defaults to None.  logger (logging.Logger) – Logger object.  embedding_function (Optional[Any], optional) – Embedding funtion class used to convert queries/documents to embeddings.  Defaults to None.  token (Optional[str], optional) – API token for the DeepMemory managed service.  Defaults to None.  creds (Optional[Dict[str, Any]], optional) – Credentials to access the dataset.  Defaults to None.  Raises ImportError – if indra is not installed cancel(job_id: str) Cancel a training job on DeepMemory managed service.  Examples >>> cancelled: bool = vectorstore.deep_memory.cancel(job_id) Copy to clipboard Parameters job_id (str) – job_id of the training job.  Returns True if job was cancelled successfully, False otherwise.  Return type bool delete(job_id: str) Delete a training job on DeepMemory managed service.  Examples >>> deleted: bool = vectorstore.deep_memory.delete(job_id) Copy to clipboard Parameters job_id (str) – job_id of the training job.  Returns True if job was deleted successfully, False otherwise.  Return type bool evaluate(relevance: List[List[Tuple[str, int]]], queries: List[str], embedding_function: Optional[Callable[[...], List[ndarray]]] = None, embedding: Optional[Union[List[ndarray], List[List[float]]]] = None, top_k: List[int] = [1, 3, 5, 10, 50, 100], qvs_params: Optional[Dict[str, Any]] = None) → Dict[str, Dict[str, float]] Evaluate a model using the DeepMemory managed service.  Examples # 1.  Evaluate a model using an embedding function: relevance = [[('doc_id_1', 1), ('doc_id_2', 1)], [('doc_id_3', 1)]] queries = ['What is the capital of India?', 'What is the capital of France?'] embedding_function = openai_embedding.embed_documents vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding_function=embedding_function, ) # 2.  Evaluate a model with precomputed embeddings: embeddings = [[-1.2, 12, …], …] vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, ) # 3.  Evaluate a model with precomputed embeddings and log queries: vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, qvs_params={'log_queries': True}, ) # 4.  Evaluate with precomputed embeddings, log queries, and a custom branch: vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, qvs_params={ 'log_queries': True, 'branch': 'queries', } ) Parameters queries (List[str]) – Queries for model evaluation.  relevance (List[List[Tuple[str, int]]]) – Relevant documents and scores for each query.  - Outer list: matches the queries.  - Inner list: pairs of doc_id and relevance score.  - doc_id: Document ID from the corpus dataset, found in the id tensor.  - relevance_score: Between 0 (not relevant) and 1 (relevant).  embedding (Optional[np.ndarray], optional) – Query embeddings.  Defaults to None.  embedding_function (Optional[Callable[..., List[np.ndarray]]], optional) – Function to convert queries into embeddings.  Defaults to None.  top_k (List[int], optional) – Ranks for model evaluation.  Defaults to [1, 3, 5, 10, 50, 100].  qvs_params (Optional[Dict], optional) – Parameters to initialize the queries vectorstore.  When specified, creates a new vectorstore to track evaluation queries, the Deep Memory response, and the naive vector search results.  Defaults to None


VectorStore

Returns Recalls for each rank .  Return type Dict[str, Dict[str, float]] Raises ImportError – If indra is not installed.  ValueError – If no embedding_function is provided either during initialization or evaluation.  list_jobs(debug=False) List all training jobs on DeepMemory managed service.  status(job_id: str) Get the status of a training job on DeepMemory managed service.  Examples >>> vectorstore.deep_memory.status(job_id) -------------------------------------------------------------- | 6508464cd80cab681bfcfff3 | -------------------------------------------------------------- | status | pending | -------------------------------------------------------------- | progress | None | -------------------------------------------------------------- | results | not available yet | -------------------------------------------------------------- Copy to clipboard Parameters job_id (str) – job_id of the training job.  train(queries: List[str], relevance: List[List[Tuple[str, int]]], embedding_function: Optional[Callable[[str], ndarray]] = None, token: Optional[str] = None) → str Train a model on DeepMemory managed service.  Examples >>> queries: List[str] = ['What is the capital of India?', 'What is the capital of France?'] >>> relevance: List[List[Tuple[str, int]]] = [[('doc_id_1', 1), ('doc_id_2', 1)], [('doc_id_3', 1)]] >>> # doc_id_1, doc_id_2, doc_id_3 are the ids of the documents in the corpus dataset that is relevant to the queries.  It is stored in the `id` tensor of the corpus dataset.  >>> job_id: str = vectorstore.deep_memory.train(queries, relevance) Copy to clipboard Parameters queries (List[str]) – List of queries to train the model on.  relevance (List[List[Tuple[str, int]]]) – List of relevant documents for each query with their respective relevance score.  The outer list corresponds to the queries and the inner list corresponds to the doc_id, relevence_score pair for each query.  doc_id is the document id in the corpus dataset.  It is stored in the id tensor of the corpus dataset.  relevence_score is the relevance score of the document for the query.  The range is between 0 and 1, where 0 stands for not relevant and 1 stands for relevant.  embedding_function (Optional[Callable[[str], np.ndarray]], optional) – Embedding funtion used to convert queries to embeddings.  Defaults to None.  token (str, optional) – API token for the DeepMemory managed service.  Defaults to None.  Returns job_id of the training job.  Return type str Raises ValueError – if embedding_function is not specified either during initialization or during training.


deeplake.core.vectorstore.deep_memory


VectorStore

class deeplake.core.vectorstore.deep_memory.DeepMemory __init__(dataset: Dataset, client: DeepMemoryBackendClient, logger: Logger, embedding_function: Optional[Any] = None, token: Optional[str] = None, creds: Optional[Dict[str, Any]] = None) Based Deep Memory class to train and evaluate models on DeepMemory managed service.  Parameters dataset (Dataset) – deeplake dataset object.  client (DeepMemoryBackendClient) – Client to interact with the DeepMemory managed service.  Defaults to None.  logger (logging.Logger) – Logger object.  embedding_function (Optional[Any], optional) – Embedding funtion class used to convert queries/documents to embeddings.  Defaults to None.  token (Optional[str], optional) – API token for the DeepMemory managed service.  Defaults to None.  creds (Optional[Dict[str, Any]], optional) – Credentials to access the dataset.  Defaults to None.  Raises ImportError – if indra is not installed cancel(job_id: str) Cancel a training job on DeepMemory managed service.  Examples >>> cancelled: bool = vectorstore.deep_memory.cancel(job_id) Copy to clipboard Parameters job_id (str) – job_id of the training job.  Returns True if job was cancelled successfully, False otherwise.  Return type bool delete(job_id: str) Delete a training job on DeepMemory managed service.  Examples >>> deleted: bool = vectorstore.deep_memory.delete(job_id) Copy to clipboard Parameters job_id (str) – job_id of the training job.  Returns True if job was deleted successfully, False otherwise.  Return type bool evaluate(relevance: List[List[Tuple[str, int]]], queries: List[str], embedding_function: Optional[Callable[[...], List[ndarray]]] = None, embedding: Optional[Union[List[ndarray], List[List[float]]]] = None, top_k: List[int] = [1, 3, 5, 10, 50, 100], qvs_params: Optional[Dict[str, Any]] = None) → Dict[str, Dict[str, float]] Evaluate a model using the DeepMemory managed service.  Examples # 1.  Evaluate a model using an embedding function: relevance = [[('doc_id_1', 1), ('doc_id_2', 1)], [('doc_id_3', 1)]] queries = ['What is the capital of India?', 'What is the capital of France?'] embedding_function = openai_embedding.embed_documents vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding_function=embedding_function, ) # 2.  Evaluate a model with precomputed embeddings: embeddings = [[-1.2, 12, …], …] vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, ) # 3.  Evaluate a model with precomputed embeddings and log queries: vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, qvs_params={'log_queries': True}, ) # 4.  Evaluate with precomputed embeddings, log queries, and a custom branch: vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, qvs_params={ 'log_queries': True, 'branch': 'queries', } ) Parameters queries (List[str]) – Queries for model evaluation.  relevance (List[List[Tuple[str, int]]]) – Relevant documents and scores for each query.  - Outer list: matches the queries.  - Inner list: pairs of doc_id and relevance score.  - doc_id: Document ID from the corpus dataset, found in the id tensor.  - relevance_score: Between 0 (not relevant) and 1 (relevant).  embedding (Optional[np.ndarray], optional) – Query embeddings.  Defaults to None.  embedding_function (Optional[Callable[..., List[np.ndarray]]], optional) – Function to convert queries into embeddings.  Defaults to None.  top_k (List[int], optional) – Ranks for model evaluation.  Defaults to [1, 3, 5, 10, 50, 100].  qvs_params (Optional[Dict], optional) – Parameters to initialize the queries vectorstore.  When specified, creates a new vectorstore to track evaluation queries, the Deep Memory response, and the naive vector search results.  Defaults to None


deeplake.random.seed

class deeplake.core.seed.DeeplakeRandom get_seed() → Optional[int] Returns the seed which set to the deeplake to control the flows seed(seed: Optional[int] = None) Set random seed to the deeplake engines Parameters seed (int, optional) – Integer seed for initializing the computational engines, used for bringing reproducibility to random operations. Set to None to reset the seed. Defaults to None. Raises TypeError – If the provided value type is not supported. Background Specify a seed to train models and run randomized Deep Lake operations reproducibly. Features affected are: Dataloader shuffling Sampling and random operations in Tensor Query Language (TQL) Dataset.random_split The random seed can be specified using deeplake.random.seed: >>> import deeplake >>> deeplake.random.seed(0) Copy to clipboard Random number generators in other libraries The Deep Lake random seed does not affect random number generators in other libraries such as numpy. However, seeds in other libraries will affect code where Deep Lake uses those libraries, but it will not impact the methods above where Deep Lake uses its internal seed.


Background

Specify a seed to train models and run randomized Deep Lake operations reproducibly. Features affected are: Dataloader shuffling Sampling and random operations in Tensor Query Language (TQL) Dataset.random_split The random seed can be specified using deeplake.random.seed: Copy to clipboard


Random number generators in other libraries

The Deep Lake random seed does not affect random number generators in other libraries such as numpy.However, seeds in other libraries will affect code where Deep Lake uses those libraries, but it will not impact the methods above where Deep Lake uses its internal seed.


Index


_

__contains__() (deeplake.core.storage.GCSProvider method) __delitem__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) __getitem__() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) __getstate__() (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) __init__() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) (deeplake.core.sample.Sample method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.vectorstore.deep_memory.DeepMemory method), [1] (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) (deeplake.util.shape_interval.ShapeInterval method) __iter__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) __len__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) (deeplake.core.tensor.Tensor method) __repr__() (deeplake.core.index.Index method) __setitem__() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) (deeplake.core.tensor.Tensor method) __setstate__() (deeplake.core.storage.LRUCache method) __str__() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) __weakref__ (deeplake.core.index.Index attribute) (deeplake.core.index.IndexEntry attribute) (deeplake.core.storage.StorageProvider attribute) _all_keys() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) _check_compatibility_with_htype() (deeplake.core.tensor.Tensor method) _check_is_file() (deeplake.core.storage.LocalProvider method) _check_update_creds() (deeplake.core.storage.S3Provider method) _config (deeplake.core.tensor.Tensor property) _flush_if_not_read_only() (deeplake.core.storage.LRUCache method) _forward() (deeplake.core.storage.LRUCache method) _forward_value() (deeplake.core.storage.LRUCache method) _free_up_space() (deeplake.core.storage.LRUCache method) _insert_in_cache() (deeplake.core.storage.LRUCache method) _is_hub_path (deeplake.core.storage.StorageProvider attribute) _linked_sample() (deeplake.core.tensor.Tensor method) _pop_from_cache() (deeplake.core.storage.LRUCache method) _set_hub_creds_info() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.S3Provider method) _state_keys() (deeplake.core.storage.S3Provider method)


B

BadGatewayException (class in deeplake.util.exceptions) BadRequestException (class in deeplake.util.exceptions) base_htype (deeplake.core.tensor.Tensor property) batch() (deeplake.enterprise.DeepLakeDataLoader method) branch (deeplake.core.dataset.Dataset property) branches (deeplake.core.dataset.Dataset property) BufferError (class in deeplake.util.exceptions)


D

data() (deeplake.core.tensor.Tensor method) DataFrame (class in deeplake.auto.structured.dataframe) dataloader() (deeplake.core.dataset.Dataset method) dataset (class in deeplake.api.dataset) Dataset (class in deeplake.core.dataset) dataset() (in module deeplake) DatasetHandlerError (class in deeplake.util.exceptions) DatasetUnsupportedPytorch (class in deeplake.util.exceptions) DatasetViewSavingError (class in deeplake.util.exceptions) deepcopy() (deeplake.api.dataset.dataset static method) (in module deeplake) deeplake module deeplake.api.info module deeplake.integrations.wandb.wandb module DeepLakeCloudDataset (class in deeplake.core.dataset) DeepLakeDataLoader (class in deeplake.enterprise) DeeplakeRandom (class in deeplake.core.seed) DeepMemory (class in deeplake.core.vectorstore.deep_memory), [1] delete() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (deeplake.core.dataset.ViewEntry method) (deeplake.core.vectorstore.deep_memory.DeepMemory method), [1] (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) (in module deeplake) delete_branch() (deeplake.core.dataset.Dataset method) delete_by_path() (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore static method) delete_group() (deeplake.core.dataset.Dataset method) delete_tensor() (deeplake.core.dataset.Dataset method) delete_view() (deeplake.core.dataset.Dataset method) dict() (deeplake.core.tensor.Tensor method) diff() (deeplake.core.dataset.Dataset method) DirectoryAtPathException (class in deeplake.util.exceptions) disable_readonly() (deeplake.core.storage.StorageProvider method) download_kaggle_dataset() (in module deeplake.auto.unstructured.kaggle) downsample() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) dtype (deeplake.core.tensor.Tensor property) DynamicTensorNumpyError (class in deeplake.util.exceptions)


C

cancel() (deeplake.core.vectorstore.deep_memory.DeepMemory method), [1] check_readonly() (deeplake.core.storage.StorageProvider method) checkout() (deeplake.core.dataset.Dataset method) (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) CheckoutError (class in deeplake.util.exceptions) ChunkEngineError (class in deeplake.util.exceptions) ChunkIdEncoderError (class in deeplake.util.exceptions) ChunkSizeTooSmallError (class in deeplake.util.exceptions) clear() (deeplake.api.info.Info method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.GDriveProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.MemoryProvider method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) (deeplake.core.tensor.Tensor method) clear_cache() (deeplake.core.dataset.Dataset method) (deeplake.core.storage.LRUCache method) clear_deeplake_objects() (deeplake.core.storage.LRUCache method) client (deeplake.core.dataset.Dataset property) (deeplake.core.dataset.DeepLakeCloudDataset property) close() (deeplake.enterprise.DeepLakeDataLoader method) commit() (deeplake.core.dataset.Dataset method) (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) commit_id (deeplake.core.dataset.Dataset property) CommitError (class in deeplake.util.exceptions) commits (deeplake.core.dataset.Dataset property) compose() (in module deeplake) compose_at() (deeplake.core.index.Index method) compressed_bytes() (deeplake.core.sample.Sample method) compute() (in module deeplake) connect() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (in module deeplake) copy() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.storage.StorageProvider method) (in module deeplake) CorruptedMetaError (class in deeplake.util.exceptions) CorruptedSampleError (class in deeplake.util.exceptions) CouldNotCreateNewDatasetException (class in deeplake.util.exceptions) create_group() (deeplake.core.dataset.Dataset method) create_read_copy_dataset() (in module deeplake.util.remove_cache) create_tensor() (deeplake.core.dataset.Dataset method) create_tensor_like() (deeplake.core.dataset.Dataset method) creds_key() (deeplake.core.tensor.Tensor method)


E

empty() (deeplake.api.dataset.dataset static method) (in module deeplake) EmptyCommitError (class in deeplake.util.exceptions) EmptyTensorError (class in deeplake.util.exceptions) EmptyTokenException (class in deeplake.util.exceptions) enable_readonly() (deeplake.core.storage.StorageProvider method) eval() (deeplake.core.transform.Pipeline method) evaluate() (deeplake.core.vectorstore.deep_memory.DeepMemory method), [1] exists() (deeplake.api.dataset.dataset static method) (in module deeplake) extend() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) ExternalCommandError (class in deeplake.util.exceptions)


G

GatewayTimeoutException (class in deeplake.util.exceptions) GCSDefaultCredsNotFoundError (class in deeplake.util.exceptions) GCSProvider (class in deeplake.core.storage) GDriveProvider (class in deeplake.core.storage) get() (deeplake.api.info.Info method) get_base_storage() (in module deeplake.util.remove_cache) get_bytes() (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.S3Provider method) (deeplake.core.storage.StorageProvider method) get_commit_details() (deeplake.core.dataset.Dataset method) get_creds_keys() (deeplake.core.dataset.Dataset method) get_dataset_with_zero_size_cache() (in module deeplake.util.remove_cache) get_deeplake_object() (deeplake.core.storage.LRUCache method) get_managed_creds_keys() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) get_seed() (deeplake.core.seed.DeeplakeRandom method) get_view() (deeplake.core.dataset.Dataset method) get_views() (deeplake.core.dataset.Dataset method) groups (deeplake.core.dataset.Dataset property)


F

FileAtPathException (class in deeplake.util.exceptions) fill_dataset() (deeplake.auto.structured.dataframe.DataFrame method) filter() (deeplake.core.dataset.Dataset method) FilterError (class in deeplake.util.exceptions) find_axis() (deeplake.core.index.Index method) fix_vc() (deeplake.core.dataset.Dataset method) flush() (deeplake.core.dataset.Dataset method) (deeplake.core.storage.LRUCache method) (deeplake.core.storage.StorageProvider method) FullChunkError (class in deeplake.util.exceptions)


I

id (deeplake.core.dataset.ViewEntry property) ImageClassification (class in deeplake.auto.unstructured.image_classification) Index (class in deeplake.core.index) IndexEntry (class in deeplake.core.index) indices() (deeplake.core.index.IndexEntry method) Info (class in deeplake.api.info) info (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) InfoError (class in deeplake.util.exceptions) ingest_classification() (deeplake.api.dataset.dataset static method) (in module deeplake) ingest_coco() (deeplake.api.dataset.dataset static method) (in module deeplake) ingest_dataframe() (deeplake.api.dataset.dataset static method) (in module deeplake) ingest_huggingface() (in module deeplake) ingest_kaggle() (deeplake.api.dataset.dataset static method) (in module deeplake) ingest_yolo() (in module deeplake) invalidate_libdeeplake_dataset() (deeplake.core.tensor.Tensor method) InvalidBytesRequestedError (class in deeplake.util.exceptions) InvalidFileExtension (class in deeplake.util.exceptions) InvalidHubPathException (class in deeplake.util.exceptions) InvalidImageDimensions (class in deeplake.util.exceptions) InvalidInputDataError (class in deeplake.util.exceptions) InvalidKeyTypeError (class in deeplake.util.exceptions) InvalidOperationError (class in deeplake.util.exceptions) InvalidOutputDatasetError (class in deeplake.util.exceptions) InvalidPasswordException (class in deeplake.util.exceptions) InvalidPathException (class in deeplake.util.exceptions) InvalidShapeIntervalError (class in deeplake.util.exceptions) InvalidTensorGroupNameError (class in deeplake.util.exceptions) InvalidTensorLinkError (class in deeplake.util.exceptions) InvalidTensorNameError (class in deeplake.util.exceptions) InvalidTokenException (class in deeplake.util.exceptions) InvalidTransformDataset (class in deeplake.util.exceptions) is_actually_cloud (deeplake.core.dataset.DeepLakeCloudDataset property) is_colab() (in module deeplake.util.notebook) is_dynamic (deeplake.core.tensor.Tensor property) is_head_node (deeplake.core.dataset.Dataset property) is_jupyter() (in module deeplake.util.notebook) is_link (deeplake.core.tensor.Tensor property) is_notebook() (in module deeplake.util.notebook) is_sequence (deeplake.core.tensor.Tensor property) is_trivial() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) is_view (deeplake.core.dataset.Dataset property) items() (deeplake.api.info.Info method)


K

KaggleDatasetAlreadyDownloadedError (class in deeplake.util.exceptions) KaggleError (class in deeplake.util.exceptions) KaggleMissingCredentialsError (class in deeplake.util.exceptions) keys() (deeplake.api.info.Info method)


H

has_head_changes (deeplake.core.dataset.Dataset property) hidden (deeplake.core.tensor.Tensor property) htype (deeplake.core.tensor.Tensor property) HubComposeEmptyListError (class in deeplake.util.exceptions) HubComposeIncompatibleFunction (class in deeplake.util.exceptions)


A

add() (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) add_creds_key() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) AgreementError (class in deeplake.util.exceptions) AgreementNotAcceptedError (class in deeplake.util.exceptions) append() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) apply() (deeplake.core.index.Index method) apply_squeeze() (deeplake.core.index.Index method) array (deeplake.core.sample.Sample property) AuthenticationException (class in deeplake.util.exceptions) AuthorizationException (class in deeplake.util.exceptions) AutoCompressionError (class in deeplake.util.exceptions)


M

ManagedCredentialsNotFoundError (class in deeplake.util.exceptions) max_len (deeplake.core.dataset.Dataset property) max_view (deeplake.core.dataset.Dataset property) maybe_flush() (deeplake.core.storage.StorageProvider method) MemoryDatasetCanNotBePickledError (class in deeplake.util.exceptions) MemoryProvider (class in deeplake.core.storage) merge() (deeplake.core.dataset.Dataset method) merge_slices() (in module deeplake.core.index) MergeConflictError (class in deeplake.util.exceptions) MergeError (class in deeplake.util.exceptions) MergeMismatchError (class in deeplake.util.exceptions) MergeNotSupportedError (class in deeplake.util.exceptions) message (deeplake.core.dataset.ViewEntry property) meta (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) MetaAlreadyExistsError (class in deeplake.util.exceptions) MetaDoesNotExistError (class in deeplake.util.exceptions) MetaError (class in deeplake.util.exceptions) MetaInvalidKey (class in deeplake.util.exceptions) MetaInvalidRequiredMetaKey (class in deeplake.util.exceptions) min_len (deeplake.core.dataset.Dataset property) min_view (deeplake.core.dataset.Dataset property) modified_samples() (deeplake.core.tensor.Tensor method) module deeplake deeplake.api.info deeplake.integrations.wandb.wandb ModuleNotInstalledException (class in deeplake.util.exceptions)


O

offset() (deeplake.enterprise.DeepLakeDataLoader method) optimize() (deeplake.core.dataset.ViewEntry method) OutOfChunkCountError (class in deeplake.util.exceptions) OutOfSampleCountError (class in deeplake.util.exceptions) OverLimitException (class in deeplake.util.exceptions)


N

nbytes (deeplake.api.info.Info property) ndim (deeplake.core.tensor.Tensor property) need_to_reload_creds() (deeplake.core.storage.S3Provider method) no_view_dataset (deeplake.core.dataset.Dataset property) NotLoggedInError (class in deeplake.util.exceptions) num_samples (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) numpy() (deeplake.core.tensor.Tensor method) (deeplake.enterprise.DeepLakeDataLoader method)


L

length() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) like() (deeplake.api.dataset.dataset static method) (in module deeplake) link() (in module deeplake) (in module deeplake.api.link) link_tiled() (in module deeplake) (in module deeplake.api.link_tiled) LinkedSample (class in deeplake.core.linked_sample) LinkedTiledSample (class in deeplake.core.linked_tiled_sample) list() (deeplake.api.dataset.dataset static method) (deeplake.core.tensor.Tensor method) list_jobs() (deeplake.core.vectorstore.deep_memory.DeepMemory method), [1] load() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.ViewEntry method) (in module deeplake) load_view() (deeplake.core.dataset.Dataset method) LocalProvider (class in deeplake.core.storage) LockedException (class in deeplake.util.exceptions) log() (deeplake.core.dataset.Dataset method) LoginException (class in deeplake.util.exceptions) LRUCache (class in deeplake.core.storage)


R

random_split() (deeplake.core.dataset.Dataset method) read() (in module deeplake) (in module deeplake.api.read) read_only (deeplake.core.dataset.Dataset property) ReadOnlyModeError (class in deeplake.util.exceptions) rechunk() (deeplake.core.dataset.Dataset method) register_deeplake_object() (deeplake.core.storage.LRUCache method) remove_deeplake_object() (deeplake.core.storage.LRUCache method) remove_memory_cache() (in module deeplake.util.remove_cache) rename() (deeplake.api.dataset.dataset static method) (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) (deeplake.core.storage.GCSProvider method) (deeplake.core.storage.LocalProvider method) (deeplake.core.storage.S3Provider method) (in module deeplake) rename_group() (deeplake.core.dataset.Dataset method) rename_tensor() (deeplake.core.dataset.Dataset method) RenameError (class in deeplake.util.exceptions) replace_with() (deeplake.api.info.Info method) reset() (deeplake.core.dataset.Dataset method) ResourceNotFoundException (class in deeplake.util.exceptions) root (deeplake.core.dataset.Dataset property)


Q

query() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method)


P

parent (deeplake.core.dataset.Dataset property) PartialSample (class in deeplake.core.partial_sample) path() (deeplake.core.tensor.Tensor method) PathNotEmptyException (class in deeplake.util.exceptions) pending_commit_id (deeplake.core.dataset.Dataset property) pil (deeplake.core.sample.Sample property) Pipeline (class in deeplake.core.transform) play() (deeplake.core.tensor.Tensor method) pop() (deeplake.api.info.Info method) (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) popitem() (deeplake.api.info.Info method) populate_creds() (deeplake.core.dataset.Dataset method) ProviderListEmptyError (class in deeplake.util.exceptions) ProviderSizeListMismatch (class in deeplake.util.exceptions) pytorch() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method)


S

S3DeletionError (class in deeplake.util.exceptions) S3Error (class in deeplake.util.exceptions) S3GetError (class in deeplake.util.exceptions) S3ListError (class in deeplake.util.exceptions) S3Provider (class in deeplake.core.storage) S3SetError (class in deeplake.util.exceptions) SamePathException (class in deeplake.util.exceptions) Sample (class in deeplake.core.sample) sample_by() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method) sample_indices (deeplake.core.dataset.Dataset property) (deeplake.core.tensor.Tensor property) sample_info (deeplake.core.tensor.Tensor property) SampleCompressionError (class in deeplake.util.exceptions) SampleDecompressionError (class in deeplake.util.exceptions) SampleHtypeMismatchError (class in deeplake.util.exceptions) save_view() (deeplake.core.dataset.Dataset method) search() (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) seed() (deeplake.core.seed.DeeplakeRandom method) ServerException (class in deeplake.util.exceptions) set_bytes() (deeplake.core.storage.StorageProvider method) set_token() (deeplake.core.dataset.Dataset method) setdefault() (deeplake.api.info.Info method) shape (deeplake.core.tensor.Tensor property) shape_interval (deeplake.core.tensor.Tensor property) ShapeInterval (class in deeplake.util.shape_interval) shapes() (deeplake.core.tensor.Tensor method) shuffle() (deeplake.enterprise.DeepLakeDataLoader method) size_approx() (deeplake.core.dataset.Dataset method) slice_at_int() (in module deeplake.core.index) slice_length() (in module deeplake.core.index) status() (deeplake.core.vectorstore.deep_memory.DeepMemory method), [1] StorageProvider (class in deeplake.core.storage) structure() (deeplake.auto.unstructured.image_classification.ImageClassification method) StructuredDataset (class in deeplake.auto.structured.base) subscriptable() (deeplake.core.index.IndexEntry method) summary() (deeplake.core.dataset.Dataset method) (deeplake.core.tensor.Tensor method) (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) sync() (deeplake.core.storage.GDriveProvider method)


U

UnableToReadFromUrlError (class in deeplake.util.exceptions) uncompressed_bytes() (deeplake.core.sample.Sample method) UnexpectedStatusCodeException (class in deeplake.util.exceptions) UnstructuredDataset (class in deeplake.auto.unstructured.base) UnsupportedCompressionError (class in deeplake.util.exceptions) UnsupportedSchedulerError (class in deeplake.util.exceptions) UnsupportedTensorTypeError (class in deeplake.util.exceptions) update() (deeplake.api.info.Info method) (deeplake.core.dataset.Dataset method) update_creds_key() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method) update_embedding() (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) UserNotLoggedInException (class in deeplake.util.exceptions)


T

Tensor (class in deeplake.core.tensor) TensorAlreadyExistsError (class in deeplake.util.exceptions) TensorDoesNotExistError (class in deeplake.util.exceptions) TensorDtypeMismatchError (class in deeplake.util.exceptions) tensorflow() (deeplake.core.dataset.Dataset method) (deeplake.enterprise.DeepLakeDataLoader method) TensorGroupAlreadyExistsError (class in deeplake.util.exceptions) TensorGroupDoesNotExistError (class in deeplake.util.exceptions) TensorInvalidSampleShapeError (class in deeplake.util.exceptions) TensorMetaInvalidHtype (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteKey (class in deeplake.util.exceptions) TensorMetaInvalidHtypeOverwriteValue (class in deeplake.util.exceptions) TensorMetaMissingKey (class in deeplake.util.exceptions) TensorMetaMissingRequiredValue (class in deeplake.util.exceptions) TensorMetaMutuallyExclusiveKeysError (class in deeplake.util.exceptions) TensorMismatchError (class in deeplake.util.exceptions) TensorModifiedError (class in deeplake.util.exceptions) tensors (deeplake.core.dataset.Dataset property) tensors() (deeplake.core.vectorstore.deeplake_vectorstore.VectorStore method) TensorUnsupportedSampleType (class in deeplake.util.exceptions) text() (deeplake.core.tensor.Tensor method) tiled() (in module deeplake) (in module deeplake.api.tiled) timestamps (deeplake.core.tensor.Tensor property) tobytes() (deeplake.core.tensor.Tensor method) token (deeplake.core.dataset.Dataset property) (deeplake.core.dataset.DeepLakeCloudDataset property) TokenPermissionError (class in deeplake.util.exceptions) train() (deeplake.core.vectorstore.deep_memory.DeepMemory method), [1] transform() (deeplake.enterprise.DeepLakeDataLoader method) TransformError (class in deeplake.util.exceptions)


V

validate() (deeplake.core.index.Index method) (deeplake.core.index.IndexEntry method) values() (deeplake.api.info.Info method) VectorStore (class in deeplake.core.vectorstore.deeplake_vectorstore) verify (deeplake.core.tensor.Tensor property) VersionControlError (class in deeplake.util.exceptions) ViewEntry (class in deeplake.core.dataset) visualize() (deeplake.core.dataset.Dataset method) (deeplake.core.dataset.DeepLakeCloudDataset method)


W

WaitTimeoutException (class in deeplake.util.exceptions)


Python Module Index

d d deeplake deeplake.api.info deeplake.integrations.wandb.wandb


Deep Lake API Reference¶

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Point Cloud Htype Mesh Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Enterprise Features Dataloader Sampler Tensor Query Language Random Split API Reference deeplake deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log deeplake.core.transform


Indices and tables¶

Index Module Index Search Page


Deep Lake API Reference¶

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Point Cloud Htype Mesh Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Enterprise Features Dataloader Sampler Tensor Query Language API Reference deeplake deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log deeplake.core.transform


Deep Lake API Reference¶

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Point Cloud Htype Mesh Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Experimental API Dataloader Tensor Query Language API Reference deeplake deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log


Deep Lake API Reference¶

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Experimental API Dataloader Tensor Query Language API Reference deeplake deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log


Indices and tables

Index Module Index Search Page


Hub API Reference

Hub is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Sequence htype Link htype Sample Compressions PyTorch and Tensorflow Support Utility Functions General Functions Making Hub Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Experimental API Dataloader & Query API Reference hub hub.core hub.core.dataset hub.core.tensor hub.api hub.auto hub.util hub.client.log hub.experimental hub.experimental.dataloader


Activeloop Storage and Managed Datasets

In order to access datasets stored in Deep Lake, or datasets in other clouds that are managed by Activeloop, users must register and authenticate using the steps in the link below:User Authentication


AWS S3

Authentication with AWS S3 has 4 options:1.Use Deep Lake on a machine in the AWS ecosystem that has access to the relevant S3 bucket via AWS IAM, in which case there is no need to pass credentials in order to access datasets in that bucket.2.Configure AWS through the cli using aws configure. This creates a credentials file on your machine that is automatically access by Deep Lake during authentication.3.Save the AWS_ACCESS_KEY_ID ,AWS_SECRET_ACCESS_KEY , and AWS_SESSION_TOKEN (optional) in environmental variables of the same name, which are loaded as default credentials if no other credentials are specified.4.Create a dictionary with the AWS_ACCESS_KEY_ID ,AWS_SECRET_ACCESS_KEY , and AWS_SESSION_TOKEN (optional), and pass it to Deep Lake using:Note: the dictionary keys must be lowercase!# Vector Store APIvector_store = VectorStore('s3://<bucket_name>/<dataset_name>', creds = { 'aws_access_key_id': <your_access_key_id>, 'aws_secret_access_key': <your_aws_secret_access_key>, 'aws_session_token': <your_aws_session_token>, # Optional } )# Low Level APIds = deeplake.load('s3://<bucket_name>/<dataset_name>', creds = { 'aws_access_key_id': <your_access_key_id>, 'aws_secret_access_key': <your_aws_secret_access_key>, 'aws_session_token': <your_aws_session_token>, # Optional } ) can be used for connecting to other object storages supporting S3-like API such as MinIO, StorageGrid and others.


Custom Storage with S3 API

In order to connect to other object storages supporting S3-like API such as MinIO, StorageGrid and others, simply add endpoint_url the the creds dictionary.# Vector Store APIvector_store = VectorStore('s3://...', creds = { 'aws_access_key_id': <your_access_key_id>, 'aws_secret_access_key': <your_aws_secret_access_key>, 'aws_session_token': <your_aws_session_token>, # Optional 'endpoint_url': 'http://localhost:8888' } )# Low Level APIds = deeplake.load('s3://...', creds = { 'aws_access_key_id': <your_access_key_id>, 'aws_secret_access_key': <your_aws_secret_access_key>, 'aws_session_token': <your_aws_session_token>, # Optional 'endpoint_url': 'http://localhost:8888' } )


Microsoft Azure

Authentication with Microsoft Azure has 4 options:1.Log in from your machine's CLI using az login.2.Save the AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY , or other credentials in environmental variables of the same name, which are loaded as default credentials if no other credentials are specified.3.Create a dictionary with the ACCOUNT_KEY or SAS_TOKEN and pass it to Deep Lake using:Note: the dictionary keys must be lowercase!# Vector Store APIvector_store = VectorStore('azure://<account_name>/<container_name>/<dataset_name>', creds = { 'account_key': <your_account_key>, 'sas_token': <your_sas_token>, } )# Low Level APIds = deeplake.load('azure://<account_name>/<container_name>/<dataset_name>', creds = { 'account_key': <your_account_key>, #OR 'sas_token': <your_sas_token>, } )


Google Cloud Storage

Authentication with Google Cloud Storage has 2 options:1.Create a service account, download the JSON file containing the keys, and then pass that file to the creds parameter in deeplake.load('gcs://.....', creds = 'path_to_keys.json') . It is also possible to manually pass the information from the JSON file into the creds parameter using: # Vector Store APIvector_store = VectorStore('gcs://.....', creds = {<information from the JSON file>} )# Low Level APIds = deeplake.load('gcs://.....', creds = {<information from the JSON file>} )2.Authenticate through the browser using the steps below. This requires that the project credentials are stored on your machine, which happens after gcloud is initialized and logged in through the CLI. Afterwards, creds can be switched to creds = 'cache'.# Vector Store APIvector_store = VectorStore('gcs://.....', creds = 'browser' # Switch to 'cache' after doing this once )# Low Level APIds = deeplake.load('gcs://.....', creds = 'browser' # Switch to 'cache' after doing this once )


How to Customize Deep Lake Vector Stores for Images, Multi-Embedding Applications, and More.

Under-the-hood, Deep Lake vector stores use the Deep Lake tabular format, where Tensors are conceptually equivalent to columns. A unique feature in Deep Lake is that Tensors can be customized to a variety of use-cases beyond simple embeddings of text.


Creating vector stores with non-text data

To create a Vector Store for images, we should write a custom embedding function that embeds images from a file using a neural network, since we cannot use OpenAI for embedding images yet.import osimport torchfrom torchvision import transforms, modelsfrom torchvision.models.feature_extraction import create_feature_extractorfrom PIL import Imagemodel = models.resnet18(pretrained=True)return_nodes = { 'avgpool': 'embedding'}model = create_feature_extractor(model, return_nodes=return_nodes)model.eval()model.to('cpu')tform = transforms.Compose([ transforms.Resize((224,224)), transforms.ToTensor(), transforms.Lambda(lambda x: torch.cat([x, x, x], dim=0) if x.shape[0] == 1 else x), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),])def embedding_function(images, model = model, transform = tform, batch_size = 4): '''Creates a list of embeddings based on a list of image filenames. Images are processed in batches.''' if isinstance(images, str): images = [images] #Proceess the embeddings in batches, but return everything as a single list embeddings = [] for i in range(0, len(images), batch_size): batch = torch.stack([transform(Image.open(item)) for item in images[i:i+batch_size]]) batch = batch.to('cpu') with torch.no_grad(): embeddings+= model(batch)['embedding'][:,:,0,0].cpu().numpy().tolist() return embeddingsLets download and unzip 6 example images with common objects and create a list of containing their filenames.data_folder = '/Users/istranic/ActiveloopCode/Datasets/common_objects'image_fns = [os.path.join(data_folder, file) for file in os.listdir(data_folder) if os.path.splitext(file)[-1]=='.jpg']Earlier in this tutorial, we did not specify any data-structure-related information when initializing the Vector Store, which by default creates a vector store with tensors for text, metadata, id (auto-populated), and embedding. Here, we create a Vector Store for image similarity search, which should contains tensors for the image, its embedding, and the filename for the image. This can be achieved by specifying custom tensor_params.vector_store_path = '/vector_store_getting_started_images'vector_store = VectorStore( path = vector_store_path, tensor_params = [{'name': 'image', 'htype': 'image', 'sample_compression': 'jpg'}, {'name': 'embedding', 'htype': 'embedding'}, {'name': 'filename', 'htype': 'text'}],)We add data to the Vector Store just as if we were adding text data earlier in the Getting Started Guide.vector_store.add(image = image_fns, filename = image_fns, embedding_function = embedding_function, embedding_data = image_fns)


Performing image similarity search

Let's find the image in the Vector Store that is most similar to the reference image below.image_path = '/reference_image.jpg'result = vector_store.search(embedding_data = [image_path], embedding_function = embedding_function)We can display the result of the most similar image, which shows a picture of a yellow Lamborghini, which is fairly similar to the black Porsche above. Image.fromarray(result['image'][0])


Creating Vector Stores with multiple embeddings

COMING SOON


APIs for Search

Vector search can occur via a variety of APIs in Deep Lake. They are explained in the links below:Deep Lake Vector Store APIREST APILangChain API


Overview of Vector Search Options in Deep Lake

Deep Lake offers a variety of vector search options depending on the Storage Location of the Vector Store and infrastructure that should run the computations.In memory or localClient-sideDeep Lake OSS Python CodeUser cloud (must be connected to Deep Lake)Client-sideDeep Lake C++ Compute EngineDeep Lake StorageClient-sideDeep Lake C++ Compute EngineDeep Lake Managed Tensor DatabaseManaged DatabaseDeep Lake C++ Compute Engine


Overview of Options for Search Computation Execution

The optimal option for search execution is automatically selected based on the Vector Stores storage location. The different computation options are explained below.


Python (Client-Side)

Deep Lake OSS offers query execution logic that run on the client (your machine) using OSS code in Python. This compute logic is accessible in all Deep Lake Python APIs and is available for Vector Stores stored in any location. See individual APIs below for details.


Managed Tensor Database (Server-Side Running Compute Engine)

Deep Lake offers a Managed Tensor Database that executes queries on Deep Lake infrastructure while running Compute Engine under-the-hood. This compute logic is accessible in all Deep Lake Python APIs and is only available for Vector Stores stored in the Deep Lake Managed Tensor Database. See individual APIs below for details.


Compute Engine (Client-Side)

Deep Lake Compute Engine offers query execution logic that run on the client (your machine) using C++ Code that is called via Python API. This compute logic is accessible in all Deep Lake Python APIs and is only available for Vector Stores stored Deep Lake storage or in user clouds connected to Deep Lake. See individual APIs below for details. To run queries using Compute Engine, make sure to !pip install 'deeplake[enterprise]'.


How to Register and Authenticate in Deep Lake

In order to use Deep Lake features that require authentication (Activeloop storage, connecting your cloud dataset to the Deep Lake UI, etc.) you should register and login with Deep Lake.


Registration

You can register in the Deep Lake App, or in the CLI using:


Authentication in Programmatic Interfaces

After registering, you can create an API token in the Deep Lake UI (top-right corner, user settings) and authenticate in programatic interfaces using 3 options:


Environmental Variable

Set the environmental variable ACTIVELOOP_TOKEN to your API token. In Python, this can be done using:


CLI Login

Login in the CLI using two options:activeloop login -u <username> -p <password>activeloop login -t <your_token>Credentials created using the CLI login !activeloop login expire after 1000 hrs. Credentials created using API tokens in the Deep Lake App expire after the time specified for the individual token. Therefore, long-term workflows should be run using API tokens in order to avoid expiration of credentials mid-workflow.


Pass the Token to Individual Methods

You can pass your API token to individual methods that require authentication such as:


How to Write Data Concurrently to Deep Lake Datasets

Deep Lake offers 3 solutions for concurrently writing data, depending on the required scale of the application. Concurrency is not native to the Deep Lake format, so these solutions use locks and queues to schedule and linearize the write operations to Deep Lake.


Concurrency Using External Locks

Concurrent writes can be supported using an in-memory database that serves as the locking mechanism for Deep Lake datasets. Tools such as Zookeper or Redis are highly performant and reliable and can be deployed using a few lines of code. External locks are recommended for small-to-medium workloads.Concurrency Using Zookeeper Locks


Managed Concurrency

COMING SOON. Deep Lake will offer a Managed Tensor Database that supports read (search) and write operations at scale. Deep Lake ensures the operations are performant by provisioning the necessary infrastructure and executing the underlying user requests in a distributed manner. This approach is recommended for production applications that require a separate service to handle the high computational loads of vector search.


Default Behavior

By default, Deep Lake datasets are loaded in write mode and a lock file is created. This can be avoided by specifying read_only = True to APIs that load datasets. An error will occur if the Deep Lake dataset is locked and the user tries to open it in write mode. To specify a waiting time for the lock to be released, you can specify lock_timeout = <timeout_in_s> to APIs that load datasets.


Concurrency Using Deep Lake Locks

Deep Lake datasets internally support file-based locks. File-base locks are generally slower and less reliable that the other listed solutions, and they should only be used for prototyping.


Manipulating Locks

Locks can manually be set or released using:from deeplake.core.lock import lock_dataset, unlock_datasetunlock_dataset(<dataset_path>)lock_dataset(<dataset_path>)


How to Create Datasets in Deep Lake Format

This guide creates Deep Lake datasets locally. You may create datasets in the Activeloop cloud by registering, creating an API token, and replacing the local paths below with the path to your Deep Lake organization hub://organization_name/dataset_nameYou don't have to worry about uploading datasets after you've created them. They are automatically synchronized with wherever they are being stored.


Manual Creation

Let's follow along with the example below to create our first dataset manually. First, download and unzip the small classification dataset below called animals. animals datasetThe dataset has the following folder structure:_animals|_cats |_image_1.jpg |_image_2.jpg|_dogs |_image_3.jpg |_image_4.jpgNow that you have the data, you can create a Deep Lake Dataset and initialize its tensors. Running the following code will create Deep Lake dataset inside of the ./animals_deeplakefolder.import deeplakefrom PIL import Imageimport numpy as npimport osds = deeplake.empty('./animals_deeplake') # Create the dataset locallyNext, let's inspect the folder structure for the source dataset './animals' to find the class names and the files that need to be uploaded to the Deep Lake dataset.# Find the class_names and list of files that need to be uploadeddataset_folder = './animals'# Find the subfolders, but filter additional files like DS_Store that are added on Mac machines.class_names = [item for item in os.listdir(dataset_folder) if os.path.isdir(os.path.join(dataset_folder, item))]files_list = []for dirpath, dirnames, filenames in os.walk(dataset_folder): for filename in filenames: files_list.append(os.path.join(dirpath, filename))Next, let's create the dataset tensors and upload metadata. Check out our page on Storage Synchronization for details about the with syntax below.with ds: # Create the tensors with names of your choice. ds.create_tensor('images', htype = 'image', sample_compression = 'jpeg') ds.create_tensor('labels', htype = 'class_label', class_names = class_names) # Add arbitrary metadata - Optional ds.info.update(description = 'My first Deep Lake dataset') ds.images.info.update(camera_type = 'SLR')Specifying htype and dtype is not required, but it is highly recommended in order to optimize performance, especially for large datasets. Usedtypeto specify the numeric type of tensor data, and usehtypeto specify the underlying data structure.Finally, let's populate the data in the tensors. The data is automatically uploaded to the dataset, regardless of whether it's local or in the cloud. with ds: # Iterate through the files and append to Deep Lake dataset for file in files_list: label_text = os.path.basename(os.path.dirname(file)) label_num = class_names.index(label_text) #Append data to the tensors ds.append({'images': deeplake.read(file), 'labels': np.uint32(label_num)})Appending the object deeplake.read(path)is equivalent to appending PIL.Image.fromarray(path). However, the deeplake.read() method is significantly faster because it does not decompress and recompress the image if the compression matches thesample_compression for that tensor. Further details are available in Understanding Compression.In order to maintain proper indexing across tensors, ds.append({...}) requires that you to append to all tensors in the dataset. If you wish to skip tensors during appending, please use ds.append({...}, skip_ok = True) or append to a single tensor using ds.tensor_name.append(...).Check out the first image from this dataset. More details about Accessing Data are available in Step 4.Image.fromarray(ds.images[0].numpy())


Automatic Creation

If your source data conforms to one of the formats below, you can ingest them directly with 1 line of code.YOLOCOCOClassificationsFor example, the above animals dataset can be converted to Deep Lake format using:src = './animals'dest = './animals_deeplake_auto'ds = deeplake.ingest_classification(src, dest)


Dataset inspection

You can print a summary of the dataset structure using:ds.summary()Congrats! You just created your first dataset! 🎉


Creating Tensor Hierarchies

Often it's important to create tensors hierarchically, because information between tensors may be inherently coupled—such as bounding boxes and their corresponding labels. Hierarchy can be created using tensor groups:ds = deeplake.empty('./groups_test') # Creates the dataset# Create tensor hierarchiesds.create_group('my_group')ds.my_group.create_tensor('my_tensor')# Alternatively, a group can us created using create_tensor with '/'ds.create_tensor('my_group_2/my_tensor') #Automatically creates the group 'my_group_2'Tensors in groups are accessed via:ds.my_group.my_tensor#ORds['my_group/my_tensor']For more detailed information regarding accessing datasets and their tensors, check out Step 4.


This Deep Learning Getting Started guide is available as a Colab Notebook

Step 1: Hello WorldStep 2: Creating Deep Lake DatasetsStep 3: Understanding CompressionStep 4: Accessing and Updating DataStep 5: Visualizing DatasetsStep 6: Using Activeloop StorageStep 7: Connecting Deep Lake Datasets to ML FrameworksStep 8: Parallel ComputingStep 9: Dataset Version ControlStep 10: Dataset Filtering


Connecting Data From Your Cloud Using Deep Lake Managed Credentials

Connecting data from your own cloud and managing credentials in Deep Lake unlocks several important capabilities:Access to performant features such as the Deep Lake Compute EngineAccess to the Deep Lake App for datasets stored in your own cloudSimpler access to Deep Lake datasets stored in your own cloud using the Python APINo need for continuously specifying cloud access keys in Python


Managed Credentials

In order for the Deep Lake to access datasets or linked tensors stored in the user's cloud, Deep Lake must authenticate the respective cloud resources. Access can be provided using access keys or using role-based access (provisioning steps here). The video below summarizes the UI for managing your cloud credentials. Authentication Using Managed Credentials


Default Storage

Default storage enables you to map the Deep Lake path hub://org_id/dataset_nameto a cloud path of your choice. Subsequently, all datasets created using the Deep Lake path will be stored at the user-specified specified, and they can be accessed using API tokens and managed credentials from Deep Lake. By default, the default storage is set as Activeloop Storage, and you may change it using the UI below.Note: that in order to visualize data in the Deep Lake browser application, it is necessary to enable CORS in the bucket containing any source data.


Connecting Deep Lake Dataset in your Cloud to the Deep Lake to App

If you do not set the Default Storage as your own cloud, Datasets in user's clouds can be connected to the Deep Lake App using the Python API below. Once a dataset is connected to Deep Lake, it is assigned a Deep Lake path hub://org_id/dataset_name, and it can be accessed using API tokens and managed credentials from Deep Lake, without continuously having to specify cloud credentials.


Connecting Datasets in the Python API

# Step 1: Create the dataset directly in the cloud using your own cloud credsds = deeplake.empty('s3://my_bucket/dataset_name', creds = {...})# Step 2: Connect the dataset to Deep Lake and specify the managed credentials# (creds_key) for accessing the data (See Managed Credentials above)ds.connect(org_id = 'org_id', creds_key = 'my_creds_key', token = 'my_token')ORds.connect(dest_path = 'hub://org_id/dataset_name', creds_key = 'my_creds_key', token = 'my_token')Specifying org_id creates the dataset in the specified org using the dataset_name from the cloud path. Specifying the dest_path creates the dataset at the org_id and dataset_name from the specified path.


Using Manage Credentials with Linked Tensors

Managed credentials can be used for accessing data stored in linked tensors. Simply add the managed credentials to the dataset's creds_keys and assign them to each sample.ds.create_tensors('images', htype = 'link[image]', sample_compression = 'jpeg')ds.add_creds_key('my_creds_key', managed=True)ds.images.append(deeplake.link(link_to_sample, creds_key = 'my_creds_key')


Enabling CORS in AWS for Data Visualization

In order to visualize Deep Lake datasets stored in your own S3 buckets in the Deep Lake app, please enable Cross-Origin Resource Sharing (CORS) in the buckets containing the Deep Lake dataset and any source data in linked tensors, by inserting the snippet below in the CORS section of the Permissions tab for the bucket:[ { 'AllowedHeaders': [ '*' ], 'AllowedMethods': [ 'GET', 'HEAD' ], 'AllowedOrigins': [ '*.activeloop.ai', '*.activeloop.dev' ], 'ExposeHeaders': [] }]


Visualizing Your Datasets Locally

In order to visualize Deep Lake datasets stored in your own cloud using ds.visualize() or using our embedded visualizer, the AllowedOrigins values in CORS should be set to *.


How to optimize Deep Lake for training models at scale

There are several Deep Lake related tuning parameters that affect the speed of Deep Lake OSS and Performant dataloaders. The plot below shows performance of the Deep Lake dataloaders under different scenarios, and it is discussed in detail below.ImageNet data streaming speeds from S3 to a p3.8xlarge EC2 instance. The average image size is 0.114 MB. Details on the simple and complex transform are available in the appending at the end of this page.


Setting num_workers

Both the OSS and Performant dataloaders in Deep Lake have a num_workers parameter that parallelizes the data fetching, decompression, and transformation.Increasing num_workers will not improve performance in GPU-bottlenecked scenarios. Therefore, we recommend starting with 2-4 workers, and increasing num_workers it if the GPU utilization is low.Increasing num_workers beyond the number of CPUs on a machine does not improve performance.It is common for GPU machines to have 8x CPUs per GPU Increasing num_workers linearly improves streaming speed, with diminishing returns beyond 8+ workers.Increasing num_workers beyond 16 is generally unnecessary, unless you are running complex transformations.


Choosing the optimal decode_method for images

Faster dataloading is achieved by minimizing the amount of operations that take place before data is delivered to the GPU. It is important to the decode_method parameter in the OSS and Performant dataloaders based on the following guidelines:When transforming images using tools the require numpy arrays as inputs, such as Albumentations, decode_method should be to numpy, which is the default (No parameters changes are needed)When transforming images using tools the require PIL images as inputs, such as torchvision transforms, decode_method should be to {'image_tensor_name': 'pil'}. torchvision.transforms.ToPIL() should be removed from the top of the transforms stack.Leaving the decode_method as numpy may decrease dataloading speed by up to 2X, because the image is decoded to a numpy array and then re-encoded as a PIL image, instead of being directly decoded to a PIL image.


APPENDIX TO THE PLOT ABOVE

The torchvision transforms used to create the comparison in the plot above are:tform_simple = transforms.Compose( [ transforms.Resize((128, 128)), transforms.RandomAffine(20), transforms.RandomHorizontalFlip(p=0.5), transforms.ToTensor(), transforms.Lambda(lambda x: x.repeat(int(3 / x.shape[0]), 1, 1)), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), ])tform_complex = transforms.Compose( [ transforms.Resize((256, 256)), transforms.RandomAffine(20), transforms.RandomHorizontalFlip(p=0.5), transforms.RandomVerticalFlip(p=0.5), transforms.RandomPerspective(distortion_scale=0.5, p=0.5), transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.2), transforms.ToTensor(), transforms.Lambda(lambda x: x.repeat(int(3 / x.shape[0]), 1, 1)), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), ])


Choosing the optimal dataloader

The Deep Lake Performant dataloader streams data faster compared to the OSS dataloaer, due to its C++ implementation that optimizes asynchronous data fetching and decompression.The Performant dataloader is ~1.5-3X faster compared to the OSS version, depending on the complexity of the transform and the number of workers available for parallelization.Distributed training is only available in the Performant dataloader.


How to achieve full reproducibility of model training by combining Deep Lake data lineage with W&B logging

Experiment tracking tools such as Weights & Biases (W&B) improve reproducibility of your machine learning experiments by offering logging of datasets, hyper parameters, source codes, and more. When running model training with W&B and Deep Lake, Deep Lake automatically pushes information required to reproduce the data such as the uri, commit_id, and view_id to the active W&B run. By fully logging the state of your dataset, model, and source code, you can achieve full reproducibility of model training run for datasets of any size.This playbook demonstrates how to use Activeloop Deep Lake with Weights & Biases to:Upload a Deep Lake dataset in a W&B run and create a W&B artifactQuery the dataset using Activeloop and save the query result in optimized format for trainingTrain an object detection model on the saved query result and log the training parameters in a W&B runRe-train the model with adjusted parameters and use W&B to compare the different training runs.


Prerequisites

In addition to installation of commonly used packages in deep-learning, this playbook requires installation of: !pip install deeplake!pip install wandb!pip install albumentationsThe required python imports are:import deeplakeimport albumentations as Afrom albumentations.pytorch import ToTensorV2import numpy as npimport torchimport wandbimport timeimport sysimport mathimport torchvisionfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictorYou should also register with Activeloop and W&B, create API tokens for both tools, and log in to your W&B account via the CLI using:!wandb login


Creating a W&B Artifact from a Deep Lake Dataset

While the most common use case W&B is to track training and evaluation jobs, you may also wrap your dataset upload jobs in a W&B run in order to create W&B artifacts. Any future runs that consume this dataset will also consume the corresponding W&B artifact. As a result, your Deep Lake datasets will be automatically tracked by W&B and can be visualized in the W&B artifacts lineage UI.Any commit, copy or deepcopy operation inside a W&B run will create a W&B artifact. Here we emulate a dataset upload by copying a dataset that is already hosted by Activeloop.WANDB_PROJECT = 'deeplake-demos'run = wandb.init(project=WANDB_PROJECT, job_type='dataset_upload')ds_train = deeplake.deepcopy('hub://activeloop/visdrone-det-train', 'hub://dl-corp/visdrone-det-train', dest_token = 'Insert API Token')ds_val = deeplake.deepcopy('hub://activeloop/visdrone-det-val', 'hub://dl-corp/visdrone-det-val', dest_token = 'Insert API Token')run.finish()You may replace dl-corp in the dataset path above with your own Deep Lake organization in order to run the code.If we open the W&B page for the run above, we see that the datasets has been tracked that artifacts were created for both the training and validation datasets.


Querying the dataset

Let's use the query engine on the Deep Lake UI to filter out samples that contain any cars with a width or height below 20 pixels. Since most images with vehicles contain cars, this query is a reliable proxy for imaging distance.select * where not any(logical_and(logical_or(boxes[:,2]<20, boxes[:,3]<20), labels == 'car'))This query eliminates approximately 50% of the samples in the dataset.


Train a model using the Deep Lake dataset

Suppose we are building a surveillance system to count and classify vehicles in small-to-medium size parking lots. The visdrone dataset is suitable starting point because it contains a variety of images of vehicles taken in cities using a UAV. However, many images are taken from very large viewing distances, thus resulting in small objects that are difficult to detect in object detection models, and are also not relevant to our surveillance application, like the image below.Therefore, we filter out these images and train the model on a subset of the dataset that is more appropriate for our application.


Training the model

Before training the model, which will often happen on a different machine than where the dataset was created, we first re-load the data for training and optimize if for streaming performance.ds_train = deeplake.load('hub://dl-corp/visdrone-det-train', token = 'Insert API Token', read_only = True)ds_val = deeplake.load('hub://dl-corp/visdrone-det-val', token = 'Insert API Token', read_only = True)ds_train_view = ds_train.load_view('6337166131028ba8bcf7f2ff', optimize = True, num_workers = 4)ds_val_view = ds_val.load_view('633716a10c1052d4385ab5c8', optimize = True, num_workers = 4)An object detection model can be trained using the same approach that is used for all Deep Lake datasets, with several examples in our tutorials.When using subsets of datasets, it's advised to remap the input classes for model training.  In this example, the source dataset has 12 classes, but we are only interested in 9 classes containing objects we want to localize in our parking lot (bicycle, car, van, truck, tricycle, awning-tricycle, bus, motor, others).  Therefore, we remap the classes of interest to values 0,1,2,3,4,6,7,8 before feeding them into the model for training.  Later in this playbook, we will experiment with different transform resolutions, so we specify the transform resolution (WIDTH, HEIGHT), BATCH_SIZE, and minimum bounding box area for transformation (MIN_AREA).  All bounding boxes below MIN_AREA are ignored in the transformation and are not fed to the modelWIDTH = 160HEIGHT = 128MIN_AREA = 32BATCH_SIZE = 16# These are the classes we care about and they will be remapped to 0,1,2,3,4,5,6,7,8 in the modelCLASSES_OF_INTEREST = ['bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', 'motor', 'others']# The classes of interest correspond to the following array values in the current datasetINDS_OF_INTEREST = [ds_train.labels.info.class_names.index(item) for item in CLASSES_OF_INTEREST]Next, let's specify an augmentation pipeline, which mostly utilizes Albumentations.  We perform the remapping of the class labels inside the transformation function.# Augmentation pipeline for training using Albumentationstform_train = A.Compose([ A.RandomSizedBBoxSafeCrop(width=WIDTH, height=HEIGHT, erosion_rate=0.2), A.Rotate(limit=20, p=0.5), A.HorizontalFlip(p=0.5), A.RandomBrightnessContrast(brightness_limit=0.1, contrast_limit=0.1, p=0.5), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), ToTensorV2()], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['class_labels', 'bbox_ids'], min_area=MIN_AREA, min_visibility=0.6)) # 'label_fields' and 'box_ids' are all the fields that will be cut when a bounding box is cut.# Augmentation pipeline for validation using Albumentationstform_val = A.Compose([ A.Resize(width=WIDTH, height=HEIGHT), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), ToTensorV2()], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['class_labels', 'bbox_ids'], min_area=MIN_AREA, min_visibility=0.6)) # 'label_fields' and 'box_ids' are all the fields that will be cut when a bounding box is cut.# Transformation function for pre-processing the Deep Lake sample before sending it to the modeldef transform_train(sample_in): # sample_in is a row in the dataset, referenced as a dictionary


Training the model

# sample_in['images'] is like ds.images[index].numpy() # Convert any grayscale images to RGB image = sample_in['images'] shape = image.shape if shape[2] == 1: image = np.repeat(image, int(3/shape[2]), axis = 2) # Convert boxes to Pascal VOC format boxes = coco_2_pascal(sample_in['boxes'], shape) # Filter only the labels that we care about for this training run labels_all = sample_in['labels'] indices = [l for l, label in enumerate(labels_all) if label in INDS_OF_INTEREST] labels_filtered = labels_all[indices] labels_remapped = [INDS_OF_INTEREST.index(label) for label in labels_filtered] boxes_filtered = boxes[indices,:] # Make sure the number of labels and boxes is still the same after filtering assert(len(labels_remapped)) == boxes_filtered.shape[0] # Pass all data to the Albumentations transformation transformed = tform_train(image = image, bboxes = boxes_filtered, bbox_ids = np.arange(boxes_filtered.shape[0]), class_labels = labels_remapped, ) # Convert boxes and labels from lists to torch tensors, because Albumentations does not do that automatically .  # Be very careful with rounding and casting to integers, becuase that can create bounding boxes with invalid dimensions labels_torch = torch.tensor(transformed['class_labels'], dtype = torch.int64) boxes_torch = torch.zeros((len(transformed['bboxes']), 4), dtype = torch.int64) for b, box in enumerate(transformed['bboxes']): boxes_torch[b,:] = torch.tensor(np.round(box)) # Put annotations in a separate object target = {'labels': labels_torch, 'boxes': boxes_torch} return transformed['image'], target# Transformation function for pre-processing the Deep Lake validation sample before sending it to the modeldef transform_val(sample_in): # Convert any grayscale images to RGB image = sample_in['images'] shape = image.shape if shape[2] == 1: image = np.repeat(image, 3, axis = 2) # Convert boxes to Pascal VOC format boxes = coco_2_pascal(sample_in['boxes'], shape)  # Filter only the labels that we care about for this training run labels_all = sample_in['labels'] indices = [l for l, label in enumerate(labels_all) if label in INDS_OF_INTEREST] labels_filtered = labels_all[indices] labels_remapped = [INDS_OF_INTEREST.index(label) for label in labels_filtered] boxes_filtered = boxes[indices,:] # Make sure the number of labels and boxes is still the same after filtering assert(len(labels_remapped)) == boxes_filtered.shape[0] # Pass all data to the Albumentations transformation transformed = tform_val(image = image, bboxes = boxes_filtered, bbox_ids = np.arange(boxes_filtered.shape[0]), class_labels = labels_remapped, ) # Convert boxes and labels from lists to torch tensors, because Albumentations does not do that automatically.  # Be very careful with rounding and casting to integers, becuase that can create bounding boxes with invalid dimensions labels_torch = torch.tensor(transformed['class_labels'], dtype = torch.int64) boxes_torch = torch.zeros((len(transformed['bboxes']), 4), dtype = torch.int64) for b, box in enumerate(transformed['bboxes']): boxes_torch[b,:] = torch.tensor(np.round(box)) # Put annotations in a separate object target = {'labels': labels_torch, 'boxes': boxes_torch} # We also return the shape of the original image in order to resize the predictions to the dataset image size return transformed['image'], target, sample_in['index'], shape# Conversion script for bounding boxes from coco to Pascal VOC formatdef coco_2_pascal(boxes, shape): # Convert bounding boxes to Pascal VOC format and clip bounding boxes to make sure they have non-negative width and height return np.stack((np.clip(boxes[:,0], 0, None), np.clip(boxes[:,1], 0, None), np.clip(boxes[:,0]+np.clip(boxes[:,2], 1, None), 0, shape[1]), np.clip(boxes[:,1]+np.clip(boxes[:,3], 1, None), 0, shape[0])), axis = 1)def collate_fn(batch): return tuple(zip(*batch))This playbook uses a pre-trained torchvision neural network from the torchvision.models module


Training the model

Otherwise, it's on a CPU.device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')print(device)Let's initialize the model and optimizer.model = get_model_object_detection(len(CLASSES_OF_INTEREST))model.to(device)# Specify the optimizerparams = [p for p in model.parameters() if p.requires_grad]optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)Next, we initialize the W&B run and create the dataloaders for training and validation data .  We will log the training loss and validation IOU, as well as other parameters like the transform resolution.Creation of the Deep Lake dataloaders is a trigger for the W&B run to log Deep Lake-related information.  Therefore, the the W&B run should be initialized before dataloader creation.config={'width': WIDTH, 'height': HEIGHT, 'min_area': MIN_AREA, 'batch_size': BATCH_SIZE}run = wandb.init(project=WANDB_PROJECT, config = config)train_loader = ds_train_view.pytorch(num_workers = 8, shuffle = True, transform = transform_train, tensors = ['images', 'labels', 'boxes'], batch_size = BATCH_SIZE, collate_fn = collate_fn)val_loader = ds_val_view.pytorch(num_workers = 8, shuffle = False, transform = transform_val, tensors = ['images', 'labels', 'boxes'], batch_size = BATCH_SIZE, collate_fn = collate_fn, return_index = True)The model and data are ready for training 🚀!Let's train the model for 8 epochs and save the gradients, parameters, and final trained model as an artifact.# Train the model for 8 epochnum_epochs = 8lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=4, gamma=0.1) wandb.watch(model, log='all', log_freq=50)wandb.define_metric('epoch')wandb.define_metric('validation_iou', step_metric='epoch')for epoch in range(num_epochs): # loop over the dataset multiple times print('------------------ Training Epoch {} ------------------'.format(epoch)) train_one_epoch(model, optimizer, train_loader, device) lr_scheduler.step() print('----------- Evaluating the Model ------------') iou = evaluate_iou(model, val_loader, device) wandb.log({'epoch': epoch, 'validation_iou': iou})torch.save(model.state_dict(), 'model_weights_wandb.pth')model_artifact = wandb.Artifact('object_detection_model', 'model')model_artifact.add_file('model_weights_wandb.pth')run.log_artifact(model_artifact)run.finish()print('Finished Training')In the W&B UI for this run, we see that in addition to the metrics and parameters that are typically logged by W&B, the Deep Lake integration also logged the dataset uri, commit_id, and view_id for the training and evaluation data, which uniquely identifies all the data that was used in this training project.  The Deep Lake integration with W&B logs the dataset uri, commit_id, and view_id in a training run even if a W&B artifact was not created for the Deep Lake dataset.I


Training the model

We define helper functions for loading the model, training for 1 epoch (including W&B logging), and evaluating the model by computing the average IOU (intersection-over-union) for the bounding boxes.# Helper function for loading the modeldef get_model_object_detection(num_classes): # Load an instance segmentation model pre-trained on COCO model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) # Get number of input features for the classifier in_features = model.roi_heads.box_predictor.cls_score.in_features # replace the pre-trained head with a new one model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) return model # Helper function for training for 1 epochdef train_one_epoch(model, optimizer, data_loader, device): model.train() start_time = time.time() for i, data in enumerate(data_loader): images = list(image.to(device) for image in data[0]) targets = [{k: v.to(device) for k, v in t.items()} for t in data[1]] loss_dict = model(images, targets) losses = sum(loss for loss in loss_dict.values()) loss_value = losses.item() wandb.log({'loss': loss_value}) # Print performance statistics if i%10 ==0: batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] loss: %.3f, speed: %.2f' % (i, loss_value, speed)) if not math.isfinite(loss_value): print(f'Loss is {loss_value}, stopping training') print(loss_dict) break optimizer.zero_grad() losses.backward() optimizer.step() # Helper function for computing the average IOU (intersection-over-union) for all the datadef evaluate_iou(model, data_loader, device = 'cpu', score_thresh = 0.5): # This function removes predictions in the output and IUO calculation that are below a confidence threshold .  # Set the model to eval mode.  model.eval() ious_list = [] start_time = time.time() for i, data in enumerate(data_loader): images = list(image.to(device) for image in data[0]) ground_truths = [{k: v.to(device) for k, v in t.items()} for t in data[1]] model_start = time.time() with torch.no_grad(): predictions = model(images) model_end = time.time() assert len(ground_truths) == len(predictions) # Check if data in dataloader is consistent for j, pred in enumerate(predictions): # Ignore boxes below the confidence threshold thresh_inds = pred['scores']>score_thresh pred_boxes = pred['boxes'][thresh_inds] pred_labels = pred['labels'][thresh_inds] pred_scores = pred['scores'][thresh_inds] # Find the union of prediceted and groud truth labels and iterate through it all_labels = np.union1d(pred_labels.to('cpu'), ground_truths[j]['labels'].to('cpu')) ious = np.zeros((len(all_labels))) for l, label in enumerate(all_labels): # Find the boxes corresponding to the label boxes_1 = pred_boxes[pred_labels == label] boxes_2 = ground_truths[j]['boxes'][ground_truths[j]['labels'] == label] iou = torchvision.ops.box_iou(boxes_1, boxes_2).cpu() # This method returns a matrix of the IOU of each box with every other box.  # Consider the IOU as the maximum overlap of a box with any other box.  Find the max along the axis that has the most boxes.  if 0 in iou.shape: ious[l] = 0 else: if boxes_1.shape>boxes_2.shape: max_iou, _ = iou.max(dim=0) else: max_iou, _ = iou.max(dim=1) # Compute the average iou for that label ious[l] = np.mean(np.array(max_iou)) #Take the average iou for all the labels.  If there are no labels, set the iou to 0.  if len(ious)>0: ious_list.append(np.mean(ious)) else: ious_list.append(0) # Print progress if i%10 ==0: batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] speed: %.2f' % (i, speed)) return sum(ious_list)/len(ious_list) Training is performed on a GPU if possible


Notes on GPU Performance

Using W&B automatic logging of CPU and GPU performance, we observe that the GPU utilization for the training runs on this Tesla V100 GPU was approximately 90%, aside from the dips between epochs when the shuffle buffer was filling. Note that the data was streamed from Activeloop storage (not in AWS) to an AWS SageMaker instance. This is made possible by Deep Lake's efficient data format and high-performance dataloader.


Congratulations 🚀. You can now use Activeloop Deep Lake and Weights & Biases to experiment and train models will full reproducibility!


Improving the Model Performance

The average IOU of 0.29 achieved in the previous training run is likely unsatisfactory for a deploying a working product. Two potential explanations for the poor performance are:Despite filtering our samples with tiny objects, the dataset still contains fairly small bounding boxes that are difficult to detect by object detection modelsThe differences between some objects in a birds-eye view are subtle, even for human perception, such as the cars and vans in the image below.One remedy for both problems is to train models higher-resolution images, so let's increase the resolution of the transformation and examine its effect on model performance. In addition to changing the resolution, we must also scale MIN_AREA proportionally to the image area, so that the same bounding boxes are ignored in two training runs.WIDTH = 320HEIGHT = 256MIN_AREA = 128After retraining the model using the same code above, we observe that the average IOU increased from 0.29 to 0.37, which is substantial given the simple increase in image resolution. The model is still not production-ready, and further opportunities for improvement are:Assessing model performance on a per-image basis, which helps identify mislabeled or difficult data. A playbook for this workflow is available here.Adding random real-world images that do not contain the objects of interest. This helps the model eliminate false positives. Adding more data to the training set. 3000 images is likely not enough for a high-accuracy model.Strengthening of transformations that affect image color, blur, warping, and othersExploring different optimizers, learning rates, and schedulersFurther increasing the transform resolution until diminishing returns are achieved


CONTAINS and ==

# Exact match, which generally requires that the sample# has 1 value, i.e. no lists or multi-dimensional arraysselect * where tensor_name == 'text_value' # If value is numericselect * where tensor_name == numeric_value # If values is textselect * where contains(tensor_name, 'text_value')Any special characters in tensor or group names should be wrapped with double-quotes:select * where contains('tensor-name', 'text_value')select * where 'tensor_name/group_name' == numeric_value


SHAPE

select * where shape(tensor_name)[dimension_index] > numeric_value select * where shape(tensor_name)[1] > numeric_value # Second array dimension > value


LIMIT

select * where contains(tensor_name, 'text_value') limit num_samples


AND, OR, NOT

select * where contains(tensor_name, 'text_value') and NOT contains(tensor_name_2, numeric_value)select * where contains(tensor_name, 'text_value') or tensor_name_2 == numeric_valueselect * where (contains(tensor_name, 'text_value') and shape(tensor_name_2)[dimension_index]>numeric_value) or contains(tensor_name, 'text_value_2')


UNION and INTERSECT

(select * where contains(tensor_name, 'value')) intersect (select * where contains(tensor_name, 'value_2'))(select * where contains(tensor_name, 'value') limit 100) union (select * where shape(tensor_name)[0] > numeric_value limit 100)


ORDER BY

# Order by requires that sample is numeric and has 1 value, # i.e. no lists or multi-dimensional arrays# The default order is ASCENDING (asc)select * where contains(tensor_name, 'text_value') order by tensor_name asc


ANY, ALL, and ALL_STRICT

all adheres to NumPy and list logic where all(empty_sample) returns Trueall_strict is more intuitive for queries so all_strict(empty_sample) returns Falseselect * where all(tensor_name==0) # Returns True for empty samplesselect * where all_strict(tensor_name[:,2]>numeric_value) # Returns False for empty samplesselect * where any(tensor_name[0:6]>numeric_value)


IN and BETWEEN

Only works for scalar numeric values and text references to class_namesselect * where tensor_name in (1, 2, 6, 10)select * where class_label_tensor_name in ('car', 'truck')select * where tensor_name between 5 and 20LOGICAL_AND and LOGICAL_ORselect * where any(logical_and(tensor_name_1[:,3]>numeric_value, tensor_name_2 == 'text_value'))


REFERENCING SAMPLES IN EXISTING TENORS

# Select based on index (row_number)select * where row_number() == 10# Referencing values of of a tensor at index (row_number)select * order by l2_norm(<tensor_name> - data(<tensor_name>, index))# Finds rows of data with embeddings most similar to index 10select * order by l2_norm(embedding - data(embedding, 10))


SAMPLE BY

select * sample by weight_choice(expression_1: weight_1, expression_2: weight_2, ...) replace True limit Nweight_choice resolves the weight that is used when multiple expressions evaluate to True for a given sample. Options are max_weight, sum_weight. For example, if weight_choice is max_weight, then the maximum weight will be chosen for that sample.replace determines whether samples should be drawn with replacement. It defaults to True.limit specifies the number of samples that should be returned. If unspecified, the sampler will return the number of samples corresponding to the length of the dataset