Htypes

Examples >>> ds = hub.dataset('......') Copy to clipboard Add the names of the creds you want to use (not needed for http/local urls) >>> ds.add_creds_key('MY_S3_KEY') >>> ds.add_creds_key('GCS_KEY') Copy to clipboard Populate the names added with creds dictionary These creds are only present temporarily and will have to be repopulated on every reload >>> ds.populate_creds('MY_S3_KEY', {}) # add creds here >>> ds.populate_creds('GCS_KEY', {}) # add creds here Copy to clipboard Create a tensor that can contain links >>> ds.create_tensor('img', htype='link[image]', verify=True, create_shape_tensor=False, create_sample_info_tensor=False) Copy to clipboard Populate the tensor with links >>> ds.img.append(hub.link('s3://abc/def.jpeg', creds_key='MY_S3_KEY')) >>> ds.img.append(hub.link('gcs://ghi/jkl.png', creds_key='GCS_KEY')) >>> ds.img.append(hub.link('https://picsum.photos/200/300')) # http path doesn't need creds >>> ds.img.append(hub.link('./path/to/cat.jpeg')) # local path doesn't need creds >>> ds.img.append(hub.link('s3://abc/def.jpeg')) # this will throw an exception as cloud paths always need creds_key >>> ds.img.append(hub.link('s3://abc/def.jpeg', creds_key='ENV')) # this will use creds from environment Copy to clipboard Accessing the data >>> for i in range(5): .. .  ds.img[i].numpy() ...  Copy to clipboard Updating a sample >>> ds.img[0] = hub.link('./data/cat.jpeg') Copy to clipboard


Htypes

'htype' is the class of a tensor: image, bounding box, generic tensor, etc.When not specified, the unspecified options will be inferred from the data: Copy to clipboard If you know beforehand, you can use htype at creation: Copy to clipboard Specifying an htype allows for strict settings and error handling, and it is critical for increasing the performance of hub datasets containing rich data such as images and videos.Supported htypes and their respective defaults are: HTYPE DTYPE COMPRESSION image uint8 None image.rgb uint8 None image.gray uint8 None class_label uint32 None bbox float32 None video uint8 None binary_mask bool None segment_mask uint32 None keypoints_coco int32 None point int32 None audio float64 None text str None json Any None list List None dicom None dcm link str None sequence None None Sequence htype A special meta htype for tensors where each sample is a sequence.  The items in the sequence are samples of another htype.  It is a wrapper htype that can wrap other htypes like sequence[image], sequence[video], sequence[text], etc.  Examples >>> ds.create_tensor('seq', htype='sequence') >>> ds.seq.append([1, 2, 3]) >>> ds.seq.append([4, 5, 6]) >>> ds.seq.numpy() array([[[1], [2], [3]], [[4], [5], [6]]]) Copy to clipboard >>> ds.create_tensor('image_seq', htype='sequence[image]', sample_compression='jpg') >>> ds.image_seq.append([hub.read('img01.jpg'), hub.read('img02.jpg')]) Copy to clipboard Link htype Link htype is a special meta htype that allows linking of external data (files) to the dataset, without storing the data in the dataset itself.  Moreover, there can be variations in this htype, such as link[image], link[video], link[audio], etc.  that would enable the activeloop visualizer to correctly display the data.  No data is actually loaded until you try to read the sample from a dataset.  There are a few exceptions to this:- If verify=True was specified during create_tensor of the tensor to which this is being added, some metadata is read to verify the integrity of the sample.  If create_shape_tensor=True was specified during create_tensor of the tensor to which this is being added, the shape of the sample is read.  If create_sample_info_tensor=True was specified during create_tensor of the tensor to which this is being added, the sample info is read


Sequence htype

A special meta htype for tensors where each sample is a sequence. The items in the sequence are samples of another htype. It is a wrapper htype that can wrap other htypes like sequence[image], sequence[video], sequence[text], etc. Examples Copy to clipboard Copy to clipboard


Link htype

Link htype is a special meta htype that allows linking of external data (files) to the dataset, without storing the data in the dataset itself. Moreover, there can be variations in this htype, such as link[image], link[video], link[audio], etc. that would enable the activeloop visualizer to correctly display the data. No data is actually loaded until you try to read the sample from a dataset. There are a few exceptions to this:- If verify=True was specified during create_tensor of the tensor to which this is being added, some metadata is read to verify the integrity of the sample. If create_shape_tensor=True was specified during create_tensor of the tensor to which this is being added, the shape of the sample is read. If create_sample_info_tensor=True was specified during create_tensor of the tensor to which this is being added, the sample info is read. Examples Copy to clipboard Add the names of the creds you want to use (not needed for http/local urls) Copy to clipboard Populate the names added with creds dictionary These creds are only present temporarily and will have to be repopulated on every reload Copy to clipboard Create a tensor that can contain links Copy to clipboard Populate the tensor with links Copy to clipboard Accessing the data Copy to clipboard Updating a sample Copy to clipboard


How to query datasets using the Deep Lake Tensor Query Language (TQL)

Querying datasets is a critical aspect of data science workflows that enables users to filter datasets and focus their work on the most relevant data. Deep Lake offers a highly-performant query engine built in C++ and optimized for the Deep Lake data format. The Deep Lake query engine is only accessible to registered and authenticated users, and it applies usage restrictions based on your Deep Lake Plan.


Querying in the low-level Python API

Queries can also be performed in the Python API using:view = ds.query(<query_string>)


Saving and utilizing dataset query results in the low-level Python API

The query results (Dataset Views) can be saved in the UI as shown above, or if the view is generated in Python, it can be saved using the Python API below. Full details are available here.ds_view.save_view(message = 'Samples with monarchs')In order to maintain data lineage, Dataset Views are immutable and are connected to specific commits. Therefore, views can only be saved if the dataset has a commit and there are no uncommitted changes in the HEAD. You can check for this using ds.has_head_changes can be loaded in the python API and they can passed to ML frameworks just like regular datasets:ds_view = ds.load_view(view_id, optimize = True, num_workers = 2)for data in ds_view.pytorch(): # Training loop hereThe optimize parameter in ds.load_view(..., optimize = True) materializes the Dataset View into a new sub-dataset that is optimized for streaming. If the original dataset uses linked tensors, the data will be copied to Deep Lake format.Optimizing the Dataset View is critical for achieving rapid streaming.If the saved Dataset View is no longer needed, it can be deleted using:ds.delete_view(view_id)


Query Syntax

TQL Syntax


Querying in the Vector Store Python API

view = vector_store.search(query = <query_string>, exec_option = 'compute_engine')


deeplake.core.sample

class deeplake.core.sample.Sample __init__(path: Optional[str] = None, array: Optional[ndarray] = None, buffer: Optional[Union[bytes, memoryview]] = None, compression: Optional[str] = None, verify: bool = False, shape: Optional[Tuple[int]] = None, dtype: Optional[str] = None, creds: Optional[Dict] = None, storage: Optional[StorageProvider] = None) Represents a single sample for a tensor. Provides all important meta information in one place. Note If self.is_lazy is True, this Sample doesn't actually have any data loaded. To read this data, simply try to read it into a numpy array (sample.array) Parameters path (str) – Path to a sample stored on the local file system that represents a single sample. If path is provided, array should not be. Implicitly makes self.is_lazy == True. array (np.ndarray) – Array that represents a single sample. If array is provided, path should not be. Implicitly makes self.is_lazy == False. buffer – (bytes): Byte buffer that represents a single sample. If compressed, compression argument should be provided. compression (str) – Specify in case of byte buffer. verify (bool) – If a path is provided, verifies the sample if True. shape (Tuple[int]) – Shape of the sample. dtype (optional, str) – Data type of the sample. creds (optional, Dict) – Credentials for s3, gcp and http urls. storage (optional, StorageProvider) – Storage provider. property array: ndarray Return numpy array corresponding to the sample. Decompresses the sample if necessary. Example >>> sample = deeplake.read('./images/dog.jpg') >>> arr = sample.array >>> arr.shape (323, 480, 3) Copy to clipboard compressed_bytes(compression: Optional[str]) → bytes Returns this sample as compressed bytes. Note If this sample is pointing to a path and the requested compression is the same as it's stored in, the data is returned without re-compressing. Parameters compression (Optional[str]) – self.array will be compressed into this format. If compression is None, return uncompressed_bytes(). Returns Bytes for the compressed sample. Contains all metadata required to decompress within these bytes. Return type bytes Raises ValueError – On recompression of unsupported formats. property pil: Image Return PIL image corresponding to the sample. Decompresses the sample if necessary. Example >>> sample = deeplake.read('./images/dog.jpg') >>> pil = sample.pil >>> pil.size (480, 323) Copy to clipboard uncompressed_bytes() → Optional[bytes] Returns uncompressed bytes.


deeplake.core.linked_sample

class deeplake.core.linked_sample.LinkedSample(path: str, creds_key: Optional[str] = None) Represents a sample that is initialized using external links. See deeplake.link().


deeplake.core.partial_sample

class deeplake.core.partial_sample.PartialSample(sample_shape: Tuple[int, ...], tile_shape: Optional[Tuple[int, ...]] = None, dtype: Optional[Union[str, dtype]] = dtype('uint8')) Represents a sample that is initialized by just shape and the data is updated later.


deeplake.core.linked_tiled_sample

class deeplake.core.linked_tiled_sample.LinkedTiledSample(path_array: ndarray, creds_key: Optional[str] = None) Represents a sample that is initialized using external links. See deeplake.link_tiled().


deeplake.core.storage

Base Storage Provider class deeplake.core.storage.StorageProvider abstract __delitem__(path: str) Delete the object present at the path.  Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  abstract __getitem__(path: str) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  abstract __iter__() Generator function that iterates over the keys of the provider.  Yields str – the path of the object that it is iterating over, relative to the root of the provider.  abstract __len__() Returns the number of files present inside the root of the provider.  Returns the number of files present inside the root.  Return type int abstract __setitem__(path: str, value: bytes) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  __weakref__ list of weak references to the object (if defined) abstract _all_keys() → Set[str] Generator function that iterates over the keys of the provider.  Returns set of all keys present at the root of the provider.  Return type set _is_hub_path = False An abstract base class for implementing a storage provider.  To add a new provider using Provider, create a subclass and implement all 5 abstract methods below.  check_readonly() Raises an exception if the provider is in read-only mode.  abstract clear(prefix='') Delete the contents of the provider.  copy() Returns a copy of the provider.  Returns A copy of the provider.  Return type StorageProvider disable_readonly() Disables read-only mode for the provider.  enable_readonly() Enables read-only mode for the provider.  flush() Only needs to be implemented for caches.  Flushes the data to the next storage provider.  Should be a no op for Base Storage Providers like local, s3, azure, gcs, etc.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  maybe_flush() Flush cache if autoflush has been enabled


deeplake.core.storage

Called at the end of methods which write data, to ensure consistency as a default .  set_bytes(path: str, value: bytes, start_byte: Optional[int] = None, overwrite: Optional[bool] = False) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  start_byte (int, optional) – If only specific bytes starting from start_byte are to be assigned.  overwrite (boolean, optional) – If the value is True, if there is an object present at the path it is completely overwritten, without fetching it's data.  Raises InvalidBytesRequestedError – If start_byte < 0.  ReadOnlyModeError – If the provider is in read-only mode.  LRU Cache class deeplake.core.storage.LRUCache Bases: StorageProvider LRU Cache that uses StorageProvider for caching __delitem__(path: str) Deletes the object present at the path from the cache and the underlying storage.  Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str) If item is in cache_storage, retrieves from there and returns.  If item isn't in cache_storage, retrieves from next storage, stores in cache_storage (if possible) and returns.  Parameters path (str) – The path relative to the root of the underlying storage.  Raises KeyError – if an object is not found at the path.  Returns The bytes of the object present at the path.  Return type bytes __getstate__() → Dict[str, Any] Returns the state of the cache, for pickling __init__(cache_storage: StorageProvider, next_storage: Optional[StorageProvider], cache_size: int) Initializes the LRUCache.  It can be chained with other LRUCache objects to create multilayer caches.  Parameters cache_storage (StorageProvider) – The storage being used as the caching layer of the cache.  This should be a base provider such as MemoryProvider, LocalProvider or S3Provider but not another LRUCache.  next_storage (StorageProvider) – The next storage layer of the cache.  This can either be a base provider (i.e.  it is the final storage) or another LRUCache (i.e.  in case of chained cache).  While reading data, all misses from cache would be retrieved from here.  While writing data, the data will be written to the next_storage when cache_storage is full or flush is called.  cache_size (int) – The total space that can be used from the cache_storage in bytes.  This number may be less than the actual space available on the cache_storage.  Setting it to a higher value than actually available space may lead to unexpected behaviors.  __iter__() Generator function that iterates over the keys of the cache and the underlying storage.  Yields str – the path of the object that it is iterating over, relative to the root of the provider


deeplake.core.storage

__len__() Returns the number of files present in the cache and the underlying storage .  Returns the number of files present inside the root.  Return type int __setitem__(path: str, value: Union[bytes, DeepLakeMemoryObject]) Puts the item in the cache_storage (if possible), else writes to next_storage.  Parameters path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises ReadOnlyError – If the provider is in read-only mode.  __setstate__(state: Dict[str, Any]) Recreates a cache with the same configuration as the state.  Parameters state (dict) – The state to be used to recreate the cache.  Note While restoring the cache, we reset its contents.  In case the cache storage was local/s3 and is still accessible when unpickled (if same machine/s3 creds present respectively), the earlier cache contents are no longer accessible.  _all_keys() Helper function that lists all the objects present in the cache and the underlying storage.  Returns set of all the objects found in the cache and the underlying storage.  Return type set _flush_if_not_read_only() Flushes the cache if not in read-only mode.  _forward(path) Forward the value at a given path to the next storage, and un-marks its key.  _forward_value(path, value) Forwards a path-value pair to the next storage, and un-marks its key.  Parameters path (str) – the path to the object relative to the root of the provider.  value (bytes, DeepLakeMemoryObject) – the value to send to the next storage.  _free_up_space(extra_size: int) Helper function that frees up space the requred space in cache.No action is taken if there is sufficient space in the cache.  Parameters extra_size (int) – the space that needs is required in bytes.  _insert_in_cache(path: str, value: Union[bytes, DeepLakeMemoryObject]) Helper function that adds a key value pair to the cache.  Parameters path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises ReadOnlyError – If the provider is in read-only mode.  _pop_from_cache() Helper function that pops the least recently used key, value pair from the cache clear(prefix='') Deletes ALL the data from all the layers of the cache and the actual storage.  This is an IRREVERSIBLE operation.  Data once deleted can not be recovered.  clear_cache() Flushes the content of all the cache layers if not in read mode and and then deletes contents of all the layers of it.  This doesn't delete data from the actual storage.  clear_deeplake_objects() Removes all DeepLakeMemoryObjects from the cache.  flush() Writes data from cache_storage to next_storage.  Only the dirty keys are written.  This is a cascading function and leads to data being written to the final storage in case of a chained cache.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range


deeplake.core.storage

Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range .  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  S3GetAccessError – Invalid credentials for the object path storage.  S3GetError – Any other error while retrieving the object.  need_to_reload_creds(err: ClientError) → bool Checks if the credentials need to be reloaded.  This happens if the credentials were loaded from the environment and have now expired.  rename(root) Rename root folder.  Google Cloud Storage Provider class deeplake.core.storage.GCSProvider Bases: StorageProvider Provider class for using GC storage.  __contains__(key) Checks if key exists in mapping.  __delitem__(key) Remove key.  __getitem__(key) Retrieve data.  __init__(root: str, token: Optional[Union[str, Dict]] = None, project: Optional[str] = None) Initializes the GCSProvider.  Example >>> gcs_provider = GCSProvider('gcs://my-bucket/gcs_ds') Copy to clipboard Parameters root (str) – The root of the provider.  All read/write request keys will be appended to root.  token (str/Dict) – GCP token, used for fetching credentials for storage).  Can be a path to the credentials file, actual credential dictionary or one of the folowing: - google_default: Tries to load default credentials for the specified project.  - cache: Retrieves the previously used credentials from cache if exist.  - anon: Sets credentials=None.  - browser: Generates and stores new token file using cli.  project (str) – Name of the project from GCloud.  Raises ModuleNotFoundError – If google cloud packages aren't installed.  __iter__() Iterating over the structure.  __len__() Returns length of the structure.  __setitem__(key, value) Store value in key.  _all_keys() Generator function that iterates over the keys of the provider.  Returns set of all keys present at the root of the provider.  Return type set _set_hub_creds_info(hub_path: str, expiration: str, db_engine: bool = True, repository: Optional[str] = None) Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters hub_path (str) – The deeplake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  db_engine (bool) – Whether Activeloop DB Engine enabled.  repository (str, Optional) – Backend repository where the dataset is stored.  clear(prefix='') Remove all keys with given prefix below root - empties out mapping


deeplake.core.storage

Parameters path (str) – The path relative to the root of the provider .  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  get_deeplake_object(path: str, expected_class, meta: Optional[Dict] = None, url=False, partial_bytes: int = 0) If the data at path was stored using the output of a DeepLakeMemoryObject's tobytes function, this function will read it back into object form & keep the object in cache.  Parameters path (str) – Path to the stored object.  expected_class (callable) – The expected subclass of DeepLakeMemoryObject.  meta (dict, optional) – Metadata associated with the stored object url (bool) – Get presigned url instead of downloading chunk (only for videos) partial_bytes (int) – Number of bytes to read from the beginning of the file.  If 0, reads the whole file.  Defaults to 0.  Raises ValueError – If the incorrect expected_class was provided.  ValueError – If the type of the data at path is invalid.  ValueError – If url is True but expected_class is not a subclass of BaseChunk.  Returns An instance of expected_class populated with the data.  register_deeplake_object(path: str, obj: DeepLakeMemoryObject) Registers a new object in the cache.  remove_deeplake_object(path: str) Removes a DeepLakeMemoryObject from the cache.  S3 Storage Provider class deeplake.core.storage.S3Provider Bases: StorageProvider Provider class for using S3 storage.  __delitem__(path) Delete the object present at the path.  Parameters path (str) – the path to the object relative to the root of the S3Provider.  Note If the object is not found, s3 won't raise KeyError.  Raises S3DeletionError – Any S3 error encountered while deleting the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path) Gets the object present at the path.  Parameters path (str) – the path relative to the root of the S3Provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  __init__(root: str, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, endpoint_url: Optional[str] = None, aws_region: Optional[str] = None, profile_name: Optional[str] = None, token: Optional[str] = None, **kwargs) Initializes the S3Provider Example >>> s3_provider = S3Provider('snark-test/benchmarks') Copy to clipboard Parameters root (str) – The root of the provider.  All read/write request keys will be appended to root.  aws_access_key_id (str, optional) – Specifies the AWS access key used as part of the credentials to authenticate the user


deeplake.core.storage

aws_secret_access_key (str, optional) – Specifies the AWS secret key used as part of the credentials to authenticate the user .  aws_session_token (str, optional) – Specifies an AWS session token used as part of the credentials to authenticate the user.  endpoint_url (str, optional) – The complete URL to use for the constructed client.  This needs to be provided for cases in which you're interacting with MinIO, Wasabi, etc.  aws_region (str, optional) – Specifies the AWS Region to send requests to.  profile_name (str, optional) – Specifies the AWS profile name to use.  token (str, optional) – Activeloop token, used for fetching credentials for Deep Lake datasets (if this is underlying storage for Deep Lake dataset).  This is optional, tokens are normally autogenerated.  **kwargs – Additional arguments to pass to the S3 client.  Includes: expiration.  __iter__() Generator function that iterates over the keys of the S3Provider.  Yields str – the name of the object that it is iterating over.  __len__() Returns the number of files present at the root of the S3Provider.  Note This is an expensive operation.  Returns the number of files present inside the root.  Return type int Raises S3ListError – Any S3 error encountered while listing the objects.  __setitem__(path, content) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the S3Provider.  content (bytes) – the value to be assigned at the path.  Raises S3SetError – Any S3 error encountered while setting the value at the path.  ReadOnlyError – If the provider is in read-only mode.  _all_keys() Helper function that lists all the objects present at the root of the S3Provider.  Returns set of all the objects found at the root of the S3Provider.  Return type set Raises S3ListError – Any S3 error encountered while listing the objects.  _check_update_creds(force=False) If the client has an expiration time, check if creds are expired and fetch new ones.  This would only happen for datasets stored on Deep Lake storage for which temporary 12 hour credentials are generated.  _set_hub_creds_info(hub_path: str, expiration: str, db_engine: bool = True, repository: Optional[str] = None) Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters hub_path (str) – The deeplake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  db_engine (bool) – Whether Activeloop DB Engine enabled.  repository (str, Optional) – Backend repository where the dataset is stored.  _state_keys() Keys used to store the state of the provider.  clear(prefix='') Deletes ALL data with keys having given prefix on the s3 bucket (under self.root)


deeplake.core.storage

sync() Sync provider keys with actual storage Local Storage Provider class deeplake.core.storage.LocalProvider Bases: StorageProvider Provider class for using the local filesystem .  __delitem__(path: str) Delete the object present at the path.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> del local_provider['abc.txt'] Copy to clipboard Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str) Gets the object present at the path within the given byte range.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> my_data = local_provider['abc.txt'] Copy to clipboard Parameters path (str) – The path relative to the root of the provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  __init__(root: str) Initializes the LocalProvider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') Copy to clipboard Parameters root (str) – The root of the provider.  All read/write request keys will be appended to root.' Raises FileAtPathException – If the root is a file instead of a directory.  __iter__() Generator function that iterates over the keys of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> for my_data in local_provider: ...  pass Copy to clipboard Yields str – the path of the object that it is iterating over, relative to the root of the provider.  __len__() Returns the number of files present inside the root of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> len(local_provider) Copy to clipboard Returns the number of files present inside the root.  Return type int __setitem__(path: str, value: bytes) Sets the object present at the path with the value Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> local_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises Exception – If unable to set item due to directory at path or permission or space issues.  FileAtPathException – If the directory to the path is a file instead of a directory.  ReadOnlyError – If the provider is in read-only mode.  _all_keys(refresh: bool = False) → Set[str] Lists all the objects present at the root of the Provider.  Parameters refresh (bool) – refresh keys Returns set of all the objects found at the root of the Provider.  Return type set _check_is_file(path: str) Checks if the path is a file.  Returns the full_path to file if True


deeplake.core.storage

Example >>> memory_provider = MemoryProvider('xyz') >>> len(memory_provider) Copy to clipboard Returns the number of files present inside the root .  Return type int __setitem__(path: str, value: bytes) Sets the object present at the path with the value Example >>> memory_provider = MemoryProvider('xyz') >>> memory_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises ReadOnlyError – If the provider is in read-only mode.  _all_keys() Lists all the objects present at the root of the Provider.  Returns set of all the objects found at the root of the Provider.  Return type set clear(prefix='') Clears the provider.


Base Storage Provider

class deeplake.core.storage.StorageProvider abstract __delitem__(path: str) Delete the object present at the path.  Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  abstract __getitem__(path: str) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  abstract __iter__() Generator function that iterates over the keys of the provider.  Yields str – the path of the object that it is iterating over, relative to the root of the provider.  abstract __len__() Returns the number of files present inside the root of the provider.  Returns the number of files present inside the root.  Return type int abstract __setitem__(path: str, value: bytes) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  __weakref__ list of weak references to the object (if defined) abstract _all_keys() → Set[str] Generator function that iterates over the keys of the provider.  Returns set of all keys present at the root of the provider.  Return type set _is_hub_path = False An abstract base class for implementing a storage provider.  To add a new provider using Provider, create a subclass and implement all 5 abstract methods below.  check_readonly() Raises an exception if the provider is in read-only mode.  abstract clear(prefix='') Delete the contents of the provider.  copy() Returns a copy of the provider.  Returns A copy of the provider.  Return type StorageProvider disable_readonly() Disables read-only mode for the provider.  enable_readonly() Enables read-only mode for the provider.  flush() Only needs to be implemented for caches.  Flushes the data to the next storage provider.  Should be a no op for Base Storage Providers like local, s3, azure, gcs, etc.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  maybe_flush() Flush cache if autoflush has been enabled.  Called at the end of methods which write data, to ensure consistency as a default


deeplake.core.storage

Parameters path (str) – the path to the object relative to the root of the provider .  Returns the full path to the requested file.  Return type str Raises DirectoryAtPathException – If a directory is found at the path.  _set_hub_creds_info(hub_path: str, expiration: str, db_engine: bool = True, repository: Optional[str] = None) Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters hub_path (str) – The deeplake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  db_engine (bool) – Whether Activeloop DB Engine enabled.  repository (str, Optional) – Backend repository where the dataset is stored.  clear(prefix='') Deletes ALL data with keys having given prefix on the local machine (under self.root).  Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(path) Renames root folder Memory Provider class deeplake.core.storage.MemoryProvider Bases: StorageProvider Provider class for using the memory.  __delitem__(path: str) Delete the object present at the path.  Example >>> memory_provider = MemoryProvider('xyz') >>> del memory_provider['abc.txt'] Copy to clipboard Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str) Gets the object present at the path within the given byte range.  Example >>> memory_provider = MemoryProvider('xyz') >>> my_data = memory_provider['abc.txt'] Copy to clipboard Parameters path (str) – The path relative to the root of the provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  __getstate__() → str Does NOT save the in memory data in state.  __init__(root: str = '') __iter__() Generator function that iterates over the keys of the provider.  Example >>> memory_provider = MemoryProvider('xyz') >>> for my_data in memory_provider: ...  pass Copy to clipboard Yields str – the path of the object that it is iterating over, relative to the root of the provider.  __len__() Returns the number of files present inside the root of the provider


LRU Cache

get_deeplake_object(path: str, expected_class, meta: Optional[Dict] = None, url=False, partial_bytes: int = 0) If the data at path was stored using the output of a DeepLakeMemoryObject's tobytes function, this function will read it back into object form & keep the object in cache .  Parameters path (str) – Path to the stored object.  expected_class (callable) – The expected subclass of DeepLakeMemoryObject.  meta (dict, optional) – Metadata associated with the stored object url (bool) – Get presigned url instead of downloading chunk (only for videos) partial_bytes (int) – Number of bytes to read from the beginning of the file.  If 0, reads the whole file.  Defaults to 0.  Raises ValueError – If the incorrect expected_class was provided.  ValueError – If the type of the data at path is invalid.  ValueError – If url is True but expected_class is not a subclass of BaseChunk.  Returns An instance of expected_class populated with the data.  register_deeplake_object(path: str, obj: DeepLakeMemoryObject) Registers a new object in the cache.  remove_deeplake_object(path: str) Removes a DeepLakeMemoryObject from the cache.


LRU Cache

In case the cache storage was local/s3 and is still accessible when unpickled (if same machine/s3 creds present respectively), the earlier cache contents are no longer accessible .  _all_keys() Helper function that lists all the objects present in the cache and the underlying storage.  Returns set of all the objects found in the cache and the underlying storage.  Return type set _flush_if_not_read_only() Flushes the cache if not in read-only mode.  _forward(path) Forward the value at a given path to the next storage, and un-marks its key.  _forward_value(path, value) Forwards a path-value pair to the next storage, and un-marks its key.  Parameters path (str) – the path to the object relative to the root of the provider.  value (bytes, DeepLakeMemoryObject) – the value to send to the next storage.  _free_up_space(extra_size: int) Helper function that frees up space the requred space in cache.No action is taken if there is sufficient space in the cache.  Parameters extra_size (int) – the space that needs is required in bytes.  _insert_in_cache(path: str, value: Union[bytes, DeepLakeMemoryObject]) Helper function that adds a key value pair to the cache.  Parameters path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises ReadOnlyError – If the provider is in read-only mode.  _pop_from_cache() Helper function that pops the least recently used key, value pair from the cache clear(prefix='') Deletes ALL the data from all the layers of the cache and the actual storage.  This is an IRREVERSIBLE operation.  Data once deleted can not be recovered.  clear_cache() Flushes the content of all the cache layers if not in read mode and and then deletes contents of all the layers of it.  This doesn't delete data from the actual storage.  clear_deeplake_objects() Removes all DeepLakeMemoryObjects from the cache.  flush() Writes data from cache_storage to next_storage.  Only the dirty keys are written.  This is a cascading function and leads to data being written to the final storage in case of a chained cache.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path


S3 Storage Provider

class deeplake.core.storage.S3Provider Bases: StorageProvider Provider class for using S3 storage.  __delitem__(path) Delete the object present at the path.  Parameters path (str) – the path to the object relative to the root of the S3Provider.  Note If the object is not found, s3 won't raise KeyError.  Raises S3DeletionError – Any S3 error encountered while deleting the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path) Gets the object present at the path.  Parameters path (str) – the path relative to the root of the S3Provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  __init__(root: str, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, endpoint_url: Optional[str] = None, aws_region: Optional[str] = None, profile_name: Optional[str] = None, token: Optional[str] = None, **kwargs) Initializes the S3Provider Example >>> s3_provider = S3Provider('snark-test/benchmarks') Copy to clipboard Parameters root (str) – The root of the provider.  All read/write request keys will be appended to root.  aws_access_key_id (str, optional) – Specifies the AWS access key used as part of the credentials to authenticate the user.  aws_secret_access_key (str, optional) – Specifies the AWS secret key used as part of the credentials to authenticate the user.  aws_session_token (str, optional) – Specifies an AWS session token used as part of the credentials to authenticate the user.  endpoint_url (str, optional) – The complete URL to use for the constructed client.  This needs to be provided for cases in which you're interacting with MinIO, Wasabi, etc.  aws_region (str, optional) – Specifies the AWS Region to send requests to.  profile_name (str, optional) – Specifies the AWS profile name to use.  token (str, optional) – Activeloop token, used for fetching credentials for Deep Lake datasets (if this is underlying storage for Deep Lake dataset).  This is optional, tokens are normally autogenerated.  **kwargs – Additional arguments to pass to the S3 client.  Includes: expiration.  __iter__() Generator function that iterates over the keys of the S3Provider.  Yields str – the name of the object that it is iterating over.  __len__() Returns the number of files present at the root of the S3Provider.  Note This is an expensive operation.  Returns the number of files present inside the root.  Return type int Raises S3ListError – Any S3 error encountered while listing the objects.  __setitem__(path, content) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the S3Provider.  content (bytes) – the value to be assigned at the path.  Raises S3SetError – Any S3 error encountered while setting the value at the path.  ReadOnlyError – If the provider is in read-only mode


S3 Storage Provider

_all_keys() Helper function that lists all the objects present at the root of the S3Provider .  Returns set of all the objects found at the root of the S3Provider.  Return type set Raises S3ListError – Any S3 error encountered while listing the objects.  _check_update_creds(force=False) If the client has an expiration time, check if creds are expired and fetch new ones.  This would only happen for datasets stored on Deep Lake storage for which temporary 12 hour credentials are generated.  _set_hub_creds_info(hub_path: str, expiration: str, db_engine: bool = True, repository: Optional[str] = None) Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters hub_path (str) – The deeplake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  db_engine (bool) – Whether Activeloop DB Engine enabled.  repository (str, Optional) – Backend repository where the dataset is stored.  _state_keys() Keys used to store the state of the provider.  clear(prefix='') Deletes ALL data with keys having given prefix on the s3 bucket (under self.root).  Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  S3GetAccessError – Invalid credentials for the object path storage.  S3GetError – Any other error while retrieving the object.  need_to_reload_creds(err: ClientError) → bool Checks if the credentials need to be reloaded.  This happens if the credentials were loaded from the environment and have now expired.  rename(root) Rename root folder.


Google Cloud Storage Provider

class deeplake.core.storage.GCSProvider Bases: StorageProvider Provider class for using GC storage. __contains__(key) Checks if key exists in mapping. __delitem__(key) Remove key. __getitem__(key) Retrieve data. __init__(root: str, token: Optional[Union[str, Dict]] = None, project: Optional[str] = None) Initializes the GCSProvider. Example >>> gcs_provider = GCSProvider('gcs://my-bucket/gcs_ds') Copy to clipboard Parameters root (str) – The root of the provider. All read/write request keys will be appended to root. token (str/Dict) – GCP token, used for fetching credentials for storage). Can be a path to the credentials file, actual credential dictionary or one of the folowing: - google_default: Tries to load default credentials for the specified project. - cache: Retrieves the previously used credentials from cache if exist. - anon: Sets credentials=None. - browser: Generates and stores new token file using cli. project (str) – Name of the project from GCloud. Raises ModuleNotFoundError – If google cloud packages aren't installed. __iter__() Iterating over the structure. __len__() Returns length of the structure. __setitem__(key, value) Store value in key. _all_keys() Generator function that iterates over the keys of the provider. Returns set of all keys present at the root of the provider. Return type set _set_hub_creds_info(hub_path: str, expiration: str, db_engine: bool = True, repository: Optional[str] = None) Sets the tag and expiration of the credentials. These are only relevant to datasets using Deep Lake storage. This info is used to fetch new credentials when the temporary 12 hour credentials expire. Parameters hub_path (str) – The deeplake cloud path to the dataset. expiration (str) – The time at which the credentials expire. db_engine (bool) – Whether Activeloop DB Engine enabled. repository (str, Optional) – Backend repository where the dataset is stored. clear(prefix='') Remove all keys with given prefix below root - empties out mapping. Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range. Parameters path (str) – The path relative to the root of the provider. start_byte (int, optional) – If only specific bytes starting from start_byte are required. end_byte (int, optional) – If only specific bytes up to end_byte are required. Returns The bytes of the object present at the path within the given byte range. Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0. KeyError – If an object is not found at the path. rename(root) Rename root folder.


Google Drive Storage Provider

class deeplake.core.storage.GDriveProvider Bases: StorageProvider Provider class for using Google Drive storage. __delitem__(path) Delete the object present at the path. Parameters path (str) – the path to the object relative to the root of the provider. Raises KeyError – If an object is not found at the path. __getitem__(path) Gets the object present at the path within the given byte range. Parameters path (str) – The path relative to the root of the provider. Returns The bytes of the object present at the path. Return type bytes Raises KeyError – If an object is not found at the path. __init__(root: str, token: Optional[Union[str, Dict]] = None, makemap: bool = True) Initializes the GDriveProvider Example >>> gdrive_provider = GDriveProvider('gdrive://folder_name/folder_name') Copy to clipboard Parameters root (str) – The root of the provider. All read/write request keys will be appended to root. token (dict, str, optional) – Google Drive token. Can be path to the token file or the actual credentials dictionary. makemap (bool) – Creates path to id map if True. Note Requires client_secrets.json in working directory if token is not provided. Due to limits on requests per 100 seconds on google drive api, continuous requests such as uploading many small files can be slow. Users can request to increse their quotas on their google cloud platform. __iter__() Generator function that iterates over the keys of the provider. Yields str – the path of the object that it is iterating over, relative to the root of the provider. __len__() Returns the number of files present inside the root of the provider. Returns the number of files present inside the root. Return type int __setitem__(path, content) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the provider. value (bytes) – the value to be assigned at the path. _all_keys() Generator function that iterates over the keys of the provider. Returns set of all keys present at the root of the provider. Return type set clear(prefix='') Delete the contents of the provider. sync() Sync provider keys with actual storage


Local Storage Provider

class deeplake.core.storage.LocalProvider Bases: StorageProvider Provider class for using the local filesystem.  __delitem__(path: str) Delete the object present at the path.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> del local_provider['abc.txt'] Copy to clipboard Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str) Gets the object present at the path within the given byte range.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> my_data = local_provider['abc.txt'] Copy to clipboard Parameters path (str) – The path relative to the root of the provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  __init__(root: str) Initializes the LocalProvider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') Copy to clipboard Parameters root (str) – The root of the provider.  All read/write request keys will be appended to root.' Raises FileAtPathException – If the root is a file instead of a directory.  __iter__() Generator function that iterates over the keys of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> for my_data in local_provider: ...  pass Copy to clipboard Yields str – the path of the object that it is iterating over, relative to the root of the provider.  __len__() Returns the number of files present inside the root of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> len(local_provider) Copy to clipboard Returns the number of files present inside the root.  Return type int __setitem__(path: str, value: bytes) Sets the object present at the path with the value Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> local_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises Exception – If unable to set item due to directory at path or permission or space issues.  FileAtPathException – If the directory to the path is a file instead of a directory.  ReadOnlyError – If the provider is in read-only mode.  _all_keys(refresh: bool = False) → Set[str] Lists all the objects present at the root of the Provider.  Parameters refresh (bool) – refresh keys Returns set of all the objects found at the root of the Provider.  Return type set _check_is_file(path: str) Checks if the path is a file.  Returns the full_path to file if True.  Parameters path (str) – the path to the object relative to the root of the provider


Local Storage Provider

Returns the full path to the requested file .  Return type str Raises DirectoryAtPathException – If a directory is found at the path.  _set_hub_creds_info(hub_path: str, expiration: str, db_engine: bool = True, repository: Optional[str] = None) Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters hub_path (str) – The deeplake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  db_engine (bool) – Whether Activeloop DB Engine enabled.  repository (str, Optional) – Backend repository where the dataset is stored.  clear(prefix='') Deletes ALL data with keys having given prefix on the local machine (under self.root).  Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(path) Renames root folder


Memory Provider

class deeplake.core.storage.MemoryProvider Bases: StorageProvider Provider class for using the memory. __delitem__(path: str) Delete the object present at the path. Example >>> memory_provider = MemoryProvider('xyz') >>> del memory_provider['abc.txt'] Copy to clipboard Parameters path (str) – the path to the object relative to the root of the provider. Raises KeyError – If an object is not found at the path. ReadOnlyError – If the provider is in read-only mode. __getitem__(path: str) Gets the object present at the path within the given byte range. Example >>> memory_provider = MemoryProvider('xyz') >>> my_data = memory_provider['abc.txt'] Copy to clipboard Parameters path (str) – The path relative to the root of the provider. Returns The bytes of the object present at the path. Return type bytes Raises KeyError – If an object is not found at the path. __getstate__() → str Does NOT save the in memory data in state. __init__(root: str = '') __iter__() Generator function that iterates over the keys of the provider. Example >>> memory_provider = MemoryProvider('xyz') >>> for my_data in memory_provider: ... pass Copy to clipboard Yields str – the path of the object that it is iterating over, relative to the root of the provider. __len__() Returns the number of files present inside the root of the provider. Example >>> memory_provider = MemoryProvider('xyz') >>> len(memory_provider) Copy to clipboard Returns the number of files present inside the root. Return type int __setitem__(path: str, value: bytes) Sets the object present at the path with the value Example >>> memory_provider = MemoryProvider('xyz') >>> memory_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters path (str) – the path relative to the root of the provider. value (bytes) – the value to be assigned at the path. Raises ReadOnlyError – If the provider is in read-only mode. _all_keys() Lists all the objects present at the root of the Provider. Returns set of all the objects found at the root of the Provider. Return type set clear(prefix='') Clears the provider.


deeplake.core.storage

Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None) Gets the object present at the path within the given byte range .  Parameters path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns The bytes of the object present at the path within the given byte range.  Return type bytes Raises InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(root) Rename root folder.  Google Drive Storage Provider class deeplake.core.storage.GDriveProvider Bases: StorageProvider Provider class for using Google Drive storage.  __delitem__(path) Delete the object present at the path.  Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  __getitem__(path) Gets the object present at the path within the given byte range.  Parameters path (str) – The path relative to the root of the provider.  Returns The bytes of the object present at the path.  Return type bytes Raises KeyError – If an object is not found at the path.  __init__(root: str, token: Optional[Union[str, Dict]] = None, makemap: bool = True) Initializes the GDriveProvider Example >>> gdrive_provider = GDriveProvider('gdrive://folder_name/folder_name') Copy to clipboard Parameters root (str) – The root of the provider.  All read/write request keys will be appended to root.  token (dict, str, optional) – Google Drive token.  Can be path to the token file or the actual credentials dictionary.  makemap (bool) – Creates path to id map if True.  Note Requires client_secrets.json in working directory if token is not provided.  Due to limits on requests per 100 seconds on google drive api, continuous requests such as uploading many small files can be slow.  Users can request to increse their quotas on their google cloud platform.  __iter__() Generator function that iterates over the keys of the provider.  Yields str – the path of the object that it is iterating over, relative to the root of the provider.  __len__() Returns the number of files present inside the root of the provider.  Returns the number of files present inside the root.  Return type int __setitem__(path, content) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  _all_keys() Generator function that iterates over the keys of the provider.  Returns set of all keys present at the root of the provider.  Return type set clear(prefix='') Delete the contents of the provider


Base Storage Provider

set_bytes(path: str, value: bytes, start_byte: Optional[int] = None, overwrite: Optional[bool] = False) Sets the object present at the path with the value Parameters path (str) – the path relative to the root of the provider .  value (bytes) – the value to be assigned at the path.  start_byte (int, optional) – If only specific bytes starting from start_byte are to be assigned.  overwrite (boolean, optional) – If the value is True, if there is an object present at the path it is completely overwritten, without fetching it's data.  Raises InvalidBytesRequestedError – If start_byte < 0.  ReadOnlyModeError – If the provider is in read-only mode.


LRU Cache

class deeplake.core.storage.LRUCache Bases: StorageProvider LRU Cache that uses StorageProvider for caching __delitem__(path: str) Deletes the object present at the path from the cache and the underlying storage.  Parameters path (str) – the path to the object relative to the root of the provider.  Raises KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str) If item is in cache_storage, retrieves from there and returns.  If item isn't in cache_storage, retrieves from next storage, stores in cache_storage (if possible) and returns.  Parameters path (str) – The path relative to the root of the underlying storage.  Raises KeyError – if an object is not found at the path.  Returns The bytes of the object present at the path.  Return type bytes __getstate__() → Dict[str, Any] Returns the state of the cache, for pickling __init__(cache_storage: StorageProvider, next_storage: Optional[StorageProvider], cache_size: int) Initializes the LRUCache.  It can be chained with other LRUCache objects to create multilayer caches.  Parameters cache_storage (StorageProvider) – The storage being used as the caching layer of the cache.  This should be a base provider such as MemoryProvider, LocalProvider or S3Provider but not another LRUCache.  next_storage (StorageProvider) – The next storage layer of the cache.  This can either be a base provider (i.e.  it is the final storage) or another LRUCache (i.e.  in case of chained cache).  While reading data, all misses from cache would be retrieved from here.  While writing data, the data will be written to the next_storage when cache_storage is full or flush is called.  cache_size (int) – The total space that can be used from the cache_storage in bytes.  This number may be less than the actual space available on the cache_storage.  Setting it to a higher value than actually available space may lead to unexpected behaviors.  __iter__() Generator function that iterates over the keys of the cache and the underlying storage.  Yields str – the path of the object that it is iterating over, relative to the root of the provider.  __len__() Returns the number of files present in the cache and the underlying storage.  Returns the number of files present inside the root.  Return type int __setitem__(path: str, value: Union[bytes, DeepLakeMemoryObject]) Puts the item in the cache_storage (if possible), else writes to next_storage.  Parameters path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises ReadOnlyError – If the provider is in read-only mode.  __setstate__(state: Dict[str, Any]) Recreates a cache with the same configuration as the state.  Parameters state (dict) – The state to be used to recreate the cache.  Note While restoring the cache, we reset its contents


deeplake.core.index

deeplake.core.index.slice_length(s: slice, parent_length: int) → int Returns the length of a slice given the length of its parent.


deeplake.core.index

__str__() Return str(self) .  __weakref__ list of weak references to the object (if defined) apply(samples: List[ndarray]) Applies an Index to a list of ndarray samples with the same number of entries as the first entry in the Index.  apply_squeeze(samples: List[ndarray]) Applies the primary axis of an Index to a list of ndarray samples.  Will either return the list as given, or return the first sample.  compose_at(item: Union[int, slice, Tuple[int, ...]], i: Optional[int] = None) Returns a new Index representing the addition of an IndexValue, or the composition with a given axis.  Examples >>> Index([slice(None), slice(None)]).compose_at(5) Index([slice(None), slice(None), 5]) Copy to clipboard >>> Index([slice(None), slice(5, 10), slice(None)]).compose_at(3, 1) Index([slice(None), 8, slice(None)]) Copy to clipboard Parameters item (IndexValue) – The value to append or compose with the Index.  i (int, optional) – The axis to compose with the given item.  Defaults to None, meaning that the item will be appended instead.  Returns The result of the addition or composition.  Return type Index downsample(factor: int, shape: Tuple[int, ...]) Downsamples an Index by the given factor.  Parameters factor (int) – The factor to downsample by.  shape (Tuple[int, ...]) – The shape of the downsampled data.  Returns The downsampled Index.  Return type Index find_axis(offset: int = 0) Returns the index for the nth subscriptable axis in the values of an Index.  Parameters offset (int) – The number of subscriptable axes to skip before returning.  Defaults to 0, meaning that the first valid axis is returned.  Returns The index of the found axis, or None if no match is found.  Return type int is_trivial() Checks if an Index is equivalent to the trivial slice [:], aka slice(None).  length(parent_length: int) Returns the primary length of an Index given the length of the parent it is indexing.  See: IndexEntry.length() validate(parent_length) Checks that the index is not accessing values outside the range of the parent.  deeplake.core.index.merge_slices(existing_slice: slice, new_slice: slice) → slice Compose two slice objects Given an iterable x, the following should be equivalent: x[existing_slice][new_slice] == x[merge_slices(existing_slice, new_slice)] Parameters existing_slice (slice) – The existing slice to be restricted.  new_slice (slice) – The new slice to be applied to the existing slice.  Returns the composition of the given slices Return type slice Raises NotImplementedError – Composing slices with negative values is not supported.  Negative indexing for slices is only supported for the first slice.  deeplake.core.index.slice_at_int(s: slice, i: int) Returns the i th element of a slice s.  Examples >>> slice_at_int(slice(None), 10) 10 Copy to clipboard >>> slice_at_int(slice(10, 20, 2), 3) 16 Copy to clipboard Parameters s (slice) – The slice to index into.  i (int) – The integer offset into the slice.  Returns The index corresponding to the offset into the slice.  Return type int Raises NotImplementedError – Nontrivial slices should not be indexed with negative integers.  IndexError – If step is negative and start is not greater than stop


deeplake.core.index

class deeplake.core.index.IndexEntry(value: Union[int, slice, Tuple[int, ...]] = slice(None, None, None)) __getitem__(item: Union[int, slice, Tuple[int, ...]]) Combines the given item and this IndexEntry.  Returns a new IndexEntry representing the composition of the two.  Examples >>> IndexEntry()[0:100] IndexEntry(slice(0, 100, None)) Copy to clipboard >>> IndexEntry()[100:200][5] IndexEntry(105) Copy to clipboard >>> IndexEntry()[(0, 1, 2, 3)] IndexEntry((0, 1, 2, 3)) Copy to clipboard >>> IndexEntry()[1, 2, 3] IndexEntry((0, 1, 2, 3)) Copy to clipboard Parameters item – The desired sub-index to be composed with this IndexEntry.  Can be an int, a slice, or a tuple of ints.  Returns The new IndexEntry object.  Return type IndexEntry Raises TypeError – An integer IndexEntry should not be indexed further.  __init__(value: Union[int, slice, Tuple[int, ...]] = slice(None, None, None)) __str__() Return str(self).  __weakref__ list of weak references to the object (if defined) downsample(factor: int, length: int) Downsamples an IndexEntry by a given factor.  Parameters factor (int) – The factor by which to downsample.  length (int) – The length of the downsampled IndexEntry.  Returns The downsampled IndexEntry.  Return type IndexEntry Raises TypeError – If the IndexEntry cannot be downsampled.  indices(length: int) Generates the sequence of integer indices for a target of a given length.  is_trivial() Checks if an IndexEntry represents the entire slice length(parent_length: int) → int Returns the length of an IndexEntry given the length of the parent it is indexing.  Examples >>> IndexEntry(slice(5, 10)).length(100) 5 >>> len(list(range(100))[5:10]) 5 >>> IndexEntry(slice(5, 100)).length(50) 45 >>> len(list(range(50))[5:100]) 45 >>> IndexEntry(0).length(10) 1 Copy to clipboard Parameters parent_length (int) – The length of the target that this IndexEntry is indexing.  Returns The length of the index if it were applied to a parent of the given length.  Return type int subscriptable() Returns whether an IndexEntry can be further subscripted.  validate(parent_length: int) Checks that the index is not accessing values outside the range of the parent.  class deeplake.core.index.Index(item: Union[int, slice, Tuple[int, ...], Index, List[IndexEntry]] = slice(None, None, None)) __getitem__(item: Union[int, slice, List[int], Tuple[Union[int, slice, Tuple[int, ...]]], Index]) Returns a new Index representing a subscripting with the given item.  Modeled after NumPy's advanced integer indexing.  See: https://numpy.org/doc/stable/reference/arrays.indexing.html Examples >>> Index([5, slice(None)])[5] Index([5, 5]) Copy to clipboard >>> Index([5])[5:6] Index([5, slice(5, 6)]) Copy to clipboard >>> Index()[0, 1, 2:5, 3] Index([0, 1, slice(2, 5), 3]) Copy to clipboard >>> Index([slice(5, 6)])[(0, 1, 2:5, 3),] Index([(5, 1, slice(2, 5), 3)]) Copy to clipboard Parameters item – The contents of the subscript expression to add to this Index.  Returns The Index representing the result of the subscript operation.  Return type Index Raises TypeError – Given item should be another Index, or compatible with NumPy's advanced integer indexing.  __init__(item: Union[int, slice, Tuple[int, ...], Index, List[IndexEntry]] = slice(None, None, None)) Initializes an Index from an IndexValue, another Index, or the values from another Index.  Represents a list of IndexEntry objects corresponding to indexes into each axis of an ndarray.  __repr__() Return repr(self)


How Shuffling Works in Deep Lake's PyTorch DataLoader

The Deep Lake shuffling algorithm is based upon a shuffle buffer that preloads a specified amount of data (in MB) determined by the buffer_size parameter in ds.pytorch(buffer_size = 2048). First, the dataloader randomly selects chunks from the applicable tensors until the shuffle buffer is full. Next, the indices in shuffle buffer are randomly sampled to construct the batches that are returned by the dataloader. As the data in the shuffle buffer is consumed, new chunks are randomly selected and added to the buffer.In the OSS dataloader, the shuffle buffer contains the decompressed, decoded, and transformed samples. When using the PyTorch dataloaders, this corresponds to torch tensors. In the Performant dataloader, the shuffle buffer contains the non-decompressed data in the format they are stored in. For images, this typically corresponds to compressed bytes in jpeg, png, or other compressions. Since compressed data is stored more efficiently than uncompressed data, there are typically more distinct samples of data in the Performant dataloader shuffle buffer compared to the OSS shuffle buffer. If many chunks in the buffer contain data from the same class, which may occur if data was uploaded in non-random order, the shuffle buffer may contain fewer unique classes than if the samples were chosen fully randomly based on index. The most extreme case of reduced randomness occurs when datasets are much larger than the shuffle buffer, when they have many classes, and when those classes occur in sequence within the dataset indices. One example dataset is Unshuffled ImageNet, which has 1000 classes, 1.2M images, 140GB of data, and approximately 140 images per 16MB chunk. When the images are uploaded in sequence, the plot below shows how many unique classes are returned by the loader vs the number of images that have been returned in total. It is evident that fully randomly sampling returns more unique values than the Deep Lake dataloader. If reduced randomness has an impact on model performance in your workflows, the recommended countermeasures are:Store the dataset in a shuffled fashion such that the data does not appear in order by class. This completely mitigates the randomness concerns at the output of the data loader.Store the dataset with a smaller chunk size. This increases randomness because the shuffle buffer selects more discreet chunks before filling up. The current default size is 8, and reducing chunk size to 4MB significantly increases randomness (see plot above) with only a modest slowdown in data transfer speed.Increase the size of the shuffle buffer. This mitigates the randomness concerns but may not completely alleviate them.


deeplake.api.dataset

static ingest_dataframe(src, dest: Union[str, pathlib.Path], column_params: Optional[Dict] = None, src_creds: Optional[Dict] = None, dest_creds: Optional[Dict] = None, creds_key: Optional[Dict] = None, progressbar: bool = True, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) See deeplake.ingest_dataframe() .  static list(workspace: str = '', token: Optional[str] = None) → None See deeplake.list().


deeplake.api.dataset

class deeplake.api.dataset.dataset static exists(path: Union[str, pathlib.Path], creds: Optional[dict] = None, token: Optional[str] = None) → bool See deeplake.exists().  static empty(path: Union[str, pathlib.Path], overwrite: bool = False, public: bool = False, memory_cache_size: int = DEFAULT_MEMORY_CACHE_SIZE, local_cache_size: int = DEFAULT_LOCAL_CACHE_SIZE, creds: Optional[dict] = None, token: Optional[str] = None) → Dataset See deeplake.empty().  static load(path: Union[str, pathlib.Path], read_only: Optional[bool] = None, memory_cache_size: int = DEFAULT_MEMORY_CACHE_SIZE, local_cache_size: int = DEFAULT_LOCAL_CACHE_SIZE, creds: Optional[dict] = None, token: Optional[str] = None, verbose: bool = True, access_method: str = 'stream') → Dataset See deeplake.load().  static rename(old_path: Union[str, pathlib.Path], new_path: Union[str, pathlib.Path], creds: Optional[dict] = None, token: Optional[str] = None) → Dataset See deeplake.rename().  static delete(path: Union[str, pathlib.Path], force: bool = False, large_ok: bool = False, creds: Optional[dict] = None, token: Optional[str] = None, verbose: bool = False) → None See deeplake.delete().  static like(dest: Union[str, pathlib.Path], src: Union[str, Dataset, pathlib.Path], tensors: Optional[List[str]] = None, overwrite: bool = False, creds: Optional[dict] = None, token: Optional[str] = None, public: bool = False) → Dataset See deeplake.like().  static copy(src: Union[str, pathlib.Path, Dataset], dest: Union[str, pathlib.Path], tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, dest_creds=None, token=None, num_workers: int = 0, scheduler='threaded', progressbar=True) See deeplake.copy().  static deepcopy(src: Union[str, pathlib.Path], dest: Union[str, pathlib.Path], tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, dest_creds=None, token=None, num_workers: int = 0, scheduler='threaded', progressbar=True, public: bool = False, verbose: bool = True) See deeplake.deepcopy().  static connect(src_path: str, creds_key: str, dest_path: Optional[str], org_id: Optional[str], ds_name: Optional[str], token: Optional[str]) See deeplake.connect().  static ingest_classification(src: Union[str, pathlib.Path], dest: Union[str, pathlib.Path], image_params: Optional[Dict] = None, label_params: Optional[Dict]: None, dest_creds: Optional[Dict] = None, progressbar: bool = True, summary: bool = True, num_workers: int = 0, shuffle: bool = False, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) → Dataset See deeplake.ingest_classification().  static ingest_coco(images_directory: Union[str, pathlib.Path], annotation_files: Union[str, pathlib.Path, List[str]], dest: Union[str, pathlib.Path], key_to_tensor_mapping: Optional[Dict] = None, file_to_group_mapping: Optional[Dict] = None, ignore_one_group: bool = False, ignore_keys: Optional[List[str]] = None, image_settings: Optional[Dict] = None, src_creds: Optional[Dict] = None, dest_creds: Optional[Dict] = None, inspect_limit: int = 1000000, progressbar: bool = True, shuffle: bool = False, num_workers: int = 0, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) → Dataset See deeplake.ingest_coco().  def ingest_yolo(data_directory: Union[str, pathlib.Path], dest: Union[str, pathlib.Path], class_names_file: Optional[Union[str, pathlib.Path]] = None, annotations_directory: Optional[Union[str, pathlib.Path]] = None, allow_no_annotation: bool = False, image_params: Optional[Dict] = None, label_params: Optional[Dict] = None, coordinates_params: Optional[Dict] = None, src_creds: Optional[Dict] = None, dest_creds: Optional[Dict] = None, image_creds_key: Optional[str] = None, inspect_limit: int = 1000, progressbar: bool = True, shuffle: bool = False, num_workers: int = 0, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) -> Dataset: See deeplake.ingest_yolo() static ingest_kaggle(tag: str, src: Union[str, pathlib.Path], dest: Union[str, pathlib.Path], exist_ok: bool = False, images_compression: str = 'auto', dest_creds: dict = None, kaggle_credentials: dict = None, progressbar: bool = True, summary: bool = True, **dataset_kwargs) → Dataset See deeplake.ingest_kaggle()


deeplake.api.info

class deeplake.api.info.Info Contains optional key-value pairs that can be stored for datasets/tensors. clear() Clear info. get(key, default=None) Get value for key from info. items() Return all items in info. keys() Return all keys in info. property nbytes Returns size of info stored in bytes. pop(key, default=None) Pop item from info by key. popitem() Pop item from info. replace_with(d) Replace info with another dictionary. setdefault(key, default=None) Set default value for a key in info. update(*args, **kwargs) Update info. values() Return all values in info.


deeplake.api.link

deeplake.api.link.link(path: str, creds_key: Optional[str] = None) → LinkedSample See deeplake.link().


deeplake.api.read

deeplake.api.read.read(path: str, verify: bool = False, creds: Optional[Dict] = None, compression: Optional[str] = None, storage: Optional[StorageProvider] = None) → Sample See deeplake.read().


deeplake.api.tiled

deeplake.api.tiled.tiled(sample_shape: Tuple[int, ...], tile_shape: Optional[Tuple[int, ...]] = None, dtype: Union[str, np.dtype] = np.dtype('uint8')) See deeplake.tiled().


deeplake.api.link_tiled

deeplake.api.link_tiled.link_tiled(path_array: np.ndarray, creds_key: Optional[str] = None) → LinkedTiledSample: See deeplake.link_tiled().


deeplake.auto.structured

deeplake.auto.structured.base StructuredDataset deeplake.auto.structured.dataframe DataFrame


deeplake.auto.unstructured

deeplake.auto.unstructured.base UnstructuredDataset deeplake.auto.unstructured.image_classification ImageClassification deeplake.auto.unstructured.kaggle download_kaggle_dataset()


deeplake.auto.structured.base

class deeplake.auto.structured.base.StructuredDataset(source) Initializes a structured dataset. Parameters source (str) – The local path to folder or file containing a structured dataset and of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.


deeplake.auto.structured.dataframe

class deeplake.auto.structured.dataframe.DataFrame(source, column_params=None, creds=None, creds_key=None) fill_dataset(ds: Dataset, progressbar: bool = True) → Dataset Fill dataset with data from the dataframe - one tensor per column Parameters ds (Dataset) – A Deep Lake dataset object. progressbar (bool) – Defines if the method uses a progress bar. Defaults to True. Returns A Deep Lake dataset.


deeplake.auto.unstructured.base

class deeplake.auto.unstructured.base.UnstructuredDataset(source: str)


deeplake.auto.unstructured.image_classification

class deeplake.auto.unstructured.image_classification.ImageClassification(source: str) structure(ds: Dataset, progressbar: bool = True, generate_summary: bool = True, shuffle: bool = True, image_tensor_args: dict = {}, label_tensor_args: dict = {}, num_workers: int = 0) → Dataset Create a structured dataset. Parameters ds (Dataset) – A Deep Lake dataset object. progressbar (bool) – Defines if the method uses a progress bar. Defaults to True. generate_summary (bool) – Defines if the method generates ingestion summary. Defaults to True. shuffle (bool) – Defines if the file paths should be shuffled prior to ingestion. Defaults to True. image_tensor_args (dict) – Defines the parameters for the images tensor. label_tensor_args (dict) – Defines the parameters for the class_labels tensor. num_workers (int) – The number of workers passed to compute. Returns A Deep Lake dataset.


deeplake.auto.unstructured.kaggle

deeplake.auto.unstructured.kaggle.download_kaggle_dataset(tag: str, local_path: str, kaggle_credentials: Optional[dict] = None, exist_ok: bool = False) Calls the kaggle API (https://www.kaggle.com/docs/api) to download a kaggle dataset and unzip it's contents. Parameters tag (str) – Kaggle dataset tag. Example: 'coloradokb/dandelionimages' points to https://www.kaggle.com/coloradokb/dandelionimages local_path (str) – Path where the kaggle dataset will be downloaded and unzipped. Only local path downloading is supported. kaggle_credentials (dict) – Credentials are gathered from the environment variables or ~/kaggle.json. If those don't exist, the kaggle_credentials argument will be used. exist_ok (bool) – If the kaggle dataset was already downloaded, and exist_ok is True, no error is thrown. Raises KaggleMissingCredentialsError – If no kaggle credentials are found. KaggleDatasetAlreadyDownloadedError – If the dataset tag already exists in local_path.


deeplake.util.shape_interval

class deeplake.util.shape_interval.ShapeInterval __init__(lower: Sequence[int], upper: Optional[Sequence[int]] = None) Shapes in Deep Lake are best represented as intervals, this is to support dynamic tensors. Instead of having a single tuple of integers representing shape, we use 2 tuples of integers to represent the lower and upper bounds of the representing shape. If lower == upper for all cases, the shape is considered 'fixed'. If lower != upper for any cases, the shape is considered 'dynamic'. Parameters lower (sequence) – Sequence of integers that represent the lower-bound shape. upper (sequence) – Sequence of integers that represent the upper-bound shape. If None is provided, lower is used as upper (implicitly fixed-shape). Raises InvalidShapeIntervalError – If the provided lower/upper bounds are incompatible to represent a shape.


deeplake.util.remove_cache

deeplake.util.remove_cache.remove_memory_cache(storage: StorageProvider) Removes the memory cache. deeplake.util.remove_cache.get_base_storage(storage: StorageProvider) Removes all layers of caching and returns the underlying storage. deeplake.util.remove_cache.get_dataset_with_zero_size_cache(ds) Returns a dataset with same storage but cache size set to zero. deeplake.util.remove_cache.create_read_copy_dataset(dataset, commit_id: Optional[str] = None) Creates a read-only copy of the given dataset object, without copying underlying data. Parameters dataset – The Dataset object to copy. commit_id – The commit id to checkout the new read-only copy to. Returns A new Dataset object in read-only mode.


deeplake.util.notebook

deeplake.util.notebook.is_notebook() Whether running in a notebook. deeplake.util.notebook.is_jupyter() Whether running in a Jupyter notebook. deeplake.util.notebook.is_colab() Whether running in a colab notebook.


deeplake.util.exceptions

class deeplake.util.exceptions.ExternalCommandError(command: str, status: int) Bases: Exception class deeplake.util.exceptions.KaggleError Bases: Exception class deeplake.util.exceptions.KaggleMissingCredentialsError(env_var_name: str) Bases: KaggleError class deeplake.util.exceptions.KaggleDatasetAlreadyDownloadedError(tag: str, path: str) Bases: KaggleError class deeplake.util.exceptions.InvalidPathException(directory) Bases: Exception class deeplake.util.exceptions.AutoCompressionError(directory) Bases: Exception class deeplake.util.exceptions.InvalidFileExtension(directory) Bases: Exception class deeplake.util.exceptions.SamePathException(directory) Bases: Exception class deeplake.util.exceptions.TensorInvalidSampleShapeError(shape: Sequence[int], expected_dims: int) Bases: Exception class deeplake.util.exceptions.TensorMetaMissingKey(key: str, meta: dict) Bases: Exception class deeplake.util.exceptions.TensorDoesNotExistError(tensor_name: str) Bases: KeyError, AttributeError class deeplake.util.exceptions.TensorAlreadyExistsError(key: str) Bases: Exception class deeplake.util.exceptions.TensorGroupDoesNotExistError(group_name: str) Bases: KeyError class deeplake.util.exceptions.TensorGroupAlreadyExistsError(key: str) Bases: Exception class deeplake.util.exceptions.InvalidTensorNameError(name: str) Bases: Exception class deeplake.util.exceptions.InvalidTensorGroupNameError(name: str) Bases: Exception class deeplake.util.exceptions.DynamicTensorNumpyError(key: str, index, property_key: str) Bases: Exception class deeplake.util.exceptions.InvalidShapeIntervalError(message: str, lower: Optional[Sequence[int]] = None, upper: Optional[Sequence[int]] = None) Bases: Exception class deeplake.util.exceptions.InvalidKeyTypeError(item: Any) Bases: TypeError class deeplake.util.exceptions.UnsupportedTensorTypeError(item: Any) Bases: TypeError class deeplake.util.exceptions.InvalidBytesRequestedError Bases: Exception class deeplake.util.exceptions.ProviderListEmptyError Bases: Exception class deeplake.util.exceptions.DirectoryAtPathException Bases: Exception class deeplake.util.exceptions.FileAtPathException(path) Bases: Exception class deeplake.util.exceptions.ProviderSizeListMismatch Bases: Exception class deeplake.util.exceptions.ModuleNotInstalledException(message) Bases: Exception class deeplake.util.exceptions.LoginException(message='Error while logging in, invalid auth token.  Please try logging in again.') Bases: Exception class deeplake.util.exceptions.UserNotLoggedInException Bases: Exception class deeplake.util.exceptions.InvalidHubPathException(path) Bases: Exception class deeplake.util.exceptions.PathNotEmptyException(use_hub=True) Bases: Exception class deeplake.util.exceptions.AuthenticationException(message='Authentication failed.  Please try logging in again.') Bases: Exception class deeplake.util.exceptions.AuthorizationException(message='You are not authorized to access this resource on Activeloop Server.', response=None) Bases: Exception class deeplake.util.exceptions.InvalidPasswordException(message='The password you provided was invalid.') Bases: AuthorizationException class deeplake.util.exceptions.CouldNotCreateNewDatasetException(path: str) Bases: AuthorizationException class deeplake.util.exceptions.ResourceNotFoundException(message='The resource you are looking for was not found


deeplake.util.exceptions

Check if the name or id is correct.') Bases: Exception class deeplake.util.exceptions.BadRequestException(message) Bases: Exception class deeplake.util.exceptions.OverLimitException(message='You are over the allowed limits for this operation.') Bases: Exception class deeplake.util.exceptions.ServerException(message='Internal Activeloop server error.') Bases: Exception class deeplake.util.exceptions.BadGatewayException(message='Invalid response from Activeloop server.') Bases: Exception class deeplake.util.exceptions.GatewayTimeoutException(message='Activeloop server took too long to respond.') Bases: Exception class deeplake.util.exceptions.WaitTimeoutException(message='Timeout waiting for server state update.') Bases: Exception class deeplake.util.exceptions.LockedException(message='The resource is currently locked.') Bases: Exception class deeplake.util.exceptions.UnexpectedStatusCodeException(message) Bases: Exception class deeplake.util.exceptions.EmptyTokenException(message='The authentication token is empty.') Bases: Exception class deeplake.util.exceptions.S3Error Bases: Exception Catchall for all errors encountered while working with S3 class deeplake.util.exceptions.S3GetError Bases: S3Error Catchall for all errors encountered while working getting an object from S3 class deeplake.util.exceptions.S3SetError Bases: S3Error Catchall for all errors encountered while working setting an object in S3 class deeplake.util.exceptions.S3DeletionError Bases: S3Error Catchall for all errors encountered while working deleting an object in S3 class deeplake.util.exceptions.S3ListError Bases: S3Error Catchall for all errors encountered while retrieving a list of objects present in S3 class deeplake.util.exceptions.UnsupportedCompressionError(compression: Optional[str], htype: Optional[str] = None) Bases: CompressionError class deeplake.util.exceptions.SampleCompressionError(sample_shape: Tuple[int, ...], compression_format: Optional[str], message: str) Bases: CompressionError class deeplake.util.exceptions.SampleDecompressionError(path: Optional[str] = None) Bases: CompressionError class deeplake.util.exceptions.InvalidImageDimensions(actual_dims, expected_dims) Bases: Exception class deeplake.util.exceptions.TensorUnsupportedSampleType Bases: Exception class deeplake.util.exceptions.MetaError Bases: Exception class deeplake.util.exceptions.MetaDoesNotExistError(key: str) Bases: MetaError class deeplake.util.exceptions.MetaAlreadyExistsError(key: str, required_meta: dict) Bases: MetaError class deeplake.util.exceptions.MetaInvalidKey(name: str, available_keys: List[str]) Bases: MetaError class deeplake.util.exceptions.MetaInvalidRequiredMetaKey(key: str, subclass_name: str) Bases: MetaError class deeplake.util.exceptions.TensorMetaInvalidHtype(htype: str, available_htypes: Sequence[str]) Bases: MetaError class deeplake.util.exceptions.TensorMetaInvalidHtypeOverwriteValue(key: str, value: Any, explanation: str = '') Bases: MetaError class deeplake.util.exceptions.TensorMetaMissingRequiredValue(htype: str, key: Union[str, List[str]]) Bases: MetaError class deeplake.util.exceptions.TensorMetaInvalidHtypeOverwriteKey(htype: str, key: str, available_keys: Sequence[str]) Bases: MetaError class deeplake.util.exceptions.TensorDtypeMismatchError(expected: Union[dtype, str], actual: str, htype: str) Bases: MetaError class deeplake.util.exceptions.InvalidTensorLinkError(msg='Invalid tensor link.') Bases: MetaError class deeplake.util.exceptions.TensorMetaMutuallyExclusiveKeysError(keys: Optional[List[str]] = None, custom_message: Optional[str] = None) Bases: MetaError class deeplake.util.exceptions.ReadOnlyModeError(custom_message: Optional[str] = None) Bases: Exception class deeplake.util.exceptions.TransformError(index=None, sample=None, samples_processed=0, suggest=False) Bases: Exception class deeplake.util.exceptions.FilterError Bases: Exception class deeplake.util.exceptions.InvalidInputDataError(operation) Bases: TransformError class deeplake.util.exceptions.UnsupportedSchedulerError(scheduler) Bases: TransformError class deeplake.util.exceptions.TensorMismatchError(tensors, output_keys, skip_ok=False) Bases: TransformError class deeplake.util.exceptions.InvalidOutputDatasetError(message='The output Dataset to transform should not be `read_only`.') Bases: TransformError class deeplake.util.exceptions.InvalidTransformDataset(message='The TransformDataset (2nd argument to transform function) of one of the functions is invalid


deeplake.util.exceptions

All the tensors should have equal length for it to be valid.') Bases: TransformError class deeplake.util.exceptions.HubComposeEmptyListError(message='Cannot deeplake.compose an empty list.') Bases: TransformError class deeplake.util.exceptions.HubComposeIncompatibleFunction(index: int) Bases: TransformError class deeplake.util.exceptions.DatasetUnsupportedPytorch(reason) Bases: Exception class deeplake.util.exceptions.CorruptedMetaError Bases: Exception class deeplake.util.exceptions.ChunkEngineError Bases: Exception class deeplake.util.exceptions.FullChunkError Bases: ChunkEngineError class deeplake.util.exceptions.ChunkIdEncoderError Bases: ChunkEngineError class deeplake.util.exceptions.ChunkSizeTooSmallError(message='If the size of the last chunk is given, it must be smaller than the requested chunk size.') Bases: ChunkEngineError class deeplake.util.exceptions.DatasetHandlerError(message) Bases: Exception class deeplake.util.exceptions.MemoryDatasetCanNotBePickledError Bases: Exception class deeplake.util.exceptions.CorruptedSampleError(compression, path: Optional[str] = None) Bases: Exception class deeplake.util.exceptions.VersionControlError Bases: Exception class deeplake.util.exceptions.MergeError Bases: Exception class deeplake.util.exceptions.MergeNotSupportedError Bases: MergeError class deeplake.util.exceptions.MergeMismatchError(tensor_name, mismatch_type, original_value, target_value) Bases: MergeError class deeplake.util.exceptions.MergeConflictError(conflict_tensors=None, message='') Bases: MergeError class deeplake.util.exceptions.CheckoutError Bases: VersionControlError class deeplake.util.exceptions.CommitError Bases: VersionControlError class deeplake.util.exceptions.EmptyCommitError Bases: CommitError class deeplake.util.exceptions.TensorModifiedError Bases: Exception class deeplake.util.exceptions.GCSDefaultCredsNotFoundError Bases: Exception class deeplake.util.exceptions.InvalidOperationError(method: str, type: str) Bases: Exception class deeplake.util.exceptions.AgreementError Bases: Exception class deeplake.util.exceptions.AgreementNotAcceptedError(agreements=None) Bases: AgreementError class deeplake.util.exceptions.NotLoggedInError(msg=None) Bases: AgreementError class deeplake.util.exceptions.RenameError(msg='Only name of the dataset can be different in new path.') Bases: Exception class deeplake.util.exceptions.BufferError Bases: Exception class deeplake.util.exceptions.InfoError Bases: Exception class deeplake.util.exceptions.OutOfChunkCountError Bases: Exception class deeplake.util.exceptions.OutOfSampleCountError Bases: Exception class deeplake.util.exceptions.SampleHtypeMismatchError(htype, sample_type) Bases: Exception class deeplake.util.exceptions.EmptyTensorError(message) Bases: Exception class deeplake.util.exceptions.DatasetViewSavingError Bases: Exception class deeplake.util.exceptions.ManagedCredentialsNotFoundError(org_id, creds_key) Bases: Exception class deeplake.util.exceptions.UnableToReadFromUrlError(url, status_code) Bases: Exception class deeplake.util.exceptions.InvalidTokenException Bases: Exception class deeplake.util.exceptions.TokenPermissionError(message=None) Bases: Exception


Installation¶

Deep Lake can be installed with pip Copy to clipboard Deep Lake has the following extras that you can choose to install according to your needs. Installation commands¶ Install command Description Dependencies installed pip install 'deeplake[av]' Audio and video support via PyAV av pip install 'deeplake[visualizer]' Visualize Deep Lake datasets within notebooks. This is required for Dataset.visualize to work. IPython, flask pip install 'deeplake[gcp]' GCS support google-cloud-storage, google-auth, google-auth-oauthlib pip install 'deeplake[dicom]' DICOM data support pydicom pip install 'deeplake[gdrive]' Google Drive support google-api-python-client, oauth2client, google-auth, google-auth-oauthlib pip install 'deeplake[point_cloud]' Support for LiDAR point cloud data laspy pip install 'deeplake[enterprise]' Installs Deep Lake's enterprise features libdeeplake pip install 'deeplake[all]' Installs all of the above