deeplake.core.storage¶

All read/write request keys will be appended to root .  aws_access_key_id (str, optional) – Specifies the AWS access key used as part of the credentials to authenticate the user.  aws_secret_access_key (str, optional) – Specifies the AWS secret key used as part of the credentials to authenticate the user.  aws_session_token (str, optional) – Specifies an AWS session token used as part of the credentials to authenticate the user.  endpoint_url (str, optional) – The complete URL to use for the constructed client.  This needs to be provided for cases in which you're interacting with MinIO, Wasabi, etc.  aws_region (str, optional) – Specifies the AWS Region to send requests to.  profile_name (str, optional) – Specifies the AWS profile name to use.  token (str, optional) – Activeloop token, used for fetching credentials for Deep Lake datasets (if this is underlying storage for Deep Lake dataset).  This is optional, tokens are normally autogenerated.  __iter__()¶ Generator function that iterates over the keys of the S3Provider.  Yields: str – the name of the object that it is iterating over.  __len__()¶ Returns the number of files present at the root of the S3Provider.  Note This is an expensive operation.  Returns: the number of files present inside the root.  Return type: int Raises: S3ListError – Any S3 error encountered while listing the objects.  __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the S3Provider.  content (bytes) – the value to be assigned at the path.  Raises: S3SetError – Any S3 error encountered while setting the value at the path.  ReadOnlyError – If the provider is in read-only mode.  _all_keys()¶ Helper function that lists all the objects present at the root of the S3Provider.  Returns: set of all the objects found at the root of the S3Provider.  Return type: set Raises: S3ListError – Any S3 error encountered while listing the objects.  _check_update_creds(force=False)¶ If the client has an expiration time, check if creds are expired and fetch new ones.  This would only happen for datasets stored on Deep Lake storage for which temporary 12 hour credentials are generated.  _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters: hub_path (str) – The Deep Lake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  _state_keys()¶ Keys used to store the state of the provider.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the s3 bucket (under self.root).  Warning Exercise caution! get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range


deeplake.core.storage¶

get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range .  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  get_deeplake_object(path: str, expected_class, meta: Dict | None = None, url=False, partial_bytes: int = 0)¶ If the data at path was stored using the output of a DeepLakeMemoryObject's tobytes function, this function will read it back into object form & keep the object in cache.  Parameters: path (str) – Path to the stored object.  expected_class (callable) – The expected subclass of DeepLakeMemoryObject.  meta (dict, optional) – Metadata associated with the stored object url (bool) – Get presigned url instead of downloading chunk (only for videos) partial_bytes (int) – Number of bytes to read from the beginning of the file.  If 0, reads the whole file.  Defaults to 0.  Raises: ValueError – If the incorrect expected_class was provided.  ValueError – If the type of the data at path is invalid.  ValueError – If url is True but expected_class is not a subclass of BaseChunk.  Returns: An instance of expected_class populated with the data.  register_deeplake_object(path: str, obj: DeepLakeMemoryObject)¶ Registers a new object in the cache.  remove_deeplake_object(path: str)¶ Removes a DeepLakeMemoryObject from the cache.  S3 Storage Provider¶ class deeplake.core.storage.S3Provider¶ Bases: StorageProvider Provider class for using S3 storage.  __delitem__(path)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the S3Provider.  Note If the object is not found, s3 won't raise KeyError.  Raises: S3DeletionError – Any S3 error encountered while deleting the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path)¶ Gets the object present at the path.  Parameters: path (str) – the path relative to the root of the S3Provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  __init__(root: str, aws_access_key_id: str | None = None, aws_secret_access_key: str | None = None, aws_session_token: str | None = None, endpoint_url: str | None = None, aws_region: str | None = None, profile_name: str | None = None, token: str | None = None)¶ Initializes the S3Provider Example >>> s3_provider = S3Provider('snark-test/benchmarks') Copy to clipboard Parameters: root (str) – The root of the provider


deeplake.core.storage¶

DirectoryAtPathException – If a directory is found at the path .  Exception – Any other exception encountered while trying to fetch the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> my_data = local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  __init__(root: str)¶ Initializes the LocalProvider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.' Raises: FileAtPathException – If the root is a file instead of a directory.  __iter__()¶ Generator function that iterates over the keys of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> for my_data in local_provider: ...  pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> len(local_provider) Copy to clipboard Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> local_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises: Exception – If unable to set item due to directory at path or permission or space issues.  FileAtPathException – If the directory to the path is a file instead of a directory.  ReadOnlyError – If the provider is in read-only mode.  _all_keys(refresh: bool = False) → Set[str]¶ Lists all the objects present at the root of the Provider.  Parameters: refresh (bool) – refresh keys Returns: set of all the objects found at the root of the Provider.  Return type: set _check_is_file(path: str)¶ Checks if the path is a file.  Returns the full_path to file if True.  Parameters: path (str) – the path to the object relative to the root of the provider.  Returns: the full path to the requested file.  Return type: str Raises: DirectoryAtPathException – If a directory is found at the path.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the local machine (under self.root).  Exercise caution! get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range


deeplake.core.storage¶

Returns: The bytes of the object present at the path within the given byte range .  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(root)¶ Rename root folder.  Google Drive Storage Provider¶ class deeplake.core.storage.GDriveProvider¶ Bases: StorageProvider Provider class for using Google Drive storage.  __delitem__(path)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  __getitem__(path)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  __init__(root: str, token: str | Dict | None = None, makemap: bool = True)¶ Initializes the GDriveProvider Example >>> gdrive_provider = GDriveProvider('gdrive://folder_name/folder_name') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.  token (dict, str, optional) – Google Drive token.  Can be path to the token file or the actual credentials dictionary.  makemap (bool) – Creates path to id map if True.  Note Requires client_secrets.json in working directory if token is not provided.  Due to limits on requests per 100 seconds on google drive api, continuous requests such as uploading many small files can be slow.  Users can request to increse their quotas on their google cloud platform.  __iter__()¶ Generator function that iterates over the keys of the provider.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Returns: the number of files present inside the root.  Return type: int __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  _all_keys()¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set clear(prefix='')¶ Delete the contents of the provider.  sync()¶ Sync provider keys with actual storage Local Storage Provider¶ class deeplake.core.storage.LocalProvider¶ Bases: StorageProvider Provider class for using the local filesystem.  __delitem__(path: str)¶ Delete the object present at the path.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> del local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path


deeplake.core.storage¶

Parameters: path (str) – The path relative to the root of the provider .  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  need_to_reload_creds(err: ClientError) → bool¶ Checks if the credentials need to be reloaded.  This happens if the credentials were loaded from the environment and have now expired.  rename(root)¶ Rename root folder.  Google Cloud Storage Provider¶ class deeplake.core.storage.GCSProvider¶ Bases: StorageProvider Provider class for using GC storage.  __contains__(key)¶ Checks if key exists in mapping.  __delitem__(key)¶ Remove key.  __getitem__(key)¶ Retrieve data.  __init__(root: str, token: str | Dict | None = None, project: str | None = None)¶ Initializes the GCSProvider.  Example >>> gcs_provider = GCSProvider('gcs://my-bucket/gcs_ds') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.  token (str/Dict) – GCP token, used for fetching credentials for storage).  Can be a path to the credentials file, actual credential dictionary or one of the folowing: - google_default: Tries to load default credentials for the specified project.  - cache: Retrieves the previously used credentials from cache if exist.  - anon: Sets credentials=None.  - browser: Generates and stores new token file using cli.  project (str) – Name of the project from GCloud.  Raises: ModuleNotFoundError – If google cloud packages aren't installed.  __iter__()¶ Iterating over the structure.  __len__()¶ Returns length of the structure.  __setitem__(key, value)¶ Store value in key.  _all_keys()¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters: hub_path (str) – The deeplake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  clear(prefix='')¶ Remove all keys with given prefix below root - empties out mapping.  Warning Exercise caution! get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required


deeplake.core.storage¶

Parameters: path (str) – The path relative to the root of the provider .  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(path)¶ Renames root folder Memory Provider¶ class deeplake.core.storage.MemoryProvider¶ Bases: StorageProvider Provider class for using the memory.  __delitem__(path: str)¶ Delete the object present at the path.  Example >>> memory_provider = MemoryProvider('xyz') >>> del memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Example >>> memory_provider = MemoryProvider('xyz') >>> my_data = memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  __getstate__() → str¶ Does NOT save the in memory data in state.  __init__(root: str = '')¶ __iter__()¶ Generator function that iterates over the keys of the provider.  Example >>> memory_provider = MemoryProvider('xyz') >>> for my_data in memory_provider: ...  pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Example >>> memory_provider = MemoryProvider('xyz') >>> len(memory_provider) Copy to clipboard Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> memory_provider = MemoryProvider('xyz') >>> memory_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  _all_keys()¶ Lists all the objects present at the root of the Provider.  Returns: set of all the objects found at the root of the Provider.  Return type: set clear(prefix='')¶ Clears the provider.


Base Storage Provider¶

class deeplake.core.storage.StorageProvider¶ abstract __delitem__(path: str)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  abstract __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  abstract __iter__()¶ Generator function that iterates over the keys of the provider.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  abstract __len__()¶ Returns the number of files present inside the root of the provider.  Returns: the number of files present inside the root.  Return type: int abstract __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  __weakref__¶ list of weak references to the object (if defined) abstract _all_keys() → Set[str]¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set _is_hub_path = False¶ An abstract base class for implementing a storage provider.  To add a new provider using Provider, create a subclass and implement all 5 abstract methods below.  check_readonly()¶ Raises an exception if the provider is in read-only mode.  abstract clear(prefix='')¶ Delete the contents of the provider.  copy()¶ Returns a copy of the provider.  Returns: A copy of the provider.  Return type: StorageProvider disable_readonly()¶ Disables read-only mode for the provider.  enable_readonly()¶ Enables read-only mode for the provider.  flush()¶ Only needs to be implemented for caches.  Flushes the data to the next storage provider.  Should be a no op for Base Storage Providers like local, s3, azure, gcs, etc.  get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  maybe_flush()¶ Flush cache if autoflush has been enabled


LRU Cache¶

class deeplake.core.storage.LRUCache¶ Bases: StorageProvider LRU Cache that uses StorageProvider for caching __delitem__(path: str)¶ Deletes the object present at the path from the cache and the underlying storage.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ If item is in cache_storage, retrieves from there and returns.  If item isn't in cache_storage, retrieves from next storage, stores in cache_storage (if possible) and returns.  Parameters: path (str) – The path relative to the root of the underlying storage.  Raises: KeyError – if an object is not found at the path.  Returns: The bytes of the object present at the path.  Return type: bytes __getstate__() → Dict[str, Any]¶ Returns the state of the cache, for pickling __init__(cache_storage: StorageProvider, next_storage: StorageProvider | None, cache_size: int)¶ Initializes the LRUCache.  It can be chained with other LRUCache objects to create multilayer caches.  Parameters: cache_storage (StorageProvider) – The storage being used as the caching layer of the cache.  This should be a base provider such as MemoryProvider, LocalProvider or S3Provider but not another LRUCache.  next_storage (StorageProvider) – The next storage layer of the cache.  This can either be a base provider (i.e.  it is the final storage) or another LRUCache (i.e.  in case of chained cache).  While reading data, all misses from cache would be retrieved from here.  While writing data, the data will be written to the next_storage when cache_storage is full or flush is called.  cache_size (int) – The total space that can be used from the cache_storage in bytes.  This number may be less than the actual space available on the cache_storage.  Setting it to a higher value than actually available space may lead to unexpected behaviors.  __iter__()¶ Generator function that iterates over the keys of the cache and the underlying storage.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present in the cache and the underlying storage.  Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes | DeepLakeMemoryObject)¶ Puts the item in the cache_storage (if possible), else writes to next_storage.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  __setstate__(state: Dict[str, Any])¶ Recreates a cache with the same configuration as the state.  Parameters: state (dict) – The state to be used to recreate the cache.  Note While restoring the cache, we reset its contents


LRU Cache¶

In case the cache storage was local/s3 and is still accessible when unpickled (if same machine/s3 creds present respectively), the earlier cache contents are no longer accessible .  _all_keys()¶ Helper function that lists all the objects present in the cache and the underlying storage.  Returns: set of all the objects found in the cache and the underlying storage.  Return type: set _flush_if_not_read_only()¶ Flushes the cache if not in read-only mode.  _forward(path)¶ Forward the value at a given path to the next storage, and un-marks its key.  _forward_value(path, value)¶ Forwards a path-value pair to the next storage, and un-marks its key.  Parameters: path (str) – the path to the object relative to the root of the provider.  value (bytes, DeepLakeMemoryObject) – the value to send to the next storage.  _free_up_space(extra_size: int)¶ Helper function that frees up space the requred space in cache.No action is taken if there is sufficient space in the cache.  Parameters: extra_size (int) – the space that needs is required in bytes.  _insert_in_cache(path: str, value: bytes | DeepLakeMemoryObject)¶ Helper function that adds a key value pair to the cache.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  _pop_from_cache()¶ Helper function that pops the least recently used key, value pair from the cache clear(prefix='')¶ Deletes ALL the data from all the layers of the cache and the actual storage.  This is an IRREVERSIBLE operation.  Data once deleted can not be recovered.  clear_cache()¶ Flushes the content of all the cache layers if not in read mode and and then deletes contents of all the layers of it.  This doesn't delete data from the actual storage.  clear_deeplake_objects()¶ Removes all DeepLakeMemoryObjects from the cache.  flush()¶ Writes data from cache_storage to next_storage.  Only the dirty keys are written.  This is a cascading function and leads to data being written to the final storage in case of a chained cache.  get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path


S3 Storage Provider¶

class deeplake.core.storage.S3Provider¶ Bases: StorageProvider Provider class for using S3 storage.  __delitem__(path)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the S3Provider.  Note If the object is not found, s3 won't raise KeyError.  Raises: S3DeletionError – Any S3 error encountered while deleting the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path)¶ Gets the object present at the path.  Parameters: path (str) – the path relative to the root of the S3Provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  __init__(root: str, aws_access_key_id: str | None = None, aws_secret_access_key: str | None = None, aws_session_token: str | None = None, endpoint_url: str | None = None, aws_region: str | None = None, profile_name: str | None = None, token: str | None = None)¶ Initializes the S3Provider Example >>> s3_provider = S3Provider('snark-test/benchmarks') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.  aws_access_key_id (str, optional) – Specifies the AWS access key used as part of the credentials to authenticate the user.  aws_secret_access_key (str, optional) – Specifies the AWS secret key used as part of the credentials to authenticate the user.  aws_session_token (str, optional) – Specifies an AWS session token used as part of the credentials to authenticate the user.  endpoint_url (str, optional) – The complete URL to use for the constructed client.  This needs to be provided for cases in which you're interacting with MinIO, Wasabi, etc.  aws_region (str, optional) – Specifies the AWS Region to send requests to.  profile_name (str, optional) – Specifies the AWS profile name to use.  token (str, optional) – Activeloop token, used for fetching credentials for Deep Lake datasets (if this is underlying storage for Deep Lake dataset).  This is optional, tokens are normally autogenerated.  __iter__()¶ Generator function that iterates over the keys of the S3Provider.  Yields: str – the name of the object that it is iterating over.  __len__()¶ Returns the number of files present at the root of the S3Provider.  Note This is an expensive operation.  Returns: the number of files present inside the root.  Return type: int Raises: S3ListError – Any S3 error encountered while listing the objects.  __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the S3Provider.  content (bytes) – the value to be assigned at the path.  Raises: S3SetError – Any S3 error encountered while setting the value at the path.  ReadOnlyError – If the provider is in read-only mode


LRU Cache¶

get_deeplake_object(path: str, expected_class, meta: Dict | None = None, url=False, partial_bytes: int = 0)¶ If the data at path was stored using the output of a DeepLakeMemoryObject's tobytes function, this function will read it back into object form & keep the object in cache .  Parameters: path (str) – Path to the stored object.  expected_class (callable) – The expected subclass of DeepLakeMemoryObject.  meta (dict, optional) – Metadata associated with the stored object url (bool) – Get presigned url instead of downloading chunk (only for videos) partial_bytes (int) – Number of bytes to read from the beginning of the file.  If 0, reads the whole file.  Defaults to 0.  Raises: ValueError – If the incorrect expected_class was provided.  ValueError – If the type of the data at path is invalid.  ValueError – If url is True but expected_class is not a subclass of BaseChunk.  Returns: An instance of expected_class populated with the data.  register_deeplake_object(path: str, obj: DeepLakeMemoryObject)¶ Registers a new object in the cache.  remove_deeplake_object(path: str)¶ Removes a DeepLakeMemoryObject from the cache.


S3 Storage Provider¶

_all_keys()¶ Helper function that lists all the objects present at the root of the S3Provider .  Returns: set of all the objects found at the root of the S3Provider.  Return type: set Raises: S3ListError – Any S3 error encountered while listing the objects.  _check_update_creds(force=False)¶ If the client has an expiration time, check if creds are expired and fetch new ones.  This would only happen for datasets stored on Deep Lake storage for which temporary 12 hour credentials are generated.  _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters: hub_path (str) – The Deep Lake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  _state_keys()¶ Keys used to store the state of the provider.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the s3 bucket (under self.root).  Warning Exercise caution! get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  need_to_reload_creds(err: ClientError) → bool¶ Checks if the credentials need to be reloaded.  This happens if the credentials were loaded from the environment and have now expired.  rename(root)¶ Rename root folder.


Google Drive Storage Provider¶

class deeplake.core.storage.GDriveProvider¶ Bases: StorageProvider Provider class for using Google Drive storage. __delitem__(path)¶ Delete the object present at the path. Parameters: path (str) – the path to the object relative to the root of the provider. Raises: KeyError – If an object is not found at the path. __getitem__(path)¶ Gets the object present at the path within the given byte range. Parameters: path (str) – The path relative to the root of the provider. Returns: The bytes of the object present at the path. Return type: bytes Raises: KeyError – If an object is not found at the path. __init__(root: str, token: str | Dict | None = None, makemap: bool = True)¶ Initializes the GDriveProvider Example >>> gdrive_provider = GDriveProvider('gdrive://folder_name/folder_name') Copy to clipboard Parameters: root (str) – The root of the provider. All read/write request keys will be appended to root. token (dict, str, optional) – Google Drive token. Can be path to the token file or the actual credentials dictionary. makemap (bool) – Creates path to id map if True. Note Requires client_secrets.json in working directory if token is not provided. Due to limits on requests per 100 seconds on google drive api, continuous requests such as uploading many small files can be slow. Users can request to increse their quotas on their google cloud platform. __iter__()¶ Generator function that iterates over the keys of the provider. Yields: str – the path of the object that it is iterating over, relative to the root of the provider. __len__()¶ Returns the number of files present inside the root of the provider. Returns: the number of files present inside the root. Return type: int __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider. value (bytes) – the value to be assigned at the path. _all_keys()¶ Generator function that iterates over the keys of the provider. Returns: set of all keys present at the root of the provider. Return type: set clear(prefix='')¶ Delete the contents of the provider. sync()¶ Sync provider keys with actual storage


Local Storage Provider¶

class deeplake.core.storage.LocalProvider¶ Bases: StorageProvider Provider class for using the local filesystem.  __delitem__(path: str)¶ Delete the object present at the path.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> del local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> my_data = local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  __init__(root: str)¶ Initializes the LocalProvider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.' Raises: FileAtPathException – If the root is a file instead of a directory.  __iter__()¶ Generator function that iterates over the keys of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> for my_data in local_provider: ...  pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> len(local_provider) Copy to clipboard Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> local_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises: Exception – If unable to set item due to directory at path or permission or space issues.  FileAtPathException – If the directory to the path is a file instead of a directory.  ReadOnlyError – If the provider is in read-only mode.  _all_keys(refresh: bool = False) → Set[str]¶ Lists all the objects present at the root of the Provider.  Parameters: refresh (bool) – refresh keys Returns: set of all the objects found at the root of the Provider.  Return type: set _check_is_file(path: str)¶ Checks if the path is a file.  Returns the full_path to file if True.  Parameters: path (str) – the path to the object relative to the root of the provider


Base Storage Provider¶

Called at the end of methods which write data, to ensure consistency as a default .  set_bytes(path: str, value: bytes, start_byte: int | None = None, overwrite: bool | None = False)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  start_byte (int, optional) – If only specific bytes starting from start_byte are to be assigned.  overwrite (boolean, optional) – If the value is True, if there is an object present at the path it is completely overwritten, without fetching it's data.  Raises: InvalidBytesRequestedError – If start_byte < 0.  ReadOnlyModeError – If the provider is in read-only mode.


Local Storage Provider¶

Returns: the full path to the requested file .  Return type: str Raises: DirectoryAtPathException – If a directory is found at the path.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the local machine (under self.root).  Exercise caution! get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(path)¶ Renames root folder


Google Cloud Storage Provider¶

class deeplake.core.storage.GCSProvider¶ Bases: StorageProvider Provider class for using GC storage. __contains__(key)¶ Checks if key exists in mapping. __delitem__(key)¶ Remove key. __getitem__(key)¶ Retrieve data. __init__(root: str, token: str | Dict | None = None, project: str | None = None)¶ Initializes the GCSProvider. Example >>> gcs_provider = GCSProvider('gcs://my-bucket/gcs_ds') Copy to clipboard Parameters: root (str) – The root of the provider. All read/write request keys will be appended to root. token (str/Dict) – GCP token, used for fetching credentials for storage). Can be a path to the credentials file, actual credential dictionary or one of the folowing: - google_default: Tries to load default credentials for the specified project. - cache: Retrieves the previously used credentials from cache if exist. - anon: Sets credentials=None. - browser: Generates and stores new token file using cli. project (str) – Name of the project from GCloud. Raises: ModuleNotFoundError – If google cloud packages aren't installed. __iter__()¶ Iterating over the structure. __len__()¶ Returns length of the structure. __setitem__(key, value)¶ Store value in key. _all_keys()¶ Generator function that iterates over the keys of the provider. Returns: set of all keys present at the root of the provider. Return type: set _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials. These are only relevant to datasets using Deep Lake storage. This info is used to fetch new credentials when the temporary 12 hour credentials expire. Parameters: hub_path (str) – The deeplake cloud path to the dataset. expiration (str) – The time at which the credentials expire. clear(prefix='')¶ Remove all keys with given prefix below root - empties out mapping. Warning Exercise caution! get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range. Parameters: path (str) – The path relative to the root of the provider. start_byte (int, optional) – If only specific bytes starting from start_byte are required. end_byte (int, optional) – If only specific bytes up to end_byte are required. Returns: The bytes of the object present at the path within the given byte range. Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0. KeyError – If an object is not found at the path. rename(root)¶ Rename root folder.


Memory Provider¶

class deeplake.core.storage.MemoryProvider¶ Bases: StorageProvider Provider class for using the memory. __delitem__(path: str)¶ Delete the object present at the path. Example >>> memory_provider = MemoryProvider('xyz') >>> del memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider. Raises: KeyError – If an object is not found at the path. ReadOnlyError – If the provider is in read-only mode. __getitem__(path: str)¶ Gets the object present at the path within the given byte range. Example >>> memory_provider = MemoryProvider('xyz') >>> my_data = memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider. Returns: The bytes of the object present at the path. Return type: bytes Raises: KeyError – If an object is not found at the path. __getstate__() → str¶ Does NOT save the in memory data in state. __init__(root: str = '')¶ __iter__()¶ Generator function that iterates over the keys of the provider. Example >>> memory_provider = MemoryProvider('xyz') >>> for my_data in memory_provider: ... pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider. __len__()¶ Returns the number of files present inside the root of the provider. Example >>> memory_provider = MemoryProvider('xyz') >>> len(memory_provider) Copy to clipboard Returns: the number of files present inside the root. Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> memory_provider = MemoryProvider('xyz') >>> memory_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider. value (bytes) – the value to be assigned at the path. Raises: ReadOnlyError – If the provider is in read-only mode. _all_keys()¶ Lists all the objects present at the root of the Provider. Returns: set of all the objects found at the root of the Provider. Return type: set clear(prefix='')¶ Clears the provider.


deeplake.core.index¶

apply_squeeze(samples: List[ndarray])¶ Applies the primary axis of an Index to a list of ndarray samples .  Will either return the list as given, or return the first sample.  compose_at(item: int | slice | Tuple[int], i: int | None = None)¶ Returns a new Index representing the addition of an IndexValue, or the composition with a given axis.  Examples >>> Index([slice(None), slice(None)]).compose_at(5) Index([slice(None), slice(None), 5]) Copy to clipboard >>> Index([slice(None), slice(5, 10), slice(None)]).compose_at(3, 1) Index([slice(None), 8, slice(None)]) Copy to clipboard Parameters: item (IndexValue) – The value to append or compose with the Index.  i (int, optional) – The axis to compose with the given item.  Defaults to None, meaning that the item will be appended instead.  Returns: The result of the addition or composition.  Return type: Index find_axis(offset: int = 0)¶ Returns the index for the nth subscriptable axis in the values of an Index.  Parameters: offset (int) – The number of subscriptable axes to skip before returning.  Defaults to 0, meaning that the first valid axis is returned.  Returns: The index of the found axis, or None if no match is found.  Return type: int is_trivial()¶ Checks if an Index is equivalent to the trivial slice [:], aka slice(None).  length(parent_length: int)¶ Returns the primary length of an Index given the length of the parent it is indexing.  See: IndexEntry.length() validate(parent_length)¶ Checks that the index is not accessing values outside the range of the parent.  deeplake.core.index.merge_slices(existing_slice: slice, new_slice: slice) → slice¶ Compose two slice objects Given an iterable x, the following should be equivalent: x[existing_slice][new_slice] == x[merge_slices(existing_slice, new_slice)] Parameters: existing_slice (slice) – The existing slice to be restricted.  new_slice (slice) – The new slice to be applied to the existing slice.  Returns: the composition of the given slices Return type: slice Raises: NotImplementedError – Composing slices with negative values is not supported.  Negative indexing for slices is only supported for the first slice.  deeplake.core.index.slice_at_int(s: slice, i: int)¶ Returns the i th element of a slice s.  Examples >>> slice_at_int(slice(None), 10) 10 Copy to clipboard >>> slice_at_int(slice(10, 20, 2), 3) 16 Copy to clipboard Parameters: s (slice) – The slice to index into.  i (int) – The integer offset into the slice.  Returns: The index corresponding to the offset into the slice.  Return type: int Raises: NotImplementedError – Nontrivial slices should not be indexed with negative integers.  IndexError – If step is negative and start is not greater than stop.  deeplake.core.index.slice_length(s: slice, parent_length: int) → int¶ Returns the length of a slice given the length of its parent.


deeplake.core.index¶

class deeplake.core.index.IndexEntry(value: int | slice | Tuple[int] = slice(None, None, None))¶ __getitem__(item: int | slice | Tuple[int])¶ Combines the given item and this IndexEntry.  Returns a new IndexEntry representing the composition of the two.  Examples >>> IndexEntry()[0:100] IndexEntry(slice(0, 100, None)) Copy to clipboard >>> IndexEntry()[100:200][5] IndexEntry(105) Copy to clipboard >>> IndexEntry()[(0, 1, 2, 3)] IndexEntry((0, 1, 2, 3)) Copy to clipboard >>> IndexEntry()[1, 2, 3] IndexEntry((0, 1, 2, 3)) Copy to clipboard Parameters: item – The desired sub-index to be composed with this IndexEntry.  Can be an int, a slice, or a tuple of ints.  Returns: The new IndexEntry object.  Return type: IndexEntry Raises: TypeError – An integer IndexEntry should not be indexed further.  __init__(value: int | slice | Tuple[int] = slice(None, None, None))¶ __str__()¶ Return str(self).  __weakref__¶ list of weak references to the object (if defined) indices(length: int)¶ Generates the sequence of integer indices for a target of a given length.  is_trivial()¶ Checks if an IndexEntry represents the entire slice length(parent_length: int) → int¶ Returns the length of an IndexEntry given the length of the parent it is indexing.  Examples >>> IndexEntry(slice(5, 10)).length(100) 5 >>> len(list(range(100))[5:10]) 5 >>> IndexEntry(slice(5, 100)).length(50) 45 >>> len(list(range(50))[5:100]) 45 >>> IndexEntry(0).length(10) 1 Copy to clipboard Parameters: parent_length (int) – The length of the target that this IndexEntry is indexing.  Returns: The length of the index if it were applied to a parent of the given length.  Return type: int subscriptable()¶ Returns whether an IndexEntry can be further subscripted.  validate(parent_length: int)¶ Checks that the index is not accessing values outside the range of the parent.  class deeplake.core.index.Index(item: int | slice | Tuple[int] | Index | List[IndexEntry] = slice(None, None, None))¶ __getitem__(item: int | slice | List[int] | Tuple[int | slice | Tuple[int]] | Index)¶ Returns a new Index representing a subscripting with the given item.  Modeled after NumPy's advanced integer indexing.  See: https://numpy.org/doc/stable/reference/arrays.indexing.html Examples >>> Index([5, slice(None)])[5] Index([5, 5]) Copy to clipboard >>> Index([5])[5:6] Index([5, slice(5, 6)]) Copy to clipboard >>> Index()[0, 1, 2:5, 3] Index([0, 1, slice(2, 5), 3]) Copy to clipboard >>> Index([slice(5, 6)])[(0, 1, 2:5, 3),] Index([(5, 1, slice(2, 5), 3)]) Copy to clipboard Parameters: item – The contents of the subscript expression to add to this Index.  Returns: The Index representing the result of the subscript operation.  Return type: Index Raises: TypeError – Given item should be another Index, or compatible with NumPy's advanced integer indexing.  __init__(item: int | slice | Tuple[int] | Index | List[IndexEntry] = slice(None, None, None))¶ Initializes an Index from an IndexValue, another Index, or the values from another Index.  Represents a list of IndexEntry objects corresponding to indexes into each axis of an ndarray.  __repr__()¶ Return repr(self).  __str__()¶ Return str(self).  __weakref__¶ list of weak references to the object (if defined) apply(samples: List[ndarray])¶ Applies an Index to a list of ndarray samples with the same number of entries as the first entry in the Index


How Shuffling Works in Deep Lake's PyTorch DataLoader

The Deep Lake shuffling algorithm is based upon a shuffle buffer that preloads a specified amount of data (in MB) determined by the buffer_size parameter in ds.pytorch(buffer_size = 2048). First, the dataloader randomly selects chunks from the applicable tensors until the shuffle buffer is full. Next, the indices in shuffle buffer are randomly sampled to construct the batches that are returned by the dataloader. As the data in the shuffle buffer is consumed, new chunks are randomly selected and added to the buffer.In the OSS dataloader, the shuffle buffer contains the decompressed, decoded, and transformed samples. When using the PyTorch dataloaders, this corresponds to torch tensors. In the Performant dataloader, the shuffle buffer contains the non-decompressed data in the format they are stored in. For images, this typically corresponds to compressed bytes in jpeg, png, or other compressions. Since compressed data is stored more efficiently than uncompressed data, there are typically more distinct samples of data in the Performant dataloader shuffle buffer compared to the OSS shuffle buffer. If many chunks in the buffer contain data from the same class, which may occur if data was uploaded in non-random order, the shuffle buffer may contain fewer unique classes than if the samples were chosen fully randomly based on index. The most extreme case of reduced randomness occurs when datasets are much larger than the shuffle buffer, when they have many classes, and when those classes occur in sequence within the dataset indices. One example dataset is Unshuffled ImageNet, which has 1000 classes, 1.2M images, 140GB of data, and approximately 140 images per 16MB chunk. When the images are uploaded in sequence, the plot below shows how many unique classes are returned by the loader vs the number of images that have been returned in total. It is evident that fully randomly sampling returns more unique values than the Deep Lake dataloader. If reduced randomness has an impact on model performance in your workflows, the recommended countermeasures are:Store the dataset in a shuffled fashion such that the data does not appear in order by class. This completely mitigates the randomness concerns at the output of the data loader.Store the dataset with a smaller chunk size. This increases randomness because the shuffle buffer selects more discreet chunks before filling up. The current default size is 8, and reducing chunk size to 4MB significantly increases randomness (see plot above) with only a modest slowdown in data transfer speed.Increase the size of the shuffle buffer. This mitigates the randomness concerns but may not completely alleviate them.


Setting up Federated Credentials in Microsoft Azure

The most secure method for connecting data from your Azure storage to Deep Lake is using Federated Credentials, which are set up using the steps below:


Step 1: Register Application Credentials with the Microsoft Identity Platform

1. Login to the Azure account where the App will be registered and where the data is stored.2. Go to the App Registrations page in the Azure UI, which can be done by searching 'App registrations' in the console.3. Click on Register an application or New registration.4. Enter the Name and Supported account type (all are supported in Deep Lake) and click Register5. In the application console, click Certificates & secrets. 6. Click on Federated credentials and Add credential.7. Click on Select scenario and select Other issuer.8. Enter the following information in the form, and click Add.Issuer: https://cognito-identity.amazonaws.comThis is for trusting Activeloop's Cognito issuer. There's no need to create AWS Cognito by the user.Subject identifier: us-east-1:7bc30eb1-bac6-494b-bf53-5747849d45aaName: enter a name with your choiceDescription (optional): enter description a with your choiceAudience: us-east-1:57e5de2f-e2ec-4514-b9b0-f3bb8c4283c3


Step 2a: Apply the Application Credentials to your Azure storage account

Skip to 2b if you want to assign Application Credentials to a specific Azure container 1. Go to the Storage accounts page in the Azure UI, which can be done by searching 'Storage accounts' in the console.2. Select the Storage account to which you want to add Application Credentials.4. Select Access Control (IAM) and click Add, and select Add role assignment.5. Search and select Storage Blob Data Contributor under the role names and click Next.6. Click on the Select members link, and in the tab that opens up on the right, search by name and select the application you created in Step 1. Click Select at the bottom of the page. 7. The application should appear in the list of Members, at which point you can click Review + assign.


IMPORTANT TO PERFORM STEPS BELOW TO COMPLETE 2b - PLEASE DO NOT SKIP

5. Perform substeps 5-7 from Step 2a above, in order to add the Application Credentials to the Container6. Execute the steps in Step 2a above on your Storage Account, except set the Storage Account Role Assignment to Storage Blob Delegator in substep 5.


Step 2b: Apply the Application Credentials to a specific Azure contained in your Azure storage account

1. Go to the Storage accounts page in the Azure UI, which can be done by searching 'Storage accounts' in the console.2. Select the Storage account to which you want to add Application Credentials.3.Select the Container to which you add the Application Credentials.4. Select Access Control (IAM) and click Add, and select Add role assignment.


Enabling CORS in Azure for Data Visualization

Cross-Origin Resource Sharing (CORS) is typically enabled by default in Azure. If that's not the case in your Azure account, in order to visualize Deep Lake datasets stored in your own Azure storage in the Deep Lake app, please enable CORS in the storage account containing the Deep Lake dataset and any source data in linked tensors.


Steps for enabling CORS in Azure

1. Login to the Azure.2. Navigate to the Storage account with the relevant data.3. Open the Resource sharing (CORS) section on the left nav.4. Add the following items to the permissions.Allowed originsAllowed methodsAllowed headershttps://app.activeloop.aiGET, HEAD* https://app-dev.activeloop.dev GET, HEAD*https://app-staging.activeloop.devGET*


deeplake.core.sample¶

class deeplake.core.sample.Sample¶ __init__(path: Optional[str] = None, array: Optional[ndarray] = None, buffer: Optional[Union[bytes, memoryview]] = None, compression: Optional[str] = None, verify: bool = False, shape: Optional[Tuple[int]] = None, dtype: Optional[str] = None, creds: Optional[Dict] = None, storage: Optional[StorageProvider] = None)¶ Represents a single sample for a tensor. Provides all important meta information in one place. Note If self.is_lazy is True, this Sample doesn't actually have any data loaded. To read this data, simply try to read it into a numpy array (sample.array) Parameters: path (str) – Path to a sample stored on the local file system that represents a single sample. If path is provided, array should not be. Implicitly makes self.is_lazy == True. array (np.ndarray) – Array that represents a single sample. If array is provided, path should not be. Implicitly makes self.is_lazy == False. buffer – (bytes): Byte buffer that represents a single sample. If compressed, compression argument should be provided. compression (str) – Specify in case of byte buffer. verify (bool) – If a path is provided, verifies the sample if True. shape (Tuple[int]) – Shape of the sample. dtype (optional, str) – Data type of the sample. creds (optional, Dict) – Credentials for s3, gcp and http urls. storage (optional, StorageProvider) – Storage provider. Raises: ValueError – Cannot create a sample from both a path and array. property array: ndarray¶ Return numpy array corresponding to the sample. Decompresses the sample if necessary. Example >>> sample = deeplake.read('./images/dog.jpg') >>> arr = sample.array >>> arr.shape (323, 480, 3) Copy to clipboard compressed_bytes(compression: Optional[str]) → bytes¶ Returns this sample as compressed bytes. Note If this sample is pointing to a path and the requested compression is the same as it's stored in, the data is returned without re-compressing. Parameters: compression (Optional[str]) – self.array will be compressed into this format. If compression is None, return uncompressed_bytes(). Returns: Bytes for the compressed sample. Contains all metadata required to decompress within these bytes. Return type: bytes Raises: ValueError – On recompression of unsupported formats. property pil: Image¶ Return PIL image corresponding to the sample. Decompresses the sample if necessary. Example >>> sample = deeplake.read('./images/dog.jpg') >>> pil = sample.pil >>> pil.size (480, 323) Copy to clipboard uncompressed_bytes() → Optional[bytes]¶ Returns uncompressed bytes.


deeplake.core.linked_sample¶

class deeplake.core.linked_sample.LinkedSample(path: str, creds_key: Optional[str] = None)¶ Represents a sample that is initialized using external links. See deeplake.link().


deeplake.core.partial_sample¶

class deeplake.core.partial_sample.PartialSample(sample_shape: Tuple[int, ...], tile_shape: Optional[Tuple[int, ...]] = None, dtype: Union[str, dtype] = dtype('uint8'))¶ Represents a sample that is initialized by just shape and the data is updated later.


deeplake.core.storage¶

Base Storage Provider¶ class deeplake.core.storage.StorageProvider¶ abstract __delitem__(path: str)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  abstract __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  abstract __iter__()¶ Generator function that iterates over the keys of the provider.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  abstract __len__()¶ Returns the number of files present inside the root of the provider.  Returns: the number of files present inside the root.  Return type: int abstract __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  __weakref__¶ list of weak references to the object (if defined) abstract _all_keys() → Set[str]¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set _is_hub_path = False¶ An abstract base class for implementing a storage provider.  To add a new provider using Provider, create a subclass and implement all 5 abstract methods below.  check_readonly()¶ Raises an exception if the provider is in read-only mode.  abstract clear(prefix='')¶ Delete the contents of the provider.  copy()¶ Returns a copy of the provider.  Returns: A copy of the provider.  Return type: StorageProvider disable_readonly()¶ Disables read-only mode for the provider.  enable_readonly()¶ Enables read-only mode for the provider.  flush()¶ Only needs to be implemented for caches.  Flushes the data to the next storage provider.  Should be a no op for Base Storage Providers like local, s3, azure, gcs, etc.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  maybe_flush()¶ Flush cache if autoflush has been enabled


deeplake.core.storage¶

__len__()¶ Returns the number of files present in the cache and the underlying storage .  Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: Union[bytes, DeepLakeMemoryObject])¶ Puts the item in the cache_storage (if possible), else writes to next_storage.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  __setstate__(state: Dict[str, Any])¶ Recreates a cache with the same configuration as the state.  Parameters: state (dict) – The state to be used to recreate the cache.  Note While restoring the cache, we reset its contents.  In case the cache storage was local/s3 and is still accessible when unpickled (if same machine/s3 creds present respectively), the earlier cache contents are no longer accessible.  _all_keys()¶ Helper function that lists all the objects present in the cache and the underlying storage.  Returns: set of all the objects found in the cache and the underlying storage.  Return type: set _flush_if_not_read_only()¶ Flushes the cache if not in read-only mode.  _forward(path)¶ Forward the value at a given path to the next storage, and un-marks its key.  _forward_value(path, value)¶ Forwards a path-value pair to the next storage, and un-marks its key.  Parameters: path (str) – the path to the object relative to the root of the provider.  value (bytes, DeepLakeMemoryObject) – the value to send to the next storage.  _free_up_space(extra_size: int)¶ Helper function that frees up space the requred space in cache.No action is taken if there is sufficient space in the cache.  Parameters: extra_size (int) – the space that needs is required in bytes.  _insert_in_cache(path: str, value: Union[bytes, DeepLakeMemoryObject])¶ Helper function that adds a key value pair to the cache.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  _pop_from_cache()¶ Helper function that pops the least recently used key, value pair from the cache clear(prefix='')¶ Deletes ALL the data from all the layers of the cache and the actual storage.  This is an IRREVERSIBLE operation.  Data once deleted can not be recovered.  clear_cache()¶ Flushes the content of all the cache layers if not in read mode and and then deletes contents of all the layers of it.  This doesn't delete data from the actual storage.  clear_deeplake_objects()¶ Removes all DeepLakeMemoryObjects from the cache.  flush()¶ Writes data from cache_storage to next_storage.  Only the dirty keys are written.  This is a cascading function and leads to data being written to the final storage in case of a chained cache.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range


deeplake.core.storage¶

Parameters: path (str) – The path relative to the root of the provider .  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  get_deeplake_object(path: str, expected_class, meta: Optional[Dict] = None, url=False, partial_bytes: int = 0)¶ If the data at path was stored using the output of a DeepLakeMemoryObject's tobytes function, this function will read it back into object form & keep the object in cache.  Parameters: path (str) – Path to the stored object.  expected_class (callable) – The expected subclass of DeepLakeMemoryObject.  meta (dict, optional) – Metadata associated with the stored object url (bool) – Get presigned url instead of downloading chunk (only for videos) partial_bytes (int) – Number of bytes to read from the beginning of the file.  If 0, reads the whole file.  Defaults to 0.  Raises: ValueError – If the incorrect expected_class was provided.  ValueError – If the type of the data at path is invalid.  ValueError – If url is True but expected_class is not a subclass of BaseChunk.  Returns: An instance of expected_class populated with the data.  register_deeplake_object(path: str, obj: DeepLakeMemoryObject)¶ Registers a new object in the cache.  remove_deeplake_object(path: str)¶ Removes a DeepLakeMemoryObject from the cache.  S3 Storage Provider¶ class deeplake.core.storage.S3Provider¶ Bases: StorageProvider Provider class for using S3 storage.  __delitem__(path)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the S3Provider.  Note If the object is not found, s3 won't raise KeyError.  Raises: S3DeletionError – Any S3 error encountered while deleting the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path)¶ Gets the object present at the path.  Parameters: path (str) – the path relative to the root of the S3Provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  __init__(root: str, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, endpoint_url: Optional[str] = None, aws_region: Optional[str] = None, profile_name: Optional[str] = None, token: Optional[str] = None)¶ Initializes the S3Provider Example >>> s3_provider = S3Provider('snark-test/benchmarks') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.  aws_access_key_id (str, optional) – Specifies the AWS access key used as part of the credentials to authenticate the user


deeplake.core.storage¶

end_byte (int, optional) – If only specific bytes up to end_byte are required .  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  need_to_reload_creds(err: ClientError) → bool¶ Checks if the credentials need to be reloaded.  This happens if the credentials were loaded from the environment and have now expired.  rename(root)¶ Rename root folder.  Google Cloud Storage Provider¶ class deeplake.core.storage.GCSProvider¶ Bases: StorageProvider Provider class for using GC storage.  __contains__(key)¶ Checks if key exists in mapping.  __delitem__(key)¶ Remove key.  __getitem__(key)¶ Retrieve data.  __init__(root: str, token: Optional[Union[str, Dict]] = None, project: Optional[str] = None)¶ Initializes the GCSProvider.  Example >>> gcs_provider = GCSProvider('gcs://my-bucket/gcs_ds') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.  token (str/Dict) – GCP token, used for fetching credentials for storage).  Can be a path to the credentials file, actual credential dictionary or one of the folowing: - google_default: Tries to load default credentials for the specified project.  - cache: Retrieves the previously used credentials from cache if exist.  - anon: Sets credentials=None.  - browser: Generates and stores new token file using cli.  project (str) – Name of the project from GCloud.  Raises: ModuleNotFoundError – If google cloud packages aren't installed.  __iter__()¶ Iterating over the structure.  __len__()¶ Returns length of the structure.  __setitem__(key, value)¶ Store value in key.  _all_keys()¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters: hub_path (str) – The deeplake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  clear(prefix='')¶ Remove all keys with given prefix below root - empties out mapping.  Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0


deeplake.core.storage¶

Called at the end of methods which write data, to ensure consistency as a default .  set_bytes(path: str, value: bytes, start_byte: Optional[int] = None, overwrite: Optional[bool] = False)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  start_byte (int, optional) – If only specific bytes starting from start_byte are to be assigned.  overwrite (boolean, optional) – If the value is True, if there is an object present at the path it is completely overwritten, without fetching it's data.  Raises: InvalidBytesRequestedError – If start_byte < 0.  ReadOnlyModeError – If the provider is in read-only mode.  LRU Cache¶ class deeplake.core.storage.LRUCache¶ Bases: StorageProvider LRU Cache that uses StorageProvider for caching __delitem__(path: str)¶ Deletes the object present at the path from the cache and the underlying storage.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ If item is in cache_storage, retrieves from there and returns.  If item isn't in cache_storage, retrieves from next storage, stores in cache_storage (if possible) and returns.  Parameters: path (str) – The path relative to the root of the underlying storage.  Raises: KeyError – if an object is not found at the path.  Returns: The bytes of the object present at the path.  Return type: bytes __getstate__() → Dict[str, Any]¶ Returns the state of the cache, for pickling __init__(cache_storage: StorageProvider, next_storage: Optional[StorageProvider], cache_size: int)¶ Initializes the LRUCache.  It can be chained with other LRUCache objects to create multilayer caches.  Parameters: cache_storage (StorageProvider) – The storage being used as the caching layer of the cache.  This should be a base provider such as MemoryProvider, LocalProvider or S3Provider but not another LRUCache.  next_storage (StorageProvider) – The next storage layer of the cache.  This can either be a base provider (i.e.  it is the final storage) or another LRUCache (i.e.  in case of chained cache).  While reading data, all misses from cache would be retrieved from here.  While writing data, the data will be written to the next_storage when cache_storage is full or flush is called.  cache_size (int) – The total space that can be used from the cache_storage in bytes.  This number may be less than the actual space available on the cache_storage.  Setting it to a higher value than actually available space may lead to unexpected behaviors.  __iter__()¶ Generator function that iterates over the keys of the cache and the underlying storage.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider


deeplake.core.storage¶

KeyError – If an object is not found at the path .  rename(root)¶ Rename root folder.  Google Drive Storage Provider¶ class deeplake.core.storage.GDriveProvider¶ Bases: StorageProvider Provider class for using Google Drive storage.  __delitem__(path)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  __getitem__(path)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  __init__(root: str, token: Optional[Union[str, Dict]] = None, makemap: bool = True)¶ Initializes the GDriveProvider Example >>> gdrive_provider = GDriveProvider('gdrive://folder_name/folder_name') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.  token (dict, str, optional) – Google Drive token.  Can be path to the token file or the actual credentials dictionary.  makemap (bool) – Creates path to id map if True.  Note Requires client_secrets.json in working directory if token is not provided.  Due to limits on requests per 100 seconds on google drive api, continuous requests such as uploading many small files can be slow.  Users can request to increse their quotas on their google cloud platform.  __iter__()¶ Generator function that iterates over the keys of the provider.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Returns: the number of files present inside the root.  Return type: int __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  _all_keys()¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set clear(prefix='')¶ Delete the contents of the provider.  sync()¶ Sync provider keys with actual storage Local Storage Provider¶ class deeplake.core.storage.LocalProvider¶ Bases: StorageProvider Provider class for using the local filesystem.  __delitem__(path: str)¶ Delete the object present at the path.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> del local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  ReadOnlyError – If the provider is in read-only mode


deeplake.core.storage¶

__getitem__(path: str)¶ Gets the object present at the path within the given byte range .  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> my_data = local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  __init__(root: str)¶ Initializes the LocalProvider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.' Raises: FileAtPathException – If the root is a file instead of a directory.  __iter__()¶ Generator function that iterates over the keys of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> for my_data in local_provider: ...  pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> len(local_provider) Copy to clipboard Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> local_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises: Exception – If unable to set item due to directory at path or permission or space issues.  FileAtPathException – If the directory to the path is a file instead of a directory.  ReadOnlyError – If the provider is in read-only mode.  _all_keys(refresh: bool = False) → Set[str]¶ Lists all the objects present at the root of the Provider.  Parameters: refresh (bool) – refresh keys Returns: set of all the objects found at the root of the Provider.  Return type: set _check_is_file(path: str)¶ Checks if the path is a file.  Returns the full_path to file if True.  Parameters: path (str) – the path to the object relative to the root of the provider.  Returns: the full path to the requested file.  Return type: str Raises: DirectoryAtPathException – If a directory is found at the path.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the local machine (under self.root).  Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required


deeplake.core.storage¶

aws_secret_access_key (str, optional) – Specifies the AWS secret key used as part of the credentials to authenticate the user .  aws_session_token (str, optional) – Specifies an AWS session token used as part of the credentials to authenticate the user.  endpoint_url (str, optional) – The complete URL to use for the constructed client.  This needs to be provided for cases in which you're interacting with MinIO, Wasabi, etc.  aws_region (str, optional) – Specifies the AWS Region to send requests to.  profile_name (str, optional) – Specifies the AWS profile name to use.  token (str, optional) – Activeloop token, used for fetching credentials for Deep Lake datasets (if this is underlying storage for Deep Lake dataset).  This is optional, tokens are normally autogenerated.  __iter__()¶ Generator function that iterates over the keys of the S3Provider.  Yields: str – the name of the object that it is iterating over.  __len__()¶ Returns the number of files present at the root of the S3Provider.  Note This is an expensive operation.  Returns: the number of files present inside the root.  Return type: int Raises: S3ListError – Any S3 error encountered while listing the objects.  __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the S3Provider.  content (bytes) – the value to be assigned at the path.  Raises: S3SetError – Any S3 error encountered while setting the value at the path.  ReadOnlyError – If the provider is in read-only mode.  _all_keys()¶ Helper function that lists all the objects present at the root of the S3Provider.  Returns: set of all the objects found at the root of the S3Provider.  Return type: set Raises: S3ListError – Any S3 error encountered while listing the objects.  _check_update_creds(force=False)¶ If the client has an expiration time, check if creds are expired and fetch new ones.  This would only happen for datasets stored on Deep Lake storage for which temporary 12 hour credentials are generated.  _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters: hub_path (str) – The Deep Lake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  _state_keys()¶ Keys used to store the state of the provider.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the s3 bucket (under self.root).  Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required


Base Storage Provider¶

set_bytes(path: str, value: bytes, start_byte: Optional[int] = None, overwrite: Optional[bool] = False)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider .  value (bytes) – the value to be assigned at the path.  start_byte (int, optional) – If only specific bytes starting from start_byte are to be assigned.  overwrite (boolean, optional) – If the value is True, if there is an object present at the path it is completely overwritten, without fetching it's data.  Raises: InvalidBytesRequestedError – If start_byte < 0.  ReadOnlyModeError – If the provider is in read-only mode.


deeplake.core.storage¶

end_byte (int, optional) – If only specific bytes up to end_byte are required .  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(path)¶ Renames root folder Memory Provider¶ class deeplake.core.storage.MemoryProvider¶ Bases: StorageProvider Provider class for using the memory.  __delitem__(path: str)¶ Delete the object present at the path.  Example >>> memory_provider = MemoryProvider('xyz') >>> del memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Example >>> memory_provider = MemoryProvider('xyz') >>> my_data = memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  __getstate__() → str¶ Does NOT save the in memory data in state.  __init__(root: str = '')¶ __iter__()¶ Generator function that iterates over the keys of the provider.  Example >>> memory_provider = MemoryProvider('xyz') >>> for my_data in memory_provider: ...  pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Example >>> memory_provider = MemoryProvider('xyz') >>> len(memory_provider) Copy to clipboard Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> memory_provider = MemoryProvider('xyz') >>> memory_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  _all_keys()¶ Lists all the objects present at the root of the Provider.  Returns: set of all the objects found at the root of the Provider.  Return type: set clear(prefix='')¶ Clears the provider.


LRU Cache¶

class deeplake.core.storage.LRUCache¶ Bases: StorageProvider LRU Cache that uses StorageProvider for caching __delitem__(path: str)¶ Deletes the object present at the path from the cache and the underlying storage.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ If item is in cache_storage, retrieves from there and returns.  If item isn't in cache_storage, retrieves from next storage, stores in cache_storage (if possible) and returns.  Parameters: path (str) – The path relative to the root of the underlying storage.  Raises: KeyError – if an object is not found at the path.  Returns: The bytes of the object present at the path.  Return type: bytes __getstate__() → Dict[str, Any]¶ Returns the state of the cache, for pickling __init__(cache_storage: StorageProvider, next_storage: Optional[StorageProvider], cache_size: int)¶ Initializes the LRUCache.  It can be chained with other LRUCache objects to create multilayer caches.  Parameters: cache_storage (StorageProvider) – The storage being used as the caching layer of the cache.  This should be a base provider such as MemoryProvider, LocalProvider or S3Provider but not another LRUCache.  next_storage (StorageProvider) – The next storage layer of the cache.  This can either be a base provider (i.e.  it is the final storage) or another LRUCache (i.e.  in case of chained cache).  While reading data, all misses from cache would be retrieved from here.  While writing data, the data will be written to the next_storage when cache_storage is full or flush is called.  cache_size (int) – The total space that can be used from the cache_storage in bytes.  This number may be less than the actual space available on the cache_storage.  Setting it to a higher value than actually available space may lead to unexpected behaviors.  __iter__()¶ Generator function that iterates over the keys of the cache and the underlying storage.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present in the cache and the underlying storage.  Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: Union[bytes, DeepLakeMemoryObject])¶ Puts the item in the cache_storage (if possible), else writes to next_storage.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  __setstate__(state: Dict[str, Any])¶ Recreates a cache with the same configuration as the state.  Parameters: state (dict) – The state to be used to recreate the cache.  Note While restoring the cache, we reset its contents


Base Storage Provider¶

class deeplake.core.storage.StorageProvider¶ abstract __delitem__(path: str)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  abstract __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  abstract __iter__()¶ Generator function that iterates over the keys of the provider.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  abstract __len__()¶ Returns the number of files present inside the root of the provider.  Returns: the number of files present inside the root.  Return type: int abstract __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  __weakref__¶ list of weak references to the object (if defined) abstract _all_keys() → Set[str]¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set _is_hub_path = False¶ An abstract base class for implementing a storage provider.  To add a new provider using Provider, create a subclass and implement all 5 abstract methods below.  check_readonly()¶ Raises an exception if the provider is in read-only mode.  abstract clear(prefix='')¶ Delete the contents of the provider.  copy()¶ Returns a copy of the provider.  Returns: A copy of the provider.  Return type: StorageProvider disable_readonly()¶ Disables read-only mode for the provider.  enable_readonly()¶ Enables read-only mode for the provider.  flush()¶ Only needs to be implemented for caches.  Flushes the data to the next storage provider.  Should be a no op for Base Storage Providers like local, s3, azure, gcs, etc.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  maybe_flush()¶ Flush cache if autoflush has been enabled.  Called at the end of methods which write data, to ensure consistency as a default


S3 Storage Provider¶

class deeplake.core.storage.S3Provider¶ Bases: StorageProvider Provider class for using S3 storage.  __delitem__(path)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the S3Provider.  Note If the object is not found, s3 won't raise KeyError.  Raises: S3DeletionError – Any S3 error encountered while deleting the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path)¶ Gets the object present at the path.  Parameters: path (str) – the path relative to the root of the S3Provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  __init__(root: str, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, endpoint_url: Optional[str] = None, aws_region: Optional[str] = None, profile_name: Optional[str] = None, token: Optional[str] = None)¶ Initializes the S3Provider Example >>> s3_provider = S3Provider('snark-test/benchmarks') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.  aws_access_key_id (str, optional) – Specifies the AWS access key used as part of the credentials to authenticate the user.  aws_secret_access_key (str, optional) – Specifies the AWS secret key used as part of the credentials to authenticate the user.  aws_session_token (str, optional) – Specifies an AWS session token used as part of the credentials to authenticate the user.  endpoint_url (str, optional) – The complete URL to use for the constructed client.  This needs to be provided for cases in which you're interacting with MinIO, Wasabi, etc.  aws_region (str, optional) – Specifies the AWS Region to send requests to.  profile_name (str, optional) – Specifies the AWS profile name to use.  token (str, optional) – Activeloop token, used for fetching credentials for Deep Lake datasets (if this is underlying storage for Deep Lake dataset).  This is optional, tokens are normally autogenerated.  __iter__()¶ Generator function that iterates over the keys of the S3Provider.  Yields: str – the name of the object that it is iterating over.  __len__()¶ Returns the number of files present at the root of the S3Provider.  Note This is an expensive operation.  Returns: the number of files present inside the root.  Return type: int Raises: S3ListError – Any S3 error encountered while listing the objects.  __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the S3Provider.  content (bytes) – the value to be assigned at the path.  Raises: S3SetError – Any S3 error encountered while setting the value at the path.  ReadOnlyError – If the provider is in read-only mode.  _all_keys()¶ Helper function that lists all the objects present at the root of the S3Provider


LRU Cache¶

get_deeplake_object(path: str, expected_class, meta: Optional[Dict] = None, url=False, partial_bytes: int = 0)¶ If the data at path was stored using the output of a DeepLakeMemoryObject's tobytes function, this function will read it back into object form & keep the object in cache .  Parameters: path (str) – Path to the stored object.  expected_class (callable) – The expected subclass of DeepLakeMemoryObject.  meta (dict, optional) – Metadata associated with the stored object url (bool) – Get presigned url instead of downloading chunk (only for videos) partial_bytes (int) – Number of bytes to read from the beginning of the file.  If 0, reads the whole file.  Defaults to 0.  Raises: ValueError – If the incorrect expected_class was provided.  ValueError – If the type of the data at path is invalid.  ValueError – If url is True but expected_class is not a subclass of BaseChunk.  Returns: An instance of expected_class populated with the data.  register_deeplake_object(path: str, obj: DeepLakeMemoryObject)¶ Registers a new object in the cache.  remove_deeplake_object(path: str)¶ Removes a DeepLakeMemoryObject from the cache.