Tracking the training progress

job_id is string, which can be used to track the training progress. You can use db.deep_memory.status(job_id) to get the status of the job.when the model is still in pending state (not started yet) you will see the following: >>> db.deep_memory.status(job_id) ————————————————————– | 6508464cd80cab681bfcfff3 | ————————————————————– | status | pending | ————————————————————– | progress | None | ————————————————————– | results | not available yet | ————————————————————–After some time the model will start training and you will see the following: Copy to clipboard If you want to get all training jobs you can use db.deep_memory.list_jobs() which will show all jobs that happened on this dataset. Copy to clipboard


Training

To start training you should first create a vectostore object, and then preprocess the data and use deep memory with it: Copy to clipboard To train a deepmemory model you need to preprocess the dataset so that, corpus, will become a list of list of tuples, where outer list corresponds to the query and inner list to the relevant documents. Each tuple should contain the document id (id tensor from the corpus dataset) and the relevance score (range is 0-1, where 0 represents unrelated document and 1 related). queries should be a list of strings. Copy to clipboard


Deep Memory Search

After the model is trained you also can search using it: Copy to clipboard


deeplake

The deeplake package provides a database which stores data as compressed chunked arrays that can be stored anywhere and later streamed to deep learning models.  deeplake.dataset(path: Union[str, Path], runtime: Optional[Dict] = None, read_only: Optional[bool] = None, overwrite: bool = False, public: bool = False, memory_cache_size: int = 2000, local_cache_size: int = 0, creds: Optional[Union[str, Dict]] = None, token: Optional[str] = None, org_id: Optional[str] = None, verbose: bool = True, access_method: str = 'stream', unlink: bool = False, reset: bool = False, check_integrity: bool = True, lock_enabled: Optional[bool] = True, lock_timeout: Optional[int] = 0, index_params: Optional[Dict[str, Union[int, str]]] = None) Returns a Dataset object referencing either a new or existing dataset.  Examples >>> ds = deeplake.dataset('hub://username/dataset') >>> ds = deeplake.dataset('s3://mybucket/my_dataset') >>> ds = deeplake.dataset('./datasets/my_dataset', overwrite=True) Copy to clipboard Loading to a specfic version: >>> ds = deeplake.dataset('hub://username/dataset@new_branch') >>> ds = deeplake.dataset('hub://username/dataset@3e49cded62b6b335c74ff07e97f8451a37aca7b2) Copy to clipboard >>> my_commit_id = '3e49cded62b6b335c74ff07e97f8451a37aca7b2' >>> ds = deeplake.dataset(f'hub://username/dataset@{my_commit_id}') Copy to clipboard Parameters path (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  Loading to a specific version: You can also specify a commit_id or branch to load the dataset to that version directly by using the @ symbol.  The path will then be of the form hub://username/dataset@{branch} or hub://username/dataset@{commit_id}.  See examples above.  runtime (dict) – Parameters for Activeloop DB Engine.  Only applicable for hub:// paths.  read_only (bool, optional) – Opens dataset in read only mode if this is passed as True.  Defaults to False.  Datasets stored on Deep Lake cloud that your account does not have write access to will automatically open in read mode.  overwrite (bool) – If set to True this overwrites the dataset if it already exists.  Defaults to False.  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to True.  memory_cache_size (int) – The size of the memory cache to be used in MB.  local_cache_size (int) – The size of the local filesystem cache to be used in MB.  creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file


deeplake

creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path .  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  org_id (str, Optional) – Organization id to be used for enabling high-performance features.  Only applicable for local datasets.  verbose (bool) – If True, logs will be printed.  Defaults to True.  lock_timeout (int) – Number of seconds to wait before throwing a LockException.  If None, wait indefinitely lock_enabled (bool) – If true, the dataset manages a write lock.  NOTE: Only set to False if you are managing concurrent access externally.  index_params – Optional[Dict[str, Union[int, str]]]: Index parameters used while creating vector store, passed down to dataset.  Returns Dataset created using the arguments provided.  Return type Dataset Raises DatasetHandlerError – If a Dataset already exists at the given path and overwrite is False.  UserNotLoggedInException – When user is not logged in InvalidTokenException – If the specified toke is invalid TokenPermissionError – When there are permission or other errors related to token ValueError – If version is specified in the path Danger Setting overwrite to True will delete all of your data if it exists! Be very careful when setting this parameter.  deeplake.like(dest: Union[str, Path], src: Union[str, Dataset, Path], runtime: Optional[Dict] = None, tensors: Optional[List[str]] = None, overwrite: bool = False, creds: Optional[Union[dict, str]] = None, token: Optional[str] = None, org_id: Optional[str] = None, public: bool = False, verbose: bool = True) → Dataset Creates a new dataset by copying the source dataset's structure to a new location.  No samples are copied, only the meta/info for the dataset and it's tensors.  Parameters dest – Empty Dataset or Path where the new dataset will be created.  src (Union[str, Dataset]) – Path or dataset object that will be used as the template for the new dataset.  runtime (dict) – Parameters for Activeloop DB Engine.  Only applicable for hub:// paths.  tensors (List[str], optional) – Names of tensors (and groups) to be replicated.  If not specified all tensors in source dataset are considered.  overwrite (bool) – If True and a dataset exists at destination, it will be overwritten.  Defaults to False


deeplake

Currently only works with s3 paths .  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  org_id (str, Optional) – Organization id to be used for enabling high-performance features.  Only applicable for local datasets.  verbose (bool) – If True, logs will be printed.  Defaults to True.  access_method (str) – The access method to use for the dataset.  Can be: 'stream' Streams the data from the dataset i.e.  only fetches data when required.  This is the default value.  'download' Downloads the data to the local filesystem to the path specified in environment variable DEEPLAKE_DOWNLOAD_PATH.  This will overwrite DEEPLAKE_DOWNLOAD_PATH.  Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set or if the dataset does not exist.  The ‘download' access method can be modified to specify num_workers and/or scheduler.  For example: ‘download:2:processed' will use 2 workers and use processed scheduler, while ‘download:3' will use 3 workers and default scheduler (threaded), and ‘download:processed' will use a single worker and use processed scheduler.  'local' Downloads the dataset if it doesn't already exist, otherwise loads from local storage.  Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set.  The ‘local' access method can be modified to specify num_workers and/or scheduler to be used in case dataset needs to be downloaded.  If dataset needs to be downloaded, ‘local:2:processed' will use 2 workers and use processed scheduler, while ‘local:3' will use 3 workers and default scheduler (threaded), and ‘local:processed' will use a single worker and use processed scheduler.  unlink (bool) – Downloads linked samples if set to True.  Only applicable if access_method is download or local.  Defaults to False.  reset (bool) – If the specified dataset cannot be loaded due to a corrupted HEAD state of the branch being loaded, setting reset=True will reset HEAD changes and load the previous version.  check_integrity (bool) – If the param is True it will do integrity check during dataset loading otherwise the check is not performed lock_timeout (int) – Number of seconds to wait before throwing a LockException.  If None, wait indefinitely lock_enabled (bool) – If true, the dataset manages a write lock.  NOTE: Only set to False if you are managing concurrent access externally index_params – Optional[Dict[str, Union[int, str]]] = None : The index parameters used while creating vector store is passed down to dataset.  Returns Dataset created using the arguments provided


deeplake

summary (bool) – If True, a summary of skipped files will be printed after completion .  Defaults to True.  num_workers (int) – The number of workers to use for ingestion.  Set to 0 by default.  shuffle (bool) – Shuffles the input data prior to ingestion.  Since data arranged in folders by class is highly non-random, shuffling is important in order to produce optimal results when training.  Defaults to True.  token (Optional[str]) – The token to use for accessing the dataset.  connect_kwargs (Optional[Dict]) – If specified, the dataset will be connected to Deep Lake, and connect_kwargs will be passed to Dataset.connect.  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function see deeplake.empty().  Returns New dataset object with structured dataset.  Return type Dataset Raises InvalidPathException – If the source directory does not exist.  SamePathException – If the source and destination path are same.  AutoCompressionError – If the source director is empty or does not contain a valid extension.  InvalidFileExtension – If the most frequent file extension is found to be ‘None' during auto-compression.  Note Currently only local source paths and image classification datasets / csv files are supported for automatic ingestion.  Supported filetypes: png/jpeg/jpg/csv.  All files and sub-directories with unsupported filetypes are ignored.  Valid source directory structures for image classification look like: data/ img0.jpg img1.jpg ...  Copy to clipboard or: data/ class0/ cat0.jpg ...  class1/ dog0.jpg ...  ...  Copy to clipboard or: data/ train/ class0/ img0.jpg ...  ...  val/ class0/ img0.jpg ...  ...  ...  Copy to clipboard Classes defined as sub-directories can be accessed at ds['test/labels'].info.class_names.  Support for train and test sub directories is present under ds['train/images'], ds['train/labels'] and ds['test/images'], ds['test/labels'].  Mapping filenames to classes from an external file is currently not supported.  deeplake.ingest_coco(images_directory: Union[str, Path], annotation_files: Union[str, Path, List[str]], dest: Union[str, Path], key_to_tensor_mapping: Optional[Dict] = None, file_to_group_mapping: Optional[Dict] = None, ignore_one_group: bool = True, ignore_keys: Optional[List[str]] = None, image_params: Optional[Dict] = None, image_creds_key: Optional[str] = None, src_creds: Optional[Union[str, Dict]] = None, dest_creds: Optional[Union[str, Dict]] = None, inspect_limit: int = 1000000, progressbar: bool = True, shuffle: bool = False, num_workers: int = 0, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) → Dataset Ingest images and annotations in COCO format to a Deep Lake Dataset.  The source data can be stored locally or in the cloud.  Examples >>> # Ingest local data in COCO format to a Deep Lake dataset stored in Deep Lake storage.  >>> ds = deeplake.ingest_coco( >>> '<path/to/images/directory>', >>> ['path/to/annotation/file1.json', 'path/to/annotation/file2.json'], >>> dest='hub://org_id/dataset', >>> key_to_tensor_mapping={'category_id': 'labels', 'bbox': 'boxes'}, >>> file_to_group_mapping={'file1.json': 'group1', 'file2.json': 'group2'}, >>> ignore_keys=['area', 'image_id', 'id'], >>> num_workers=4, >>> ) >>> # Ingest data from your cloud into another Deep Lake dataset in your cloud, and connect that dataset to the Deep Lake backend


deeplake

Return type Dataset Raises IngestionError – If either key_to_tensor_mapping or file_to_group_mapping are not one-to-one .  deeplake.ingest_yolo(data_directory: Union[str, Path], dest: Union[str, Path], class_names_file: Optional[Union[str, Path]] = None, annotations_directory: Optional[Union[str, Path]] = None, allow_no_annotation: bool = False, image_params: Optional[Dict] = None, label_params: Optional[Dict] = None, coordinates_params: Optional[Dict] = None, src_creds: Optional[Union[str, Dict]] = None, dest_creds: Optional[Union[str, Dict]] = None, image_creds_key: Optional[str] = None, inspect_limit: int = 1000, progressbar: bool = True, shuffle: bool = False, num_workers: int = 0, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) → Dataset Ingest images and annotations (bounding boxes or polygons) in YOLO format to a Deep Lake Dataset.  The source data can be stored locally or in the cloud.  Examples >>> # Ingest local data in YOLO format to a Deep Lake dataset stored in Deep Lake storage.  >>> ds = deeplake.ingest_yolo( >>> 'path/to/data/directory', >>> dest='hub://org_id/dataset', >>> allow_no_annotation=True, >>> token='my_activeloop_token', >>> num_workers=4, >>> ) >>> # Ingest data from your cloud into another Deep Lake dataset in your cloud, and connect that dataset to the Deep Lake backend.  >>> ds = deeplake.ingest_yolo( >>> 's3://bucket/data_directory', >>> dest='s3://bucket/dataset_name', >>> image_params={'name': 'image_links', 'htype': 'link[image]'}, >>> image_creds_key='my_s3_managed_credentials', >>> src_creds=aws_creds, # Can also be inferred from environment >>> dest_creds=aws_creds, # Can also be inferred from environment >>> connect_kwargs={'creds_key': 'my_s3_managed_credentials', 'org_id': 'org_id'}, >>> num_workers=4, >>> ) Copy to clipboard Parameters data_directory (str, pathlib.Path) – The path to the directory containing the data (images files and annotation files(see ‘annotations_directory' input for specifying annotations in a separate directory).  dest (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://org_id/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line), or pass in a token using the ‘token' parameter.  an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  class_names_file – Path to the file containing the class names on separate lines.  This is typically a file titled classes.names.  annotations_directory (Optional[Union[str, pathlib.Path]]) – Path to directory containing the annotations.  If specified, the ‘data_directory' will not be examined for annotations.  allow_no_annotation (bool) – Flag to determine whether missing annotations files corresponding to an image should be treated as empty annoations.  Set to False by default.  image_params (Optional[Dict]) – A dictionary containing parameters for the images tensor.  label_params (Optional[Dict]) – A dictionary containing parameters for the labels tensor.  coordinates_params (Optional[Dict]) – A dictionary containing parameters for the ccoordinates tensor.  This tensor either contains bounding boxes or polygons.  src_creds (Optional[Union[str, Dict]]) – Credentials to access the source data


deeplake

Return type Dataset Raises AgreementError – When agreement is rejected UserNotLoggedInException – When user is not logged in InvalidTokenException – If the specified token is invalid TokenPermissionError – When there are permission or other errors related to token CheckoutError – If version address specified in the path cannot be found DatasetCorruptError – If loading the dataset failed due to corruption and reset is not True ValueError – If version is specified in the path when creating a dataset or If the org id is provided but dataset is ot local, or If the org id is provided but dataset is ot local ReadOnlyModeError – If reset is attempted in read-only mode LockedException – When attempting to open a dataset for writing when it is locked by another machine Exception – Re-raises caught exception if reset cannot fix the issue Danger Setting overwrite to True will delete all of your data if it exists! Be very careful when setting this parameter .  Warning Setting access_method to download will overwrite the local copy of the dataset if it was previously downloaded.  Note Any changes made to the dataset in download / local mode will only be made to the local copy and will not be reflected in the original dataset.  deeplake.empty(path: Union[str, Path], runtime: Optional[dict] = None, overwrite: bool = False, public: bool = False, memory_cache_size: int = 2000, local_cache_size: int = 0, creds: Optional[Union[str, Dict]] = None, token: Optional[str] = None, org_id: Optional[str] = None, lock_enabled: Optional[bool] = True, lock_timeout: Optional[int] = 0, verbose: bool = True, index_params: Optional[Dict[str, Union[int, str]]] = None) → Dataset Creates an empty dataset Parameters path (str, pathlib.Path) – The full path to the dataset.  It can be: a Deep Lake cloud path of the form hub://org_id/dataset_name.  Requires registration with Deep Lake.  an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  runtime (dict) – Parameters for creating a dataset in the Deep Lake Tensor Database.  Only applicable for paths of the form hub://org_id/dataset_name and runtime must be {'tensor_db': True}.  overwrite (bool) – If set to True this overwrites the dataset if it already exists.  Defaults to False.  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to False.  memory_cache_size (int) – The size of the memory cache to be used in MB.  local_cache_size (int) – The size of the local filesystem cache to be used in MB


deeplake

>>> ds = deeplake.ingest_coco( >>> 's3://bucket/images/directory', >>> 's3://bucket/annotation/file1.json', >>> dest='s3://bucket/dataset_name', >>> ignore_one_group=True, >>> ignore_keys=['area', 'image_id', 'id'], >>> image_settings={'name': 'images', 'htype': 'link[image]', 'sample_compression': 'jpeg'}, >>> image_creds_key='my_s3_managed_credentials', >>> src_creds=aws_creds, # Can also be inferred from environment >>> dest_creds=aws_creds, # Can also be inferred from environment >>> connect_kwargs={'creds_key': 'my_s3_managed_credentials', 'org_id': 'org_id'}, >>> num_workers=4, >>> ) Copy to clipboard Parameters images_directory (str, pathlib.Path) – The path to the directory containing images .  annotation_files (str, pathlib.Path, List[str]) – Path to JSON annotation files in COCO format.  dest (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://org_id/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line), or pass in a token using the ‘token' parameter.  an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  key_to_tensor_mapping (Optional[Dict]) – A one-to-one mapping between COCO keys and Dataset tensor names.  file_to_group_mapping (Optional[Dict]) – A one-to-one mapping between COCO annotation file names and Dataset group names.  ignore_one_group (bool) – Skip creation of group in case of a single annotation file.  Set to False by default.  ignore_keys (List[str]) – A list of COCO keys to ignore.  image_params (Optional[Dict]) – A dictionary containing parameters for the images tensor.  image_creds_key (Optional[str]) – The name of the managed credentials to use for accessing the images in the linked tensor (is applicable).  src_creds (Optional[Union[str, Dict]]) – Credentials to access the source data.  If not provided, will be inferred from the environment.  dest_creds (Optional[Union[str, Dict]]) – The string ENV or a dictionary containing credentials used to access the destination path of the dataset.  inspect_limit (int) – The maximum number of samples to inspect in the annotations json, in order to generate the set of COCO annotation keys.  Set to 1000000 by default.  progressbar (bool) – Enables or disables ingestion progress bar.  Set to True by default.  shuffle (bool) – Shuffles the input data prior to ingestion.  Set to False by default.  num_workers (int) – The number of workers to use for ingestion.  Set to 0 by default.  token (Optional[str]) – The token to use for accessing the dataset and/or connecting it to Deep Lake.  connect_kwargs (Optional[Dict]) – If specified, the dataset will be connected to Deep Lake, and connect_kwargs will be passed to Dataset.connect.  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function.  See deeplake.empty().  Returns The Dataset created from images and COCO annotations


deeplake

dest_creds (Optional[Union[str, Dict]]) – The string ENV or a dictionary containing credentials used to access the destination path of the dataset .  kaggle_credentials (dict) – A dictionary containing kaggle credentials {'username':'YOUR_USERNAME', 'key': 'YOUR_KEY'}.  If None, environment variables/the kaggle.json file will be used if available.  progressbar (bool) – Enables or disables ingestion progress bar.  Set to True by default.  summary (bool) – Generates ingestion summary.  Set to True by default.  shuffle (bool) – Shuffles the input data prior to ingestion.  Since data arranged in folders by class is highly non-random, shuffling is important in order to produce optimal results when training.  Defaults to True.  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function.  See deeplake.dataset().  Returns New dataset object with structured dataset.  Return type Dataset Raises SamePathException – If the source and destination path are same.  Note Currently only local source paths and image classification datasets are supported for automatic ingestion.  deeplake.ingest_dataframe(src, dest: Union[str, Path], column_params: Optional[Dict] = None, src_creds: Optional[Union[str, Dict]] = None, dest_creds: Optional[Union[str, Dict]] = None, creds_key: Optional[Dict] = None, progressbar: bool = True, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) Convert pandas dataframe to a Deep Lake Dataset.  The contents of the dataframe can be parsed literally, or can be treated as links to local or cloud files.  Examples >>> # Ingest local data in COCO format to a Deep Lake dataset stored in Deep Lake storage.  Copy to clipboard >>> ds = deeplake.ingest_coco( >>> '<path/to/images/directory>', >>> ['path/to/annotation/file1.json', 'path/to/annotation/file2.json'], >>> dest='hub://org_id/dataset', >>> key_to_tensor_mapping={'category_id': 'labels', 'bbox': 'boxes'}, >>> file_to_group_mapping={'file1.json': 'group1', 'file2.json': 'group2'}, >>> ignore_keys=['area', 'image_id', 'id'], >>> num_workers=4, >>> ) >>> # Ingest data from your cloud into another Deep Lake dataset in your cloud, and connect that dataset to the Deep Lake backend.  Copy to clipboard >>> # Ingest data from a DataFrame into a Deep Lake dataset stored in Deep Lake storage.  >>> ds = deeplake.ingest_dataframe( >>> df, >>> dest='hub://org_id/dataset', >>> ) >>> # Ingest data from a DataFrame into a Deep Lake dataset stored in Deep Lake storage.  The filenames in `df_column_with_cloud_paths` will be used as the filenames for loading data into the dataset.  >>> ds = deeplake.ingest_dataframe( >>> df, >>> dest='hub://org_id/dataset', >>> column_params={'df_column_with_cloud_paths': {'name': 'images', 'htype': 'image'}}, >>> src_creds=aws_creds >>> ) >>> # Ingest data from a DataFrame into a Deep Lake dataset stored in Deep Lake storage.  The filenames in `df_column_with_cloud_paths` will be used as the filenames for linked data in the dataset.  >>> ds = deeplake.ingest_dataframe( >>> df, >>> dest='hub://org_id/dataset', >>> column_params={'df_column_with_cloud_paths': {'name': 'image_links', 'htype': 'link[image]'}}, >>> creds_key='my_s3_managed_credentials' >>> ) >>> # Ingest data from a DataFrame into a Deep Lake dataset stored in your cloud, and connect that dataset to the Deep Lake backend.  The filenames in `df_column_with_cloud_paths` will be used as the filenames for linked data in the dataset


deeplake

If not provided, will be inferred from the environment .  dest_creds (Optional[Union[str, Dict]]) – The string ENV or a dictionary containing credentials used to access the destination path of the dataset.  image_creds_key (Optional[str]) – creds_key for linked tensors, applicable if the htype for the images tensor is specified as ‘link[image]' in the ‘image_params' input.  inspect_limit (int) – The maximum number of annotations to inspect, in order to infer whether they are bounding boxes of polygons.  This in put is ignored if the htype is specfied in the ‘coordinates_params'.  progressbar (bool) – Enables or disables ingestion progress bar.  Set to True by default.  shuffle (bool) – Shuffles the input data prior to ingestion.  Set to False by default.  num_workers (int) – The number of workers to use for ingestion.  Set to 0 by default.  token (Optional[str]) – The token to use for accessing the dataset and/or connecting it to Deep Lake.  connect_kwargs (Optional[Dict]) – If specified, the dataset will be connected to Deep Lake, and connect_kwargs will be passed to Dataset.connect.  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function.  See deeplake.empty().  Returns The Dataset created from the images and YOLO annotations.  Return type Dataset Raises IngestionError – If annotations are not found for all the images and ‘allow_no_annotation' is False deeplake.ingest_kaggle(tag: str, src: Union[str, Path], dest: Union[str, Path], exist_ok: bool = False, images_compression: str = 'auto', dest_creds: Optional[Union[str, Dict]] = None, kaggle_credentials: Optional[dict] = None, progressbar: bool = True, summary: bool = True, shuffle: bool = True, **dataset_kwargs) → Dataset Download and ingest a kaggle dataset and store it as a structured dataset to destination.  Parameters tag (str) – Kaggle dataset tag.  Example: 'coloradokb/dandelionimages' points to https://www.kaggle.com/coloradokb/dandelionimages src (str, pathlib.Path) – Local path to where the raw kaggle dataset will be downlaoded to.  dest (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  exist_ok (bool) – If the kaggle dataset was already downloaded and exist_ok is True, ingestion will proceed without error.  images_compression (str) – For image classification datasets, this compression will be used for the images tensor.  If images_compression is 'auto', compression will be automatically determined by the most common extension in the directory


deeplake

}), .. .  validation: Dataset({ ...  features: ['data'] ...  }), ...  test: Dataset({ ...  features: ['data'] ...  }), ...  } Copy to clipboard it will be converted to a Deep Lake Dataset with tensors ['train/data', 'validation/data', 'test/data'].  Features of the type Sequence(feature=Value(dtype='string')) are not supported.  Columns of such type are skipped.  deeplake.load(path: Union[str, Path], read_only: Optional[bool] = None, memory_cache_size: int = 2000, local_cache_size: int = 0, creds: Optional[Union[dict, str]] = None, token: Optional[str] = None, org_id: Optional[str] = None, verbose: bool = True, access_method: str = 'stream', unlink: bool = False, reset: bool = False, check_integrity: bool = True, lock_timeout: Optional[int] = 0, lock_enabled: Optional[bool] = True, index_params: Optional[Dict[str, Union[int, str]]] = None) → Dataset Loads an existing dataset Examples >>> ds = deeplake.load('hub://username/dataset') >>> ds = deeplake.load('s3://mybucket/my_dataset') >>> ds = deeplake.load('./datasets/my_dataset', overwrite=True) Copy to clipboard Loading to a specfic version: >>> ds = deeplake.load('hub://username/dataset@new_branch') >>> ds = deeplake.load('hub://username/dataset@3e49cded62b6b335c74ff07e97f8451a37aca7b2) Copy to clipboard >>> my_commit_id = '3e49cded62b6b335c74ff07e97f8451a37aca7b2' >>> ds = deeplake.load(f'hub://username/dataset@{my_commit_id}') Copy to clipboard Parameters path (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  Loading to a specific version: You can also specify a commit_id or branch to load the dataset to that version directly by using the @ symbol.  The path will then be of the form hub://username/dataset@{branch} or hub://username/dataset@{commit_id}.  See examples above.  read_only (bool, optional) – Opens dataset in read only mode if this is passed as True.  Defaults to False.  Datasets stored on Deep Lake cloud that your account does not have write access to will automatically open in read mode.  memory_cache_size (int) – The size of the memory cache to be used in MB.  local_cache_size (int) – The size of the local filesystem cache to be used in MB.  creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets


deeplake

>>> ds = deeplake.ingest_dataframe( >>> df, >>> dest='s3://bucket/dataset_name', >>> column_params={'df_column_with_cloud_paths': {'name': 'image_links', 'htype': 'link[image]'}}, >>> creds_key='my_s3_managed_credentials' >>> connect_kwargs={'creds_key': 'my_s3_managed_credentials', 'org_id': 'org_id'}, >>> ) Copy to clipboard Parameters src (pd.DataFrame) – The pandas dataframe to be converted .  dest (str, pathlib.Path) – A Dataset or The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://username/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  column_params (Optional[Dict]) – A dictionary containing parameters for the tensors corresponding to the dataframe columns.  src_creds (Optional[Union[str, Dict]]) – Credentials to access the source data.  If not provided, will be inferred from the environment.  dest_creds (Optional[Union[str, Dict]]) – The string ENV or a dictionary containing credentials used to access the destination path of the dataset.  creds_key (Optional[str]) – creds_key for linked tensors, applicable if the htype any tensor is specified as ‘link[…]' in the ‘column_params' input.  progressbar (bool) – Enables or disables ingestion progress bar.  Set to True by default.  token (Optional[str]) – The token to use for accessing the dataset.  connect_kwargs (Optional[Dict]) – A dictionary containing arguments to be passed to the dataset connect method.  See Dataset.connect().  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function.  See deeplake.empty().  Returns New dataset created from the dataframe.  Return type Dataset Raises Exception – If src is not a valid pandas dataframe object.  deeplake.ingest_huggingface(src, dest, use_progressbar=True, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) → Dataset Converts Hugging Face datasets to Deep Lake format.  Parameters src (hfDataset, DatasetDict) – Hugging Face Dataset or DatasetDict to be converted.  Data in different splits of a DatasetDict will be stored under respective tensor groups.  dest (Dataset, str, pathlib.Path) – Destination dataset or path to it.  use_progressbar (bool) – Defines if progress bar should be used to show conversion progress.  token (Optional[str]) – The token to use for accessing the dataset and/or connecting it to Deep Lake.  connect_kwargs (Optional[Dict]) – If specified, the dataset will be connected to Deep Lake, and connect_kwargs will be passed to Dataset.connect.  **dataset_kwargs – Any arguments passed here will be forwarded to the dataset creator function.  See deeplake.empty().  Returns The destination Deep Lake dataset.  Return type Dataset Raises ValueError – If dest is not a path or a Deep Lake Dataset.  Note if DatasetDict looks like: >>> { ...  train: Dataset({ ...  features: ['data'] ..


deeplake

For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones .  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  org_id (str, Optional) – Organization id to be used for enabling high-performance features.  Only applicable for local datasets.  verbose (bool) – If True, logs will be printed.  Defaults to True.  access_method (str) – The access method to use for the dataset.  Can be: 'stream' Streams the data from the dataset i.e.  only fetches data when required.  This is the default value.  'download' Downloads the data to the local filesystem to the path specified in environment variable DEEPLAKE_DOWNLOAD_PATH.  This will overwrite DEEPLAKE_DOWNLOAD_PATH.  Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set or if the dataset does not exist.  The ‘download' access method can be modified to specify num_workers and/or scheduler.  For example: ‘download:2:processed' will use 2 workers and use processed scheduler, while ‘download:3' will use 3 workers and default scheduler (threaded), and ‘download:processed' will use a single worker and use processed scheduler.  'local' Downloads the dataset if it doesn't already exist, otherwise loads from local storage.  Raises an exception if DEEPLAKE_DOWNLOAD_PATH environment variable is not set.  The ‘local' access method can be modified to specify num_workers and/or scheduler to be used in case dataset needs to be downloaded.  If dataset needs to be downloaded, ‘local:2:processed' will use 2 workers and use processed scheduler, while ‘local:3' will use 3 workers and default scheduler (threaded), and ‘local:processed' will use a single worker and use processed scheduler.  unlink (bool) – Downloads linked samples if set to True.  Only applicable if access_method is download or local.  Defaults to False.  reset (bool) – If the specified dataset cannot be loaded due to a corrupted HEAD state of the branch being loaded, setting reset=True will reset HEAD changes and load the previous version.  check_integrity (bool) – If the param is True it will do integrity check during dataset loading otherwise the check is not performed Returns Dataset loaded using the arguments provided.  Return type Dataset Raises DatasetHandlerError – If a Dataset does not exist at the given path


deeplake

Examples >>> deeplake.rename('hub://username/image_ds', 'hub://username/new_ds') >>> deeplake.rename('s3://mybucket/my_ds', 's3://mybucket/renamed_ds') Copy to clipboard Parameters old_path (str, pathlib.Path) – The path to the dataset to be renamed .  new_path (str, pathlib.Path) – Path to the dataset after renaming.  creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  Returns The renamed Dataset.  Return type Dataset Raises DatasetHandlerError – If a Dataset does not exist at the given path or if new path is to a different directory.  deeplake.copy(src: Union[str, Path, Dataset], dest: Union[str, Path], runtime: Optional[dict] = None, tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, dest_creds=None, token=None, num_workers: int = 0, scheduler='threaded', progressbar=True, **kwargs) Copies dataset at src to dest.  Version control history is not included.  Parameters src (str, Dataset, pathlib.Path) – The Dataset or the path to the dataset to be copied.  dest (str, pathlib.Path) – Destination path to copy to.  runtime (dict) – Parameters for Activeloop DB Engine.  Only applicable for hub:// paths.  tensors (List[str], optional) – Names of tensors (and groups) to be copied.  If not specified all tensors are copied.  overwrite (bool) – If True and a dataset exists at dest, it will be overwritten.  Defaults to False.  src_creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  dest_creds (dict, optional) – creds required to create / overwrite datasets at dest.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  num_workers (int) – The number of workers to use for copying


deeplake

verbose (bool) – If True, logs will be printed .  Defaults to True.  **kwargs – Additional keyword arguments Returns New dataset object.  Return type Dataset Raises DatasetHandlerError – If a dataset already exists at destination path and overwrite is False.  TypeError – If source is not a dataset.  UnsupportedParameterException – If parameter that is no longer supported is beeing called.  DatasetCorruptError – If loading source dataset fails with DatasetCorruptedError deeplake.connect(src_path: str, creds_key: str, dest_path: Optional[str] = None, org_id: Optional[str] = None, ds_name: Optional[str] = None, token: Optional[str] = None) → Dataset Connects dataset at src_path to Deep Lake via the provided path.  Examples >>> # Connect an s3 dataset >>> ds = deeplake.connect(src_path='s3://bucket/dataset', dest_path='hub://my_org/dataset', creds_key='my_managed_credentials_key', token='my_activeloop_token') >>> # or >>> ds = deeplake.connect(src_path='s3://bucket/dataset', org_id='my_org', creds_key='my_managed_credentials_key', token='my_activeloop_token') Copy to clipboard Parameters src_path (str) – Cloud path to the source dataset.  Can be: an s3 path like s3://bucket/path/to/dataset.  a gcs path like gcs://bucket/path/to/dataset.  an azure path like az://account_name/container/path/to/dataset.  creds_key (str) – The managed credentials to be used for accessing the source path.  dest_path (str, optional) – The full path to where the connected Deep Lake dataset will reside.  Can be: a Deep Lake path like hub://organization/dataset org_id (str, optional) – The organization to where the connected Deep Lake dataset will be added.  ds_name (str, optional) – The name of the connected Deep Lake dataset.  Will be infered from dest_path or src_path if not provided.  token (str, optional) – Activeloop token used to fetch the managed credentials.  Returns The connected Deep Lake dataset.  Return type Dataset Raises InvalidSourcePathError – If the src_path is not a valid s3, gcs or azure path.  InvalidDestinationPathError – If dest_path, or org_id and ds_name do not form a valid Deep Lake path.  TokenPermissionError – If the user does not have permission to create a dataset in the specified organization.  deeplake.exists(path: Union[str, Path], creds: Optional[Union[str, Dict]] = None, token: Optional[str] = None) → bool Checks if a dataset exists at the given path.  Parameters path (str, pathlib.Path) – the path which needs to be checked.  creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated


deeplake

AgreementError – When agreement is rejected UserNotLoggedInException – When user is not logged in InvalidTokenException – If the specified toke is invalid TokenPermissionError – When there are permission or other errors related to token CheckoutError – If version address specified in the path cannot be found DatasetCorruptError – If loading the dataset failed due to corruption and reset is not True ReadOnlyModeError – If reset is attempted in read-only mode LockedException – When attempting to open a dataset for writing when it is locked by another machine ValueError – If org_id is specified for a non-local dataset Exception – Re-raises caught exception if reset cannot fix the issue ValueError – If the org id is provided but the dataset is not local Warning Setting access_method to download will overwrite the local copy of the dataset if it was previously downloaded .  Note Any changes made to the dataset in download / local mode will only be made to the local copy and will not be reflected in the original dataset.  deeplake.delete(path: Union[str, Path], force: bool = False, large_ok: bool = False, creds: Optional[Union[dict, str]] = None, token: Optional[str] = None, verbose: bool = False) → None Deletes a dataset at a given path.  Parameters path (str, pathlib.Path) – The path to the dataset to be deleted.  force (bool) – Delete data regardless of whether it looks like a deeplake dataset.  All data at the path will be removed if set to True.  large_ok (bool) – Delete datasets larger than 1GB.  Disabled by default.  creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  verbose (bool) – If True, logs will be printed.  Defaults to True.  Raises DatasetHandlerError – If a Dataset does not exist at the given path and force = False.  UserNotLoggedInException – When user is not logged in.  NotImplementedError – When attempting to delete a managed view.  ValueError – If version is specified in the path Warning This is an irreversible operation.  Data once deleted cannot be recovered.  deeplake.rename(old_path: Union[str, Path], new_path: Union[str, Path], creds: Optional[Union[dict, str]] = None, token: Optional[str] = None) → Dataset Renames dataset at old_path to new_path


deeplake

Defaults to 0 .  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str) – The scheduler to be used for copying.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool) – Displays a progress bar if True (default).  **kwargs (dict) – Additional keyword arguments Returns New dataset object.  Return type Dataset Raises DatasetHandlerError – If a dataset already exists at destination path and overwrite is False.  UnsupportedParameterException – If a parameter that is no longer supported is specified.  DatasetCorruptError – If loading source dataset fails with DatasetCorruptedError.  deeplake.deepcopy(src: Union[str, Path, Dataset], dest: Union[str, Path], runtime: Optional[Dict] = None, tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, dest_creds=None, token=None, num_workers: int = 0, scheduler='threaded', progressbar=True, public: bool = False, verbose: bool = True, **kwargs) Copies dataset at src to dest including version control history.  Parameters src (str, pathlib.Path, Dataset) – The Dataset or the path to the dataset to be copied.  dest (str, pathlib.Path) – Destination path to copy to.  runtime (dict) – Parameters for Activeloop DB Engine.  Only applicable for hub:// paths.  tensors (List[str], optional) – Names of tensors (and groups) to be copied.  If not specified all tensors are copied.  overwrite (bool) – If True and a dataset exists at destination, it will be overwritten.  Defaults to False.  src_creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  dest_creds (dict, optional) – creds required to create / overwrite datasets at dest.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  num_workers (int) – The number of workers to use for copying.  Defaults to 0.  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str) – The scheduler to be used for copying.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool) – Displays a progress bar if True (default).  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to False


deeplake

Returns A boolean confirming whether the dataset exists or not at the given path .  Raises ValueError – If version is specified in the path deeplake.read(path: Union[str, Path], verify: bool = False, creds: Optional[Dict] = None, compression: Optional[str] = None, storage: Optional[StorageProvider] = None) → Sample Utility that reads raw data from supported files into Deep Lake format.  Recompresses data into format required by the tensor if permitted by the tensor htype.  Simply copies the data in the file if file format matches sample_compression of the tensor, thus maximizing upload speeds.  Examples >>> ds.create_tensor('images', htype='image', sample_compression='jpeg') >>> ds.images.append(deeplake.read('path/to/cat.jpg')) >>> ds.images.shape (1, 399, 640, 3) Copy to clipboard >>> ds.create_tensor('videos', htype='video', sample_compression='mp4') >>> ds.videos.append(deeplake.read('path/to/video.mp4')) >>> ds.videos.shape (1, 136, 720, 1080, 3) Copy to clipboard >>> ds.create_tensor('images', htype='image', sample_compression='jpeg') >>> ds.images.append(deeplake.read('https://picsum.photos/200/300')) >>> ds.images[0].shape (300, 200, 3) Copy to clipboard Supported file types: Image: 'bmp', 'dib', 'gif', 'ico', 'jpeg', 'jpeg2000', 'pcx', 'png', 'ppm', 'sgi', 'tga', 'tiff', 'webp', 'wmf', 'xbm' Audio: 'flac', 'mp3', 'wav' Video: 'mp4', 'mkv', 'avi' Dicom: 'dcm' Nifti: 'nii', 'nii.gz' Copy to clipboard Parameters path (str) – Path to a supported file.  verify (bool) – If True, contents of the file are verified.  creds (optional, Dict) – Credentials for s3, gcp and http urls.  compression (optional, str) – Format of the file.  Only required if path does not have an extension.  storage (optional, StorageProvider) – Storage provider to use to retrieve remote files.  Useful if multiple files are being read from same storage to minimize overhead of creating a new provider.  Returns Sample object.  Call sample.array to get the np.ndarray.  Return type Sample Note No data is actually loaded until you try to get a property of the returned Sample.  This is useful for passing along to Tensor.append and Tensor.extend.  deeplake.link(path: str, creds_key: Optional[str] = None) → LinkedSample Utility that stores a link to raw data.  Used to add data to a Deep Lake Dataset without copying it.  See Link htype.  Supported file types: Image: 'bmp', 'dib', 'gif', 'ico', 'jpeg', 'jpeg2000', 'pcx', 'png', 'ppm', 'sgi', 'tga', 'tiff', 'webp', 'wmf', 'xbm' Audio: 'flac', 'mp3', 'wav' Video: 'mp4', 'mkv', 'avi' Dicom: 'dcm' Nifti: 'nii', 'nii.gz' Copy to clipboard Parameters path (str) – Path to a supported file.  creds_key (optional, str) – The credential key to use to read data for this sample.  The actual credentials are fetched from the dataset.  Returns LinkedSample object that stores path and creds.  Return type LinkedSample Examples >>> ds = deeplake.dataset('test/test_ds') >>> ds.create_tensor('images', htype='link[image]', sample_compression='jpeg') >>> ds.images.append(deeplake.link('https://picsum.photos/200/300')) Copy to clipboard See more examples here.  deeplake.link_tiled(path_array: ndarray, creds_key: Optional[str] = None) → LinkedTiledSample Utility that stores links to multiple images that act as tiles and together form a big image.  These images must all have the exact same dimensions.  Used to add data to a Deep Lake Dataset without copying it.  See Link htype


deeplake

Supported file types: Image: 'bmp', 'dib', 'gif', 'ico', 'jpeg', 'jpeg2000', 'pcx', 'png', 'ppm', 'sgi', 'tga', 'tiff', 'webp', 'wmf', 'xbm' Copy to clipboard Parameters path_array (np.ndarray) – N dimensional array of paths to the data, with paths corresponding to respective tiles .  The array must have dtype=object and have string values.  Each string must point to an image file with the same dimensions.  creds_key (optional, str) – The credential key to use to read data for this sample.  The actual credentials are fetched from the dataset.  Returns LinkedTiledSample object that stores path_array and creds.  Return type LinkedTiledSample Examples >>> ds = deeplake.dataset('test/test_ds') >>> ds.create_tensor('images', htype='link[image]', sample_compression='jpeg') >>> arr = np.empty((10, 10), dtype=object) >>> for j, i in itertools.product(range(10), range(10)): ...  arr[j, i] = f's3://my_bucket/my_image_{j}_{i}.jpeg' ...  >>> ds.images.append(deeplake.link_tiled(arr, creds_key='my_s3_key')) >>> # If all images are 1000x1200x3, we now have a 10000x12000x3 image in our dataset.  Copy to clipboard deeplake.tiled(sample_shape: Tuple[int, ...], tile_shape: Optional[Tuple[int, ...]] = None, dtype: Union[str, dtype] = dtype('uint8')) Allocates an empty sample of shape sample_shape, broken into tiles of shape tile_shape (except for edge tiles).  Example >>> with ds: ...  ds.create_tensor('image', htype='image', sample_compression='png') ...  ds.image.append(deeplake.tiled(sample_shape=(1003, 1103, 3), tile_shape=(10, 10, 3))) ...  ds.image[0][-217:, :212, 1:] = np.random.randint(0, 256, (217, 212, 2), dtype=np.uint8) Copy to clipboard Parameters sample_shape (Tuple[int, ...]) – Full shape of the sample.  tile_shape (Optional, Tuple[int, ...]) – The sample will be will stored as tiles where each tile will have this shape (except edge tiles).  If not specified, it will be computed such that each tile is close to half of the tensor's max_chunk_size (after compression).  dtype (Union[str, np.dtype]) – Dtype for the sample array.  Default uint8.  Returns A PartialSample instance which can be appended to a Tensor.  Return type PartialSample deeplake.compute(fn, name: Optional[str] = None) → Callable[[...], ComputeFunction] Compute is a decorator for functions.  The functions should have atleast 2 argument, the first two will correspond to sample_in and samples_out.  There can be as many other arguments as required.  The output should be appended/extended to the second argument in a deeplake like syntax.  Any value returned by the fn will be ignored.  Example: @deeplake.compute def my_fn(sample_in: Any, samples_out, my_arg0, my_arg1=0): samples_out.my_tensor.append(my_arg0 * my_arg1) # This transform can be used using the eval method in one of these 2 ways:- # Directly evaluating the method # here arg0 and arg1 correspond to the 3rd and 4th argument in my_fn my_fn(arg0, arg1).eval(data_in, ds_out, scheduler='threaded', num_workers=5) # As a part of a Transform pipeline containing other functions pipeline = deeplake.compose([my_fn(a, b), another_function(x=2)]) pipeline.eval(data_in, ds_out, scheduler='processed', num_workers=2) Copy to clipboard The eval method evaluates the pipeline/transform function.  It has the following arguments: data_in: Input passed to the transform to generate output dataset.  It should support __getitem__ and __len__.  This can be a Deep Lake dataset.  ds_out (Dataset, optional): The dataset object to which the transform will get written


deeplake

If this is not provided, data_in will be overwritten if it is a Deep Lake dataset, otherwise error will be raised .  It should have all keys being generated in output already present as tensors.  It's initial state should be either: Empty i.e.  all tensors have no samples.  In this case all samples are added to the dataset.  All tensors are populated and have same length.  In this case new samples are appended to the dataset.  num_workers (int): The number of workers to use for performing the transform.  Defaults to 0.  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str): The scheduler to be used to compute the transformation.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool): Displays a progress bar if True (default).  skip_ok (bool): If True, skips the check for output tensors generated.  This allows the user to skip certain tensors in the function definition.  This is especially useful for inplace transformations in which certain tensors are not modified.  Defaults to False.  ignore_errors (bool): If True, input samples that causes transform to fail will be skipped and the errors will be ignored if possible.  It raises the following errors: InvalidInputDataError: If data_in passed to transform is invalid.  It should support __getitem__ and __len__ operations.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as data_in will also raise this.  InvalidOutputDatasetError: If all the tensors of ds_out passed to transform don't have the same length.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as ds_out will also raise this.  TensorMismatchError: If one or more of the outputs generated during transform contain different tensors than the ones present in ‘ds_out' provided to transform.  UnsupportedSchedulerError: If the scheduler passed is not recognized.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  TransformError: All other exceptions raised if there are problems while running the pipeline.


deeplake

If this is not provided, data_in will be overwritten if it is a Deep Lake dataset, otherwise error will be raised .  It should have all keys being generated in output already present as tensors.  It's initial state should be either: Empty i.e.  all tensors have no samples.  In this case all samples are added to the dataset.  All tensors are populated and have same length.  In this case new samples are appended to the dataset.  num_workers (int): The number of workers to use for performing the transform.  Defaults to 0.  When set to 0, it will always use serial processing, irrespective of the scheduler.  scheduler (str): The scheduler to be used to compute the transformation.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  Defaults to ‘threaded'.  progressbar (bool): Displays a progress bar if True (default).  skip_ok (bool): If True, skips the check for output tensors generated.  This allows the user to skip certain tensors in the function definition.  This is especially useful for inplace transformations in which certain tensors are not modified.  Defaults to False.  check_lengths (bool): If True, checks whether ds_out has tensors of same lengths initially.  pad_data_in (bool): If True, pads tensors of data_in to match the length of the largest tensor in data_in.  Defaults to False.  ignore_errors (bool): If True, input samples that causes transform to fail will be skipped and the errors will be ignored if possible.  Note pad_data_in is only applicable if data_in is a Deep Lake dataset.  It raises the following errors: InvalidInputDataError: If data_in passed to transform is invalid.  It should support __getitem__ and __len__ operations.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as data_in will also raise this.  InvalidOutputDatasetError: If all the tensors of ds_out passed to transform don't have the same length.  Using scheduler other than 'threaded' with deeplake dataset having base storage as memory as ds_out will also raise this.  TensorMismatchError: If one or more of the outputs generated during transform contain different tensors than the ones present in ds_out provided to transform.  UnsupportedSchedulerError: If the scheduler passed is not recognized.  Supported values include: ‘serial', ‘threaded', ‘processed' and ‘ray'.  TransformError: All other exceptions raised if there are problems while running the pipeline.  deeplake.compose(functions: List[ComputeFunction]) Takes a list of functions decorated using deeplake.compute() and creates a pipeline that can be evaluated using .eval Example: pipeline = deeplake.compose([my_fn(a=3), another_function(b=2)]) pipeline.eval(data_in, ds_out, scheduler='processed', num_workers=2) Copy to clipboard The eval method evaluates the pipeline/transform function.  It has the following arguments: data_in: Input passed to the transform to generate output dataset.  It should support __getitem__ and __len__.  This can be a Deep Lake dataset.  ds_out (Dataset, optional): The dataset object to which the transform will get written


deeplake

creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path .  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  token (str, optional) – Activeloop token, used for fetching credentials to the dataset at path if it is a Deep Lake dataset.  This is optional, tokens are normally autogenerated.  org_id (str, Optional) – Organization id to be used for enabling high-performance features.  Only applicable for local datasets.  public (bool) – Defines if the dataset will have public access.  Applicable only if Deep Lake cloud storage is used and a new Dataset is being created.  Defaults to False.  verbose (bool) – If True, logs will be printed.  Defaults to True.  Returns New dataset object.  Return type Dataset Raises ValueError – If org_id is specified for a non-local dataset.  deeplake.ingest_classification(src: Union[str, Path], dest: Union[str, Path], image_params: Optional[Dict] = None, label_params: Optional[Dict] = None, dest_creds: Optional[Union[str, Dict]] = None, progressbar: bool = True, summary: bool = True, num_workers: int = 0, shuffle: bool = True, token: Optional[str] = None, connect_kwargs: Optional[Dict] = None, **dataset_kwargs) → Dataset Ingest a dataset of images from a local folder to a Deep Lake Dataset.  Images should be stored in subfolders by class name.  Parameters src (str, pathlib.Path) – Local path to where the unstructured dataset of images is stored or path to csv file.  dest (str, pathlib.Path) – The full path to the dataset.  Can be: a Deep Lake cloud path of the form hub://org_id/datasetname.  To write to Deep Lake cloud datasets, ensure that you are logged in to Deep Lake (use ‘activeloop login' from command line) an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  image_params (Optional[Dict]) – A dictionary containing parameters for the images tensor.  label_params (Optional[Dict]) – A dictionary containing parameters for the labels tensor.  dest_creds (Optional[Union[str, Dict]]) – The string ENV or a dictionary containing credentials used to access the destination path of the dataset.  progressbar (bool) – Enables or disables ingestion progress bar.  Defaults to True


deeplake.VectorStore

class deeplake.core.vectorstore.deeplake_vectorstore.VectorStore(path: ~typing.Union[str, ~pathlib.Path], tensor_params: ~typing.List[~typing.Dict[str, object]] = [{'name': 'text', 'htype': 'text', 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': False}, {'name': 'metadata', 'htype': 'json', 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': False}, {'name': 'embedding', 'htype': 'embedding', 'dtype': <class 'numpy.float32'>, 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': True, 'max_chunk_size': 64000000}, {'name': 'id', 'htype': 'text', 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': False}], embedding_function: ~typing.Optional[~typing.Any] = None, read_only: ~typing.Optional[bool] = None, ingestion_batch_size: int = 1000, index_params: ~typing.Optional[~typing.Dict[str, ~typing.Union[int, str]]] = None, num_workers: int = 0, exec_option: str = 'auto', token: ~typing.Optional[str] = None, overwrite: bool = False, verbose: bool = True, runtime: ~typing.Optional[~typing.Dict] = None, creds: ~typing.Optional[~typing.Union[str, ~typing.Dict]] = None, org_id: ~typing.Optional[str] = None, logger: ~logging.Logger = <Logger deeplake.core.vectorstore.deeplake_vectorstore (INFO)>, branch: str = 'main', **kwargs: ~typing.Any) Bases: object Base class for VectorStore __init__(path: ~typing.Union[str, ~pathlib.Path], tensor_params: ~typing.List[~typing.Dict[str, object]] = [{'name': 'text', 'htype': 'text', 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': False}, {'name': 'metadata', 'htype': 'json', 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': False}, {'name': 'embedding', 'htype': 'embedding', 'dtype': <class 'numpy.float32'>, 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': True, 'max_chunk_size': 64000000}, {'name': 'id', 'htype': 'text', 'create_id_tensor': False, 'create_sample_info_tensor': False, 'create_shape_tensor': False}], embedding_function: ~typing.Optional[~typing.Any] = None, read_only: ~typing.Optional[bool] = None, ingestion_batch_size: int = 1000, index_params: ~typing.Optional[~typing.Dict[str, ~typing.Union[int, str]]] = None, num_workers: int = 0, exec_option: str = 'auto', token: ~typing.Optional[str] = None, overwrite: bool = False, verbose: bool = True, runtime: ~typing.Optional[~typing.Dict] = None, creds: ~typing.Optional[~typing.Union[str, ~typing.Dict]] = None, org_id: ~typing.Optional[str] = None, logger: ~logging.Logger = <Logger deeplake.core.vectorstore.deeplake_vectorstore (INFO)>, branch: str = 'main', **kwargs: ~typing.Any) → None Creates an empty VectorStore or loads an existing one if it exists at the specified path.  Examples >>> # Create a vector store with default tensors >>> data = VectorStore( ...  path = './my_vector_store', ...  ) >>> # Create a vector store in the Deep Lake Managed Tensor Database >>> data = VectorStore( ...  path = 'hub://org_id/dataset_name', ...  runtime = {'tensor_db': True}, ...  ) >>> # Create a vector store with custom tensors >>> data = VectorStore( ...  path = './my_vector_store', ...  tensor_params = [{'name': 'text', 'htype': 'text'}, ...  {'name': 'embedding_1', 'htype': 'embedding'}, ...  {'name': 'embedding_2', 'htype': 'embedding'}, ...  {'name': 'source', 'htype': 'text'}, ...  {'name': 'metadata', 'htype': 'json'} ...  ] ...  ) Copy to clipboard Parameters path (str, pathlib.Path) – The full path for storing to the Deep Lake Vector Store.  It can be: a Deep Lake cloud path of the form hub://org_id/dataset_name.  Requires registration with Deep Lake.  an s3 path of the form s3://bucketname/path/to/dataset.  Credentials are required in either the environment or passed to the creds argument.  a local file system path of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.  a memory path of the form mem://path/to/dataset which doesn't save the dataset but keeps it in memory instead.  Should be used only for testing as it does not persist.  tensor_params (List[Dict[str, dict]], optional) – List of dictionaries that contains information about tensors that user wants to create.  See create_tensor in Deep Lake API docs for more information


deeplake.VectorStore

- If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file .  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  runtime (Dict, optional) – Parameters for creating the Vector Store in Deep Lake's Managed Tensor Database.  Not applicable when loading an existing Vector Store.  To create a Vector Store in the Managed Tensor Database, set runtime = {'tensor_db': True}.  branch (str) – Branch name to use for the Vector Store.  Defaults to 'main'.  **kwargs (Any) – Additional keyword arguments.  Danger Setting overwrite to True will delete all of your data if the Vector Store exists! Be very careful when setting this parameter.  add(embedding_function: Optional[Union[Callable, List[Callable]]] = None, embedding_data: Optional[Union[List, List[List]]] = None, embedding_tensor: Optional[Union[str, List[str]]] = None, return_ids: bool = False, rate_limiter: Dict = {'batch_byte_size': 10000, 'bytes_per_minute': 1800000.0, 'enabled': False}, **tensors) → Optional[List[str]] Adding elements to deeplake vector store.  Tensor names are specified as parameters, and data for each tensor is specified as parameter values.  All data must of equal length.  Examples >>> # Dummy data >>> texts = ['Hello', 'World'] >>> embeddings = [[1, 2, 3], [4, 5, 6]] >>> metadatas = [{'timestamp': '01:20'}, {'timestamp': '01:22'}] >>> emebdding_fn = lambda x: [[1, 2, 3]] * len(x) >>> embedding_fn_2 = lambda x: [[4, 5]] * len(x) >>> # Directly upload embeddings >>> deeplake_vector_store.add( ...  text = texts, ...  embedding = embeddings, ...  metadata = metadatas, ...  ) >>> # Upload embedding via embedding function >>> deeplake_vector_store.add( ...  text = texts, ...  metadata = metadatas, ...  embedding_function = embedding_fn, ...  embedding_data = texts, ...  ) >>> # Upload embedding via embedding function to a user-defined embedding tensor >>> deeplake_vector_store.add( ...  text = texts, ...  metadata = metadatas, ...  embedding_function = embedding_fn, ...  embedding_data = texts, ...  embedding_tensor = 'embedding_1', ...  ) >>> # Multiple embedding functions (user defined embedding tensors must be specified) >>> deeplake_vector_store.add( ...  embedding_tensor = ['embedding_1', 'embedding_2'] ...  embedding_function = [embedding_fn, embedding_fn_2], ...  embedding_data = [texts, texts], ...  ) >>> # Alternative syntax for multiple embedding functions >>> deeplake_vector_store.add( ...  text = texts, ...  metadata = metadatas, ...  embedding_tensor_1 = (embedding_fn, texts), ...  embedding_tensor_2 = (embedding_fn_2, texts), ...  ) >>> # Add data to fully custom tensors >>> deeplake_vector_store.add( ...  tensor_A = [1, 2], ...  tensor_B = ['a', 'b'], ...  tensor_C = ['some', 'data'], ...  embedding_function = embedding_fn, ...  embedding_data = texts, ...  embedding_tensor = 'embedding_1', ..


deeplake.VectorStore

Defaults to DEFAULT_VECTORSTORE_TENSORS .  embedding_function (Optional[Any], optional) – Function or class that converts the embeddable data into embeddings.  Input to embedding_function is a list of data and output is a list of embeddings.  Defaults to None.  read_only (bool, optional) – Opens dataset in read-only mode if True.  Defaults to False.  num_workers (int) – Number of workers to use for parallel ingestion.  ingestion_batch_size (int) – Batch size to use for parallel ingestion.  index_params (Dict[str, Union[int, str]]) – Dictionary containing information about vector index that will be created.  Defaults to None, which will utilize DEFAULT_VECTORSTORE_INDEX_PARAMS from deeplake.constants.  The specified key-values override the default ones.  - threshold: The threshold for the dataset size above which an index will be created for the embedding tensor.  When the threshold value is set to -1, index creation is turned off.  Defaults to -1, which turns off the index.  distance_metric: This key specifies the method of calculating the distance between vectors when creating the vector database (VDB) index.  It can either be a string that corresponds to a member of the DistanceType enumeration, or the string value itself.  If no value is provided, it defaults to 'L2'.  'L2' corresponds to DistanceType.L2_NORM.  'COS' corresponds to DistanceType.COSINE_SIMILARITY.  additional_params: Additional parameters for fine-tuning the index.  exec_option (str) – Default method for search execution.  It could be either 'auto', 'python', 'compute_engine' or 'tensor_db'.  Defaults to 'auto'.  If None, it's set to 'auto'.  - auto- Selects the best execution method based on the storage location of the Vector Store.  It is the default option.  - python - Pure-python implementation that runs on the client and can be used for data stored anywhere.  WARNING: using this option with big datasets is discouraged because it can lead to memory issues.  - compute_engine - Performant C++ implementation of the Deep Lake Compute Engine that runs on the client and can be used for any data stored in or connected to Deep Lake.  It cannot be used with in-memory or local datasets.  - tensor_db - Performant and fully-hosted Managed Tensor Database that is responsible for storage and query execution.  Only available for data stored in the Deep Lake Managed Database.  Store datasets in this database by specifying runtime = {'tensor_db': True} during dataset creation.  token (str, optional) – Activeloop token, used for fetching user credentials.  This is Optional, tokens are normally autogenerated.  Defaults to None.  overwrite (bool) – If set to True this overwrites the Vector Store if it already exists.  Defaults to False.  verbose (bool) – Whether to print summary of the dataset created.  Defaults to True.  creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path


deeplake.VectorStore

) Copy to clipboard Parameters embedding_function (Optional[Callable]) – embedding function used to convert embedding_data into embeddings .  Input to embedding_function is a list of data and output is a list of embeddings.  Overrides the embedding_function specified when initializing the Vector Store.  embedding_data (Optional[List]) – Data to be converted into embeddings using the provided embedding_function.  Defaults to None.  embedding_tensor (Optional[str]) – Tensor where results from the embedding function will be stored.  If None, the embedding tensor is automatically inferred (when possible).  Defaults to None.  return_ids (bool) – Whether to return added ids as an ouput of the method.  Defaults to False.  rate_limiter (Dict) – Rate limiter configuration.  Defaults to {'enabled': False, 'bytes_per_minute': MAX_BYTES_PER_MINUTE, 'batch_byte_size': TARGET_BYTE_SIZE}.  **tensors – Keyword arguments where the key is the tensor name, and the value is a list of samples that should be uploaded to that tensor.  Returns List of ids if return_ids is set to True.  Otherwise, None.  Return type Optional[List[str]] checkout(branch: str = 'main') → None Checkout the Vector Store to a specific branch.  Parameters branch (str) – Branch name to checkout.  Defaults to 'main'.  commit(allow_empty: bool = True) → None Commits the Vector Store.  Parameters allow_empty (bool) – Whether to allow empty commits.  Defaults to True.  delete(row_ids: Optional[List[str]] = None, ids: Optional[List[str]] = None, filter: Optional[Union[Dict, Callable]] = None, query: Optional[str] = None, exec_option: Optional[str] = None, delete_all: Optional[bool] = None) → bool Delete the data in the Vector Store.  Does not delete the tensor definitions.  To delete the vector store completely, first run VectorStore.delete_by_path().  Examples >>> # Delete using ids: >>> data = vector_store.delete(ids) >>> # Delete data using filter >>> data = vector_store.delete( ...  filter = {'json_tensor_name': {'key: value'}, 'json_tensor_name_2': {'key_2: value_2'}}, ...  ) >>> # Delete data using TQL >>> data = vector_store.delete( ...  query = 'select * where .....  <add TQL syntax>', ...  exec_option = 'compute_engine', ...  ) Copy to clipboard Parameters ids (Optional[List[str]]) – List of unique ids.  Defaults to None.  row_ids (Optional[List[str]]) – List of absolute row indices from the dataset.  Defaults to None.  filter (Union[Dict, Callable], optional) – Filter for finding samples for deletion.  - Dict - Key-value search on tensors of htype json, evaluated on an AND basis (a sample must satisfy all key-value filters to be True) Dict = {'tensor_name_1': {'key': value}, 'tensor_name_2': {'key': value}} - Function - Any function that is compatible with deeplake.filter.  query (Optional[str]) – TQL Query string for direct evaluation for finding samples for deletion, without application of additional filters.  exec_option (Optional[str]) – Method for search execution.  It could be either 'python', 'compute_engine' or 'tensor_db'.  Defaults to None, which inherits the option from the Vector Store initialization.  - python - Pure-python implementation that runs on the client and can be used for data stored anywhere


deeplake.VectorStore

WARNING: using this option with big datasets is discouraged because it can lead to memory issues .  - compute_engine - Performant C++ implementation of the Deep Lake Compute Engine that runs on the client and can be used for any data stored in or connected to Deep Lake.  It cannot be used with in-memory or local datasets.  - tensor_db - Performant and fully-hosted Managed Tensor Database that is responsible for storage and query execution.  Only available for data stored in the Deep Lake Managed Database.  Store datasets in this database by specifying runtime = {'tensor_db': True} during dataset creation.  delete_all (Optional[bool]) – Whether to delete all the samples and version history of the dataset.  Defaults to None.  Returns Returns True if deletion was successful, otherwise it raises a ValueError.  Return type bool Raises ValueError – If neither ids, filter, query, nor delete_all are specified, or if an invalid exec_option is provided.  static delete_by_path(path: Union[str, Path], token: Optional[str] = None, force: bool = False, creds: Optional[Union[str, Dict]] = None) → None Deleted the Vector Store at the specified path.  Parameters path (str, pathlib.Path) – The full path to the Deep Lake Vector Store.  token (str, optional) – Activeloop token, used for fetching user credentials.  This is optional, as tokens are normally autogenerated.  Defaults to None.  creds (dict, str, optional) – The string ENV or a dictionary containing credentials used to access the dataset at the path.  - If ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token' are present, these take precedence over credentials present in the environment or in credentials file.  Currently only works with s3 paths.  - It supports ‘aws_access_key_id', ‘aws_secret_access_key', ‘aws_session_token', ‘endpoint_url', ‘aws_region', ‘profile_name' as keys.  - If ‘ENV' is passed, credentials are fetched from the environment variables.  This is also the case when creds is not passed for cloud datasets.  For datasets connected to hub cloud, specifying ‘ENV' will override the credentials fetched from Activeloop and use local ones.  force (bool) – delete the path in a forced manner without rising an exception.  Defaults to True.  Danger This method permanently deletes all of your data if the Vector Store exists! Be very careful when using this method.  search(embedding_data: Optional[Union[str, List[str]]] = None, embedding_function: Optional[Callable] = None, embedding: Optional[Union[List[float], ndarray]] = None, k: int = 4, distance_metric: Optional[str] = None, query: Optional[str] = None, filter: Optional[Union[Dict, Callable]] = None, exec_option: Optional[str] = None, embedding_tensor: str = 'embedding', return_tensors: Optional[List[str]] = None, return_view: bool = False, deep_memory: bool = False) → Union[Dict, Dataset] VectorStore search method that combines embedding search, metadata search, and custom TQL search.  Examples >>> # Search using an embedding >>> data = vector_store.search( ...  embedding = [1, 2, 3], ...  exec_option = 'python', ...  ) >>> # Search using an embedding function and data for embedding >>> data = vector_store.search( ...  embedding_data = 'What does this chatbot do?', ..


deeplake.VectorStore

embedding_function = query_embedding_fn, .. .  exec_option = 'compute_engine', ...  ) >>> # Add a filter to your search >>> data = vector_store.search( ...  embedding = np.ones(3), ...  exec_option = 'python', ...  filter = {'json_tensor_name': {'key: value'}, 'json_tensor_name_2': {'key_2: value_2'},...}, # Only valid for exec_option = 'python' ...  ) >>> # Search using TQL >>> data = vector_store.search( ...  query = 'select * where .....  <add TQL syntax>', ...  exec_option = 'tensor_db', # Only valid for exec_option = 'compute_engine' or 'tensor_db' ...  ) Copy to clipboard Parameters embedding (Union[np.ndarray, List[float]], optional) – Embedding representation for performing the search.  Defaults to None.  The embedding_data and embedding cannot both be specified.  embedding_data (List[str]) – Data against which the search will be performed by embedding it using the embedding_function.  Defaults to None.  The embedding_data and embedding cannot both be specified.  embedding_function (Optional[Callable], optional) – function for converting embedding_data into embedding.  Only valid if embedding_data is specified.  Input to embedding_function is a list of data and output is a list of embeddings.  k (int) – Number of elements to return after running query.  Defaults to 4.  distance_metric (str) – Distance metric to use for sorting the data.  Avaliable options are: 'L1', 'L2', 'COS', 'MAX'.  Defaults to None, which uses same distance metric specified in index_params.  If there is no index, it performs linear search using DEFAULT_VECTORSTORE_DISTANCE_METRIC.  query (Optional[str]) – TQL Query string for direct evaluation, without application of additional filters or vector search.  filter (Union[Dict, Callable], optional) – Additional filter evaluated prior to the embedding search.  Dict - Key-value search on tensors of htype json, evaluated on an AND basis (a sample must satisfy all key-value filters to be True) Dict = {'tensor_name_1': {'key': value}, 'tensor_name_2': {'key': value}} Function - Any function that is compatible with Dataset.filter.  exec_option (Optional[str]) – Method for search execution.  It could be either 'python', 'compute_engine' or 'tensor_db'.  Defaults to None, which inherits the option from the Vector Store initialization.  python - Pure-python implementation that runs on the client and can be used for data stored anywhere.  WARNING: using this option with big datasets is discouraged because it can lead to memory issues.  compute_engine - Performant C++ implementation of the Deep Lake Compute Engine that runs on the client and can be used for any data stored in or connected to Deep Lake.  It cannot be used with in-memory or local datasets.  tensor_db - Performant and fully-hosted Managed Tensor Database that is responsible for storage and query execution.  Only available for data stored in the Deep Lake Managed Database.  Store datasets in this database by specifying runtime = {'tensor_db': True} during dataset creation.  embedding_tensor (str) – Name of tensor with embeddings.  Defaults to 'embedding'.  return_tensors (Optional[List[str]]) – List of tensors to return data for


deeplake.VectorStore

Defaults to None, which returns data for all tensors except the embedding tensor (in order to minimize payload) .  To return data for all tensors, specify return_tensors = '*'.  return_view (bool) – Return a Deep Lake dataset view that satisfied the search parameters, instead of a dictionary with data.  Defaults to False.  If True return_tensors is set to '*' beucase data is lazy-loaded and there is no cost to including all tensors in the view.  deep_memory (bool) – Whether to use the Deep Memory model for improving search results.  Defaults to False if deep_memory is not specified in the Vector Store initialization.  If True, the distance metric is set to 'deepmemory_distance', which represents the metric with which the model was trained.  The search is performed using the Deep Memory model.  If False, the distance metric is set to 'COS' or whatever distance metric user specifies.  Raises ValueError – When invalid parameters are specified.  ValueError – when deep_memory is True.  Deep Memory is only available for datasets stored in the Deep Lake Managed Database for paid accounts.  DeepMemoryWaitingListError – if user is not waitlisted to use deep_memory.  Returns Dictionary where keys are tensor names and values are the results of the search Return type Dict summary() Prints a summary of the dataset tensors() Returns the list of tensors present in the dataset update_embedding(row_ids: Optional[List[str]] = None, ids: Optional[List[str]] = None, filter: Optional[Union[Dict, Callable]] = None, query: Optional[str] = None, exec_option: Optional[str] = None, embedding_function: Optional[Union[Callable, List[Callable]]] = None, embedding_source_tensor: Union[str, List[str]] = 'text', embedding_tensor: Optional[Union[str, List[str]]] = None) Recompute existing embeddings of the VectorStore, that match either query, filter, ids or row_ids.  Examples >>> # Update using ids: >>> data = vector_store.update( ...  ids, ...  embedding_source_tensor = 'text', ...  embedding_tensor = 'embedding', ...  embedding_function = embedding_function, ...  ) >>> # Update data using filter and several embedding_tensors, several embedding_source_tensors >>> # and several embedding_functions: >>> data = vector_store.update( ...  embedding_source_tensor = ['text', 'metadata'], ...  embedding_function = ['text_embedding_function', 'metadata_embedding_function'], ...  filter = {'json_tensor_name': {'key: value'}, 'json_tensor_name_2': {'key_2: value_2'}}, ...  embedding_tensor = ['text_embedding', 'metadata_embedding'] ...  ) >>> # Update data using TQL, if new embedding function is not specified the embedding_function used >>> # during initialization will be used >>> data = vector_store.update( ...  embedding_source_tensor = 'text', ...  query = 'select * where .....  <add TQL syntax>', ...  exec_option = 'compute_engine', ...  embedding_tensor = 'embedding_tensor', ...  ) Copy to clipboard Parameters row_ids (Optional[List[str]], optional) – Row ids of the elements for replacement.  Defaults to None.  ids (Optional[List[str]], optional) – hash ids of the elements for replacement.  Defaults to None.  filter (Optional[Union[Dict, Callable]], optional) – Filter for finding samples for replacement


deeplake.VectorStore

- Dict - Key-value search on tensors of htype json, evaluated on an AND basis (a sample must satisfy all key-value filters to be True) Dict = {'tensor_name_1': {'key': value}, 'tensor_name_2': {'key': value}} - Function - Any function that is compatible with deeplake.filter query (Optional[str], optional) – TQL Query string for direct evaluation for finding samples for deletion, without application of additional filters .  Defaults to None.  exec_option (Optional[str]) – Method for search execution.  It could be either 'python', 'compute_engine' or 'tensor_db'.  Defaults to None, which inherits the option from the Vector Store initialization.  - python - Pure-python implementation that runs on the client and can be used for data stored anywhere.  WARNING: using this option with big datasets is discouraged because it can lead to memory issues.  - compute_engine - Performant C++ implementation of the Deep Lake Compute Engine that runs on the client and can be used for any data stored in or connected to Deep Lake.  It cannot be used with in-memory or local datasets.  - tensor_db - Performant and fully-hosted Managed Tensor Database that is responsible for storage and query execution.  Only available for data stored in the Deep Lake Managed Database.  Store datasets in this database by specifying runtime = {'tensor_db': True} during dataset creation.  embedding_function (Optional[Union[Callable, List[Callable]]], optional) – function for converting embedding_source_tensor into embedding.  Only valid if embedding_source_tensor is specified.  Defaults to None.  embedding_source_tensor (Union[str, List[str]], optional) – Name of tensor with data that needs to be converted to embeddings.  Defaults to text.  embedding_tensor (Optional[Union[str, List[str]]], optional) – Name of the tensor with embeddings.  Defaults to None.


deeplake.core

deeplake.core.sample Sample deeplake.core.linked_sample LinkedSample deeplake.core.partial_sample PartialSample deeplake.core.link_tiled_sample LinkedTiledSample deeplake.core.storage Base Storage Provider LRU Cache S3 Storage Provider Google Cloud Storage Provider Google Drive Storage Provider Local Storage Provider Memory Provider deeplake.core.index IndexEntry Index merge_slices() slice_at_int() slice_length()


deeplake.core.dataset

Dataset class deeplake.core.dataset.Dataset add_creds_key(creds_key: str, managed: bool = False) Adds a new creds key to the dataset.  These keys are used for tensors that are linked to external data.  Examples >>> # create/load a dataset >>> ds = deeplake.empty('path/to/dataset') >>> # add a new creds key >>> ds.add_creds_key('my_s3_key') Copy to clipboard Parameters creds_key (str) – The key to be added.  managed (bool) – If True, the creds corresponding to the key will be fetched from Activeloop platform.  Defaults to False.  Raises ValueError – If the dataset is not connected to Activeloop platform and managed is True.  Note managed parameter is applicable only for datasets that are connected to Activeloop platform.  append(sample: Dict[str, Any], skip_ok: bool = False, append_empty: bool = False) Append samples to mutliple tensors at once.  This method expects all tensors being updated to be of the same length.  Parameters sample (dict) – Dictionary with tensor names as keys and samples as values.  skip_ok (bool) – Skip tensors not in sample if set to True.  append_empty (bool) – Append empty samples to tensors not specified in sample if set to True.  If True, skip_ok is ignored.  Raises KeyError – If any tensor in the dataset is not a key in sample and skip_ok is False.  TensorDoesNotExistError – If tensor in sample does not exist.  ValueError – If all tensors being updated are not of the same length.  NotImplementedError – If an error occurs while writing tiles.  Exception – Error while attempting to rollback appends.  SampleAppendingError – Error that occurs when someone tries to append a tensor value directly to the dataset without specifying tensor name.  Examples >>> ds = deeplake.empty('../test/test_ds') >>> ds.create_tensor('data') Tensor(key='data') >>> ds.create_tensor('labels') Tensor(key='labels') >>> ds.append({'data': [1, 2, 3, 4], 'labels':[0, 1, 2, 3]}) Copy to clipboard property branch: str The current branch of the dataset property branches Lists all the branches of the dataset.  Returns List of branches.  checkout(address: str, create: bool = False, reset: bool = False) → Optional[str] Checks out to a specific commit_id or branch.  If create = True, creates a new branch with name address.  Parameters address (str) – The commit_id or branch to checkout to.  create (bool) – If True, creates a new branch with name as address.  reset (bool) – If checkout fails due to a corrupted HEAD state of the branch, setting reset=True will reset HEAD changes and attempt the checkout again.  Returns The commit_id of the dataset after checkout.  Return type Optional[str] Raises CheckoutError – If address could not be found.  ReadOnlyModeError – If branch creation or reset is attempted in read-only mode.  DatasetCorruptError – If checkout failed due to dataset corruption and reset is not True.  Exception – If the dataset is a filtered view