Loss

The loss component calculates loss values during training, estimating the difference between model predictions and ground truth labels. Users can choose suitable loss functions (e.g., Focal Loss, GHMLoss, L1 Loss) for specific use cases to evaluate and improve the model's performance.


How to Train models using Deep Lake and PyTorch Lightning

This tutorial is also available as a Colab Notebook.Deep Lake's integration with PyTorch can also be used to train models using an integration with PyTorch Lightning, a popular open-source high-level interface for PyTorch.


Overview

At a high-level, Deep Lake is connected to PyTorch lightning by passing the Deep Lake's PyTorch dataloader to any PyTorch Lightning API that expects a dataloader parameter, such as trainer.fit(..., train_dataloaders = deeplake_dataloader). The only caveats are:Deep Lake handles distributed training via it's distributed parameter in the .pytorch() method. Therefore, the PyTorch Lightning Trainer class should be initialized with replace_sampler_ddp = False.


Example Code

This tutorial uses PyTorch Lightning to execute the identical training workflow that is shown here in PyTorch.


Data Preprocessing

The first step is to load the dataset for training. This tutorial uses the Fashion MNIST dataset that has already been converted into Deep Leake format. It is a simple image classification dataset that categorizes images by clothing type (trouser, shirt, etc.)import deeplakefrom PIL import Imageimport torchfrom torchvision import transforms, modelsimport pytorch_lightning as pl# Connect to the training and testing datasetsds_train = deeplake.load('hub://activeloop/fashion-mnist-train')ds_val = deeplake.load('hub://activeloop/fashion-mnist-test')The next step is to define a transformation function that will process the data and convert it into a format that can be passed into a deep learning model. In this particular example, torchvision.transforms is used as a part of the transformation pipeline that performs operations such as normalization and image augmentation (rotation).tform = transforms.Compose([ transforms.RandomRotation(20), # Image augmentation transforms.ToTensor(), # Must convert to pytorch tensor for subsequent operations to run transforms.Normalize([0.5], [0.5]),])You can now create a PyTorch dataloader that connects the Deep Lake dataset to the PyTorch model using the provided method ds.pytorch(). This method automatically applies the transformation function and takes care of random shuffling (if desired). The num_workers parameter can be used to parallelize data preprocessing, which is critical for ensuring that preprocessing does not bottleneck the overall training workflow.The transform input is a dictionary where the key is the tensor name and the value is the transformation function that should be applied to that tensor. If a specific tensor's data does not need to be returned, it should be omitted from the keys. If the transformation function is set as None, the input tensor is converted to a torch tensor without additional modification.batch_size = 32# Since torchvision transforms expect PIL images, we use the 'pil' decode_method for the 'images' tensor. This is much faster than running ToPILImage inside the transformtrain_loader = ds_train.pytorch(num_workers = 0, shuffle = True, transform = {'images': tform, 'labels': None}, batch_size = batch_size, decode_method = {'images': 'pil'})val_loader = ds_val.pytorch(num_workers = 0, transform = {'images': tform, 'labels': None}, batch_size = batch_size, decode_method = {'images': 'pil'})


Model and LightningModule Definition

This tutorial uses a pre-trained ResNet18 neural network from the torchvision.models module, converted to a single-channel network for grayscale images. The LightningModule organizes the training code.# Use a pre-trained ResNet18def get_model(num_classes): model = models.resnet18(pretrained=True) # Convert model to grayscale model.conv1 = torch.nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False) # Update the fully connected layer based on the number of classes in the dataset model.fc = torch.nn.Linear(model.fc.in_features, num_classes) return modelclass FashionMnistModule(pl.LightningModule): def __init__(self, num_classes): ''' Inputs: num_classes: Number of classes in the dataset and model ''' super().__init__() # Create the model self.model = get_model(num_classes) # Create loss module self.loss_module = torch.nn.CrossEntropyLoss() def forward(self, imgs): return self.model(imgs) def configure_optimizers(self): return torch.optim.SGD(self.model.parameters(), lr=0.001, momentum=0.1)  def training_step(self, batch, batch_idx): images = batch['images'] labels = torch.squeeze(batch['labels']) preds = self.model(images) loss = self.loss_module(preds, labels) acc = (preds.argmax(dim=-1) == labels).float().mean() self.log('train_acc', acc, on_step=True, on_epoch=True) self.log('train_loss', loss) return loss  def validation_step(self, batch, batch_idx): images = batch['images'] labels = torch.squeeze(batch['labels']) preds = self.model(images).argmax(dim=-1) acc = (labels == preds).float().mean() # Log the valdation accuracy to the progress bar at the end of each epoch self.log('val_acc', acc, on_epoch=True, prog_bar=True)


Training the Model

PyTorchLightning takes care of the training loop, so the remaining steps are to initialize the Trainer and call the .fit() method using the training and validation dataloaders.trainer = pl.Trainer(max_epochs = 3)trainer.fit(model=FashionMnistModule(len(ds_train.labels.info.class_names)), train_dataloaders = train_loader, val_dataloaders = val_loader)Congrats! You successfully trained an image classification model using PyTorch Lightning while streaming data directly from the cloud! 🎉


How to Train an PyTorch Image Classification Model on AWS SageMaker Using Deep Lake Datasets

AWS SageMaker provides scalable infrastructure for developing, training, and deploying deep learning models. In this tutorial, we demonstrate how to run SageMaker training jobs for training a PyTorch image classification model using a Deep Lake dataset. This tutorial will focus on the SageMaker integration, and less so on the details of the training (see other training tutorials for details)


Dataset

In this tutorial we will use the Stanford Cars Dataset, which classifies the make+model+year of various vehicles. Though the dataset contains bounding boxes, we ignore those and only use the data for classification purposes.


Training Script

ds_train = deeplake.load(args.train_dataset, read_only = True, token = args.token, creds = args.creds) ds_val = deeplake.load(args.val_dataset, read_only = True, token = args.token, creds = args.creds) train_loader = ds_train.dataloader()\ .batch(args.batch_size)\ .shuffle(args.shuffle)\ .transform(transform = {'images': tform_train, 'car_models': None})\ .pytorch(num_workers = args.num_workers, decode_method = {'images': 'pil'}) val_loader = ds_val.dataloader()\ .batch(args.batch_size)\ .transform(transform = {'images': tform_val, 'car_models': None})\ .pytorch(num_workers = args.num_workers, decode_method = {'images': 'pil'}) # Load the model model = get_model_classification(len(ds_train.car_models.info.class_names)) model = model.to(device) # Define the optimizer, loss, and learning rate scheduler optimizer = optim.Adam(model.parameters(), lr=args.lr) criterion = nn.CrossEntropyLoss() lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.2) # Run the training for epoch in range(args.num_epochs): logger.debug('Training Epoch: {}'.format(epoch)) train_one_epoch(model, optimizer, criterion, train_loader, device, args.log_interval) lr_scheduler.step() accuracy = test_model(model, val_loader, device) logger.debug('Validation Accuracy: {}'.format(accuracy)) logger.debug('Finished Training') save_model(model, args.model_dir)if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( '--train-dataset', type=str, required=True, help='path to deeplake training dataset', ) parser.add_argument( '--val-dataset', type=str, required=True, help='path to deeplake validation dataset', ) parser.add_argument( '--batch-size', type=int, default=64, help='input batch size for training (default: 64)', ) parser.add_argument( '--num-workers', type=int, default=8, help='number of workers for the dataloaders (default: 8)', ) parser.add_argument( '--shuffle', type=bool, default=True, help='shuffling for the training dataloader (default: True)', ) parser.add_argument( '--num-epochs', type=int, default=10, help='number of epochs to train (default: 10)', ) parser.add_argument( '--lr', type=float, default=0.001, help='learning rate (default: 0.001)' ) parser.add_argument( '--log-interval', type=int, default=10, metavar='N', help='how many batches to wait before logging training status (default: 10)', ) parser.add_argument( '--token', type=str, default=None, help='token for accessing the Deep Lake dataset (default: None)' ) parser.add_argument( '--creds', type=dict, default=None, help='creds dictionary for accessing the Deep Lake dataset (default: None)' ) parser.add_argument( '--model_dir', type=str, default=os.environ['SM_MODEL_DIR']) train(parser.parse_args())Congrats! You're now able to train models using AWS SageMaker Jobs while streaming Deep Lake Datasets! 🎉


Running the Sagemaker Job

We run the SageMaker job using the docker container below that can be found among these deep learning containers provided by AWS. The SageMaker job is initiated using the script below. By also running this script in a SageMaker notebook, the permissions and role access are automatically taken care of within the AWS environment.import sagemakersess = sagemaker.Session()role = sagemaker.get_execution_role()The training script (entry_point) and the directory (source_dir) containing the training script and requirements.txt file is passed to the Estimator. The argparse parameters for the training script are passed via the hyperparameters dictinary in the Estimator. Note that we also pass the Deep Lake paths to the training and validation datasets via this input. estimator = sagemaker.estimator.Estimator( source_dir = './train_code', # Directory of the training script entry_point = 'train_cars.py', # File for the training script image_uri = image_name, role = role, instance_count = 1, instance_type = instance_type, output_path = output_path, sagemaker_session = sess, max_run = 2*60*60, hyperparameters = {'train-dataset': 'hub://activeloop/stanford-cars-train', 'val-dataset': 'hub://activeloop/stanford-cars-test', 'batch-size': 64, 'num-epochs': 40, })The training job is triggered using the command below. Typically, the .fit() function accepts as inputs the S3 bucket containing the training data, which is then downloaded onto the local storage of the SageMaker job. Since we've passed the Deep Lake dataset paths via the hyperparameters, and since Deep Lake does not require data to be downloaded prior to training, we skip these inputs. estimator.fit()SageMaker offers a variety of method for advanced data logging. In this example, we can monitor the training performance in real-time in the training notebook where the jobs are triggered, or in the CloudWatch logs for each job. We observe that the validation accuracy after 40 epochs is 75%.


Training Script

The contents of the train_code folder, as well as the train_cars.py file, are shown below.  The training script follow the same workflow as other PyTorch training workflows using Deep Lake.  As mentioned above, the inputs to the argparse function are those from the hyperparameters inputs in the estimator.  import deeplakeimport argparseimport loggingimport osimport sysimport time import torchimport torch.nn as nnimport torch.optim as optimimport torch.utils.dataimport torch.utils.data.distributedfrom torchvision import transforms, modelslogger = logging.getLogger(__name__)logger.setLevel(logging.DEBUG)logger.addHandler(logging.StreamHandler(sys.stdout))#----------- Define transformations and their parameters -----------#WIDTH = 320HEIGHT = 320tform_train = transforms.Compose([# transforms.ToPILImage(), # Not needed because decode_method is set to PIL in the dataloader transforms.RandomResizedCrop((WIDTH, HEIGHT), scale=(0.75, 1.0), ratio=(0.75, 1.25)), transforms.RandomRotation(25), transforms.ColorJitter(brightness=(0.8,1.2), contrast=(0.8,1.2), saturation=(0.8,1.2), hue=(-0.1,0.1)), transforms.ToTensor(), transforms.Lambda(lambda x: x.repeat(int(3 / x.shape[0]), 1, 1)), # Adjust tensor if the image is grayscale transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),]) tform_val = transforms.Compose([# transforms.ToPILImage(), # Not needed because decode_method is set to PIL in the dataloader transforms.Resize((WIDTH, HEIGHT)), transforms.ToTensor(), transforms.Lambda(lambda x: x.repeat(int(3 / x.shape[0]), 1, 1)), # Adjust tensor if the image is grayscale transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),]) #----------- Define helper functions -----------## Helper function for loading the modeldef get_model_classification(num_classes): # Load a pre-trained classification model model = models.resnet34(pretrained=True) # Adjust the fully connected layer based on the number of classes in the dataset model.fc = torch.nn.Linear(model.fc.in_features, num_classes) return model# Helper function for training for 1 epochdef train_one_epoch(model, optimizer, criterion, data_loader, device, log_interval): # Set the model to train mode model.train() # Zero the performance stats for each epoch running_loss = 0.0 start_time = time.time() total = 0 correct = 0 for i, data in enumerate(data_loader): # Parse the inputs inputs = data['images'] labels = data['car_models'][:, 0] # Get rid of the extra axis inputs = inputs.to(device) labels = labels.to(device) # Zero the parameter gradients optimizer.zero_grad() # forward + backward + optimize outputs = model(inputs.float()) loss = criterion(outputs, labels.long()) loss.backward() optimizer.step() # Update the accuracy for the epoch _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() accuracy = 100 * correct / total # Print statistics running_loss += loss.item() batch_time = time.time() if i % log_interval == 0: # print every 100 mini-batches speed_cumulative = (i+1)/(batch_time-start_time) logger.debug('[%5d] running loss: %.3f, epoch accuracy: %.3f, cumulative speed: %.2f ' % (i, running_loss, accuracy, speed_cumulative)) running_loss = 0.0 # Helper function for testing the model def test_model(model, data_loader, device): # Set the model to eval mode model.eval() total = 0 correct = 0 with torch.no_grad(): for i, data in enumerate(data_loader): # Parse the inputs inputs = data['images'] labels = data['car_models'][:, 0] inputs = inputs.to(device) labels = labels.to(device) outputs = model(inputs.float()) _, predicted = torch.max(outputs.data, 1) total += labels.size(0) correct += (predicted == labels).sum().item() accuracy = 100 * correct / total return accuracy # Helper function for saving the model def save_model(model, model_dir): logger.info('Saving the model') path = os.path.join(model_dir, 'model.pth') torch.save(model.state_dict(), path) def train(args): device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') # Load dataset and create dataloaders


This tutorial is also available as a Colab Notebook

The primary objective for Deep Lake is to enable users to manage their data more easily so they can train better ML models. This tutorial shows you how to train an object detection and instance segmentation model while streaming data from a Deep Lake dataset stored in the cloud.Since these models are often complex, this tutorial will focus on data-preprocessing for connecting the data to the model. The user should take additional steps to scale up the code for logging, collecting additional metrics, model testing, and running on GPUs.This tutorial is inspired by this PyTorch tutorial on training object detection and segmentation models.


Data Preprocessing

The first step is to select a dataset for training.  This tutorial uses the COCO dataset that has already been converted into Deep Lake format.  It is a multi-modal image dataset that contains bounding boxes, segmentation masks, keypoints, and other data.import deeplakeimport numpy as npimport mathimport sysimport timeimport torchvisionimport albumentations as Afrom albumentations.pytorch import ToTensorV2import torchfrom torchvision.models.detection.faster_rcnn import FastRCNNPredictorfrom torchvision.models.detection.mask_rcnn import MaskRCNNPredictorimport torchvision.models.detection.mask_rcnn# Connect to the training datasetds_train = deeplake.load('hub://activeloop/coco-train')Note that the dataset can be visualized at the link printed by the deeplake.load command above.We extract the number of classes for use later:num_classes = len(ds_train.categories.info.class_names)For complex dataset like this one, it's critical to carefully define the pre-processing function that returns the torch tensors that are use for training.  Here we use an Albumentations augmentation pipeline combined with additional pre-processing steps that are necessary for this particular model.Note: This tutorial assumes that the number of masks and bounding boxes for each image is equal# Augmentation pipeline using Albumentationstform_train = A.Compose([ A.RandomSizedBBoxSafeCrop(width=128, height=128, erosion_rate = 0.2), A.HorizontalFlip(p=0.5), A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), ToTensorV2(), # transpose_mask = True], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['class_labels', 'bbox_ids'], min_area=25, min_visibility=0.6)) # 'label_fields' and 'box_ids' are all the fields that will be cut when a bounding box is cut.# Transformation function for pre-processing the Deep Lake sample before sending it to the modeldef transform(sample_in): # Convert boxes to Pascal VOC format boxes = coco_2_pascal(sample_in['boxes']) # Convert any grayscale images to RGB images = sample_in['images'] if images.shape[2] == 1: images = np.repeat(images, int(3/images.shape[2]), axis = 2) # Pass all data to the Albumentations transformation # Mask must be converted to a list masks = sample_in['masks'] mask_shape = masks.shape # This if-else statement was not necessary in Albumentations <1.3.x, because the empty mask scenario was handled gracefully inside of Albumentations.  In Albumebtations >1.3.x, empty list of masks fails if mask_shape[2]>0: transformed = tform_train(image = images, masks = [masks[:,:,i].astype(np.uint8) for i in range(mask_shape[2])], bboxes = boxes, bbox_ids = np.arange(boxes.shape[0]), class_labels = sample_in['categories'], ) else: transformed = tform_train(image = images, bboxes = boxes, bbox_ids = np.arange(boxes.shape[0]), class_labels = sample_in['categories'], )  # Convert boxes and labels from lists to torch tensors, because Albumentations does not do that automatically


Data Preprocessing

# Be very careful with rounding and casting to integers, becuase that can create bounding boxes with invalid dimensions labels_torch = torch.tensor(transformed['class_labels'], dtype = torch.int64) boxes_torch = torch.zeros((len(transformed['bboxes']), 4), dtype = torch.int64) for b, box in enumerate(transformed['bboxes']): boxes_torch[b,:] = torch.tensor(np.round(box))  # Filter out the masks that were dropped by filtering of bounding box area and visibility masks_torch = torch.zeros((len(transformed['bbox_ids']), transformed['image'].shape[1], transformed['image'].shape[2]), dtype = torch.int64) if len(transformed['bbox_ids'])>0: masks_torch = torch.tensor(np.stack([transformed['masks'][i] for i in transformed['bbox_ids']], axis = 0), dtype = torch.uint8)  # Put annotations in a separate object target = {'masks': masks_torch, 'labels': labels_torch, 'boxes': boxes_torch} return transformed['image'], target# Conversion script for bounding boxes from coco to Pascal VOC formatdef coco_2_pascal(boxes): # Convert bounding boxes to Pascal VOC format and clip bounding boxes to make sure they have non-negative width and height return np.stack((boxes[:,0], boxes[:,1], boxes[:,0]+np.clip(boxes[:,2], 1, None), boxes[:,1]+np.clip(boxes[:,3], 1, None)), axis = 1)def collate_fn(batch): return tuple(zip(*batch))You can now create a PyTorch dataloader that connects the Deep Lake dataset to the PyTorch model using the provided method ds.pytorch() .  This method automatically applies the transformation function and takes care of random shuffling (if desired).  The num_workers parameter can be used to parallelize data preprocessing, which is critical for ensuring that preprocessing does not bottleneck the overall training workflow.Since the dataset contains many tensors that are not used for training, a list of tensors for loading is specified in order to avoid streaming of unused data.batch_size = 8train_loader = ds_train.pytorch(num_workers = 2, shuffle = False, tensors = ['images', 'masks', 'categories', 'boxes'], # Specify the tensors that are needed, so we don't load unused data transform = transform, batch_size = batch_size, collate_fn = collate_fn)


Model Definition

This tutorial uses a pre-trained torchvision neural network from the torchvision.models module.Training is performed on a GPU if possible. Otherwise, it's on a CPU.device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')print(device)# Helper function for loading the modeldef get_model_instance_segmentation(num_classes): # Load an instance segmentation model pre-trained on COCO model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True) # Get number of input features for the classifier in_features = model.roi_heads.box_predictor.cls_score.in_features # replace the pre-trained head with a new one model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes) # Get the number of input features for the mask classifier in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels hidden_layer = 256 # Replace the mask predictor with a new one model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes) return modelLet's initialize the model and optimizer.model = get_model_instance_segmentation(num_classes)model.to(device)# Specity the optimizerparams = [p for p in model.parameters() if p.requires_grad]optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)


Training the Model

Helper functions for training and testing the model are defined. Note that the output from Deep Lake's PyTorch dataloader is fed into the model just like data from ordinary PyTorch dataloaders.# Helper function for training for 1 epochdef train_one_epoch(model, optimizer, data_loader, device): model.train() start_time = time.time() for i, data in enumerate(data_loader): images = list(image.to(device) for image in data[0]) targets = [{k: v.to(device) for k, v in t.items()} for t in data[1]] loss_dict = model(images, targets) losses = sum(loss for loss in loss_dict.values()) loss_value = losses.item() # Print performance statistics batch_time = time.time() speed = (i+1)/(batch_time-start_time) print('[%5d] loss: %.3f, speed: %.2f' % (i, loss_value, speed)) if not math.isfinite(loss_value): print(f'Loss is {loss_value}, stopping training') print(loss_dict) break optimizer.zero_grad() losses.backward() optimizer.step()The model and data are ready for training 🚀!# Train the model for 1 epochnum_epochs = 1for epoch in range(num_epochs): # loop over the dataset multiple times print('------------------ Training Epoch {} ------------------'.format(epoch+1)) train_one_epoch(model, optimizer, train_loader, device) # --- Insert Testing Code Here --- print('Finished Training')Congrats! You successfully trained an object detection and instance segmentation model while streaming data directly from the cloud! 🎉


After creating a Deep Lake dataset, you may need to edit it by adding, deleting, and modifying the data. In this tutorial, we show best practices for updating datasets.


Create a Representative Deep Lake Dataset

First, let's download and unzip representative source data and create a Deep Lake dataset for this tutorial:This dataset includes segmentation and object detection of vehicle damage, but for this tutorial, we will only upload the images and labels (damage location)import deeplakeimport pandas as pdimport osfrom PIL import Imageimages_directory = '/damaged_cars_tutorial' # Path to the COCO images directoryannotation_file = '/damaged_cars_tutorial/COCO_mul_val_annos.json' # Path to the COCO annotations filedeeplake_path = '/damaged_cars_dataset' # Path to the Deep Lake datasetds = deeplake.ingest_coco(images_directory, annotation_file, deeplake_path, key_to_tensor_mapping={'category_id': 'labels'}, # Rename category_id to labels ignore_keys=['area', 'image_id', 'id', 'segmentation', 'image_id', 'bbox', 'iscrowd']) shows the dataset has two tensors with 11 samples: tensor htype shape dtype compression ------- ------- ------- ------- ------- images image (11, 1024, 1024, 3) uint8 jpeg labels class_label (11, 2:7) uint32 None We can explore the damage in the first sample using ds.labels[0].data(), which prints:{'value': array([0, 1, 2], dtype=uint32), 'text': ['rear_bumper', 'door', 'headlamp']}


Add Data to a New Tensor

Suppose you have another data source with supplemental data about the color of the vehicles. Let's create a Pandas DataFrame with this data.color_data = {'filename': ['1.jpg', '9.jpg', '62.jpg', '24.jpg'], 'color': ['gray', 'blue', 'green', 'gray']} df_color = pd.DataFrame(color_data)There are two approaches for adding this new data to the Deep Lake dataset:


Iterate through the supplemental data and add data at the corresponding Deep Lake index

This approach is recommended when the data updates are sparseFirst, let's create a color2 tensor, and the load all the existing Deep Lake filenames into memory. We then iterate through the supplemental data and find the corresponding Deep Lake index to insert the color information.with ds: ds.create_tensor('color2', htype = 'class_label') filenames = [os.path.basename(sample_info['filename']) for sample_info in ds.images.sample_info] for fn in df_color['filename'].values: index = filenames.index(fn) ds.color2[index] = df_color[df_color['filename'] == fn]['color'].values[0]Now we see that ds.summary() shows 4 tensors, each with 11 samples (though the color and color2 tensors have several empty samples): tensor htype shape dtype compression ------- ------- ------- ------- ------- images image (10, 1024, 1024, 3) uint8 jpeg labels class_label (10, 2:7) uint32 None color class_label (10, 0:1) uint32 None color2 class_label (10, 0:1) uint32 None


1. Iterate through the Deep Lake samples and append data

This approach is recommended when most Deep Lake samples are being updated using the supplemental data (dense update).First, we create a color tensor and iterate through the samples. For each sample, we lookup the color from the df_color DataFrame and append it to the color tensor. If no color exists for a filename, it is appended as None. We use the filename as the key to perform the lookup, which is available in ds.images[index].sample_info dictionary.with ds: ds.create_tensor('color', htype = 'class_label') # After creating an empty tensor, the length of the dataset is 0 # Therefore, we iterate over ds.max_view, which is the padded version of the dataset for i, sample in enumerate(ds.max_view): filename = os.path.basename(sample.images.sample_info['filename']) color = df_color[df_color['filename'] == filename]['color'].values ds.color.append(None if len(color)==0 else color)Learn more about dataset lengths and padding here.Now we see that ds.summary() shows 3 tensors, each with 11 samples (though the color tensor has several empty samples): tensor htype shape dtype compression ------- ------- ------- ------- ------- images image (11, 1024, 1024, 3) uint8 jpeg labels class_label (11, 2:7) uint32 None color class_label (11, 0:1) uint32 None


Delete Samples

Rows from a dataset can be deleted using ds.pop(). To delete the row at index 8 we run:ds.pop(8)Now we see that ds.summary() shows 10 rows in the dataset (instead of 11): tensor htype shape dtype compression ------- ------- ------- ------- ------- images image (10, 1024, 1024, 3) uint8 jpeg labels class_label (10, 2:7) uint32 None color class_label (10, 0:1) uint32 None color2 class_label (10, 0:1) uint32 None To replace data with empty data without deleting a row, you can run: ds.color[index] = NoneCongrats! You just learned how to make a variety of updates to Deep Lake datasets! 🎉


Update Existing Rows without TQL

Originally, we did not specify a color for image 3.jpg. Let's find the index for this image, look at it, and add the color manually. We've already loaded the Deep Lake dataset's filenames into memory above, so we can find the index using:index = filenames.index('3.jpg')Let's visualize the image using PIL. We could also visualize it using ds.visualize() (must pip install 'deeplake[visualizer]') or using the Deep Lake App.Image.fromarray(ds.images[index].numpy())Since the image is white, let's update the color using:ds.color[index] = 'white'


Video datasets are becoming increasingly common in Computer Vision applications. This tutorial demonstrates how to convert a simple video classification dataset into Deep Lake format. Uploading videos in Deep Lake is nearly identical as uploading images, aside from minor differences in sample compression that are described below.When using Deep Lake with videos, make sure to install it using one of the following options:


Create the Deep Lake Dataset

The first step is to download the small dataset below called running walking.animals object detection datasetThe dataset has the following folder structure:data_dir|_running |_video_1.mp4 |_video_2.mp4|_walking |_video_3.mp4 |_video_4.mp4Now that you have the data, let's create a Deep Lake Dataset in the ./running_walking_deeplake folder by running:import deeplakefrom PIL import Image, ImageDrawimport numpy as npimport osds = deeplake.empty('./running_walking_deeplake') # Create the dataset locallyNext, let's inspect the folder structure for the source dataset ./running_walking to find the class names and the files that need to be uploaded to the Deep Lake dataset.# Find the class_names and list of files that need to be uploadeddataset_folder = './running_walking'class_names = os.listdir(dataset_folder)fn_vids = []for dirpath, dirnames, filenames in os.walk(dataset_folder): for filename in filenames: fn_vids.append(os.path.join(dirpath, filename))Finally, let's create the tensors and iterate through all the images in the dataset in order to upload the data in Deep Lake.They key difference between video and image htypes is that Deep Lake does not explicitly perform compression for videos. The sample_compression input in the create_tensor function is used to verify that the compression of the input video file to deeplake.read()matches the sample_compression parameter. If there is a match, the video is uploaded in compressed format. Otherwise, an error is thrown. Images have a slightly different behavior, because the input image files are stored and re-compressed (if necessary) to the sample_compression format.with ds: ds.create_tensor('videos', htype='video', sample_compression = 'mp4') ds.create_tensor('labels', htype='class_label', class_names = class_names) for fn_vid in fn_vids: label_text = os.path.basename(os.path.dirname(fn_vid)) label_num = class_names.index(label_text) # Append data to tensors ds.videos.append(deeplake.read(fn_vid)) ds.labels.append(np.uint32(label_num))In order for Activeloop Platform to correctly visualize the labels, class_names must be a list of strings, where the numerical labels correspond to the index of the label in the list.


Inspect the Deep Lake Dataset

Let's check out the first frame in the second sample from this dataset. video_ind = 1frame_ind = 0# Individual frames are loaded lazilyimg = Image.fromarray(ds.videos[ind][frame_ind].numpy())# Load the numberic label and read the class name from ds.labels.info.class_namesds.labels.info.class_names[ds.labels[ind].numpy()[frame_ind]]imgYou've successfully created a video dataset in Activeloop Deep Lake.Congrats! You just created a video classification dataset! 🎉


Datasets often have multiple labels such as classifications, bounding boxes, segmentations, and others. In order to create an intuitive layout of tensors, it's advisable to create a dataset hierarchy that captures the relationship between the different label types. This can be done with Deep Lake tensor groups.This example show to to use groups to create a dataset containing image classifications of 'indoor' and 'outdoor', as well as bounding boxes of objects such as 'dog' and 'cat'.


Create the Deep Lake Dataset

The first step is to download the small dataset below called animals complex.animals complex datasetThe images and their classes are stored in a classification folder where the subfolders correspond to the class names.  Bounding boxes for object detection are stored in a separate boxes subfolder, which also contains a list of class names for object detection in the file box_names.txt.  In YOLO format, images and annotations are typically matched using a common filename such as image -> filename.jpeg and annotation -> filename.txt .  The data structure for the dataset is shown below:data_dir|_classification |_indoor |_image1.png |_image2.png |_outdoor |_image3.png |_image4.png|_boxes |_image1.txt |_image3.txt |_image3.txt |_image4.txt |_classes.txtNow that you have the data, let's create a Deep Lake Dataset in the ./animals_complex_deeplake folder by running: import deeplakefrom PIL import Image, ImageDrawimport numpy as npimport osds = deeplake.empty('./animals_complex_deeplake') # Create the dataset locallyNext, let's specify the folder paths containing the classification and object detection data.  It's also helpful to create a list of all of the image files and class names for classification and object detection tasks.classification_folder = './animals_complex/classification'boxes_folder = './animals_complex/boxes'# List of all class names for classificationclass_names = os.listdir(classification_folder)fn_imgs = []for dirpath, dirnames, filenames in os.walk(classification_folder): for filename in filenames: fn_imgs.append(os.path.join(dirpath, filename))# List of all class names for object detection with open(os.path.join(boxes_folder, 'classes.txt'), 'r') as f: class_names_boxes = f.read().splitlines()Since annotations in YOLO are typically stored in text files, it's useful to write a helper function that parses the annotation file and returns numpy arrays with the bounding box coordinates and bounding box classes.def read_yolo_boxes(fn:str): ''' Function reads a label.txt YOLO file and returns a numpy array of yolo_boxes for the box geometry and yolo_labels for the corresponding box labels.  ''' box_f = open(fn) lines = box_f.read() box_f.close() # Split each box into a separate lines lines_split = lines.splitlines() yolo_boxes = np.zeros((len(lines_split),4)) yolo_labels = np.zeros(len(lines_split)) # Go through each line and parse data for l, line in enumerate(lines_split): line_split = line.split() yolo_boxes[l,:]=np.array((float(line_split[1]), float(line_split[2]), float(line_split[3]), float(line_split[4]))) yolo_labels[l]=int(line_split[0]) return yolo_boxes, yolo_labelsNext, let's create the groups and tensors for this data.  In order to separate the two annotations, a boxes group is created to wrap around the label and bbox tensors which contains the coordinates and labels for the bounding boxes.with ds: # Image ds.create_tensor('images', htype='image', sample_compression='jpeg') # Classification ds.create_tensor('labels', htype='class_label', class_names = class_names) # Object Detection ds.create_group('boxes') ds.boxes.create_tensor('bbox', htype='bbox') ds.boxes.create_tensor('label', htype='class_label', class_names = class_names_boxes) # An alternate approach is to use '/' notation, which automatically creates the boxes group # ds.create_tensor('boxes/bbox', ...) # ds.create_tensor('boxes/label', ...) # Define the format of the bounding boxes ds.boxes.bbox.info.update(coords = {'type': 'fractional', 'mode': 'LTWH'})In order for Activeloop Platform to correctly visualize the labels, class_names must be a list of strings, where the numerical labels correspond to the index of the label in the list.Finally, let's iterate through all the images in the dataset in order to upload the data in Deep Lake.  The first axis of the boxes.bbox sample array corresponds to the first-and-only axis of the boxes.label sample array (i.e


Create the Deep Lake Dataset

if there are 3 boxes in an image, the labels array is 3x1 and the boxes array is 3x4).with ds: #Iterate through the images for fn_img in fn_imgs: img_name = os.path.splitext(os.path.basename(fn_img))[0] fn_box = img_name+'.txt' # Get the class number for the classification label_text = os.path.basename(os.path.dirname(fn_img)) label_num = class_names.index(label_text) # Get the arrays for the bounding boxes and their classes yolo_boxes, yolo_labels = read_yolo_boxes(os.path.join(boxes_folder,fn_box)) # Append data to tensors ds.append({'images': deeplake.read(os.path.join(fn_img)), 'labels': np.uint32(label_num), 'boxes/label': yolo_labels.astype(np.uint32), 'boxes/bbox': yolo_boxes.astype(np.float32) })


Inspect the Deep Lake Dataset

Let's check out the second sample from this dataset and visualize the labels.# Draw bounding boxes and the classfication label for the second imageind = 1img = Image.fromarray(ds.images[ind].numpy())draw = ImageDraw.Draw(img)(w,h) = img.sizeboxes = ds.boxes.bbox[ind].numpy()for b in range(boxes.shape[0]): (xc,yc) = (int(boxes[b][0]*w), int(boxes[b][1]*h)) (x1,y1) = (int(xc-boxes[b][2]*w/2), int(yc-boxes[b][3]*h/2)) (x2,y2) = (int(xc+boxes[b][2]*w/2), int(yc+boxes[b][3]*h/2)) draw.rectangle([x1,y1,x2,y2], width=2) draw.text((x1,y1), ds.boxes.label.info.class_names[ds.boxes.label[ind].numpy()[b]]) draw.text((0,0), ds.labels.info.class_names[ds.labels[ind].numpy()[0]])# Display the image and its bounding boxesimgCongrats! You just created a dataset with multiple types of annotations! 🎉


Object detection using bounding boxes is one of the most common annotation types for Computer Vision datasets. This tutorial demonstrates how to convert an object detection dataset from YOLO format to Deep Lake, and a similar method can be used to convert object detection datasets from other formats such as COCO and PASCAL VOC.


Create the Deep Lake Dataset

The first step is to download the small dataset below called animals object detection.animals object detection datasetThe dataset has the following folder structure:data_dir|_images |_image_1.jpg |_image_2.jpg |_image_3.jpg |_image_4.jpg|_boxes |_image_1.txt |_image_2.txt |_image_3.txt |_image_4.txt |_classes.txtNow that you have the data, let's create a Deep Lake Dataset in the ./animals_od_deeplakefolder by running:import deeplakefrom PIL import Image, ImageDrawimport numpy as npimport osds = deeplake.empty('./animals_od_deeplake') # Create the dataset locallyNext, let's specify the folder paths containing the images and annotations in the dataset. In YOLO format, images and annotations are typically matched using a common filename such as image -> filename.jpeg and annotation -> filename.txt . It's also helpful to create a list of all of the image files and the class names contained in the dataset.img_folder = './animals_od/images'lbl_folder = './animals_od/boxes'# List of all imagesfn_imgs = os.listdir(img_folder)# List of all class nameswith open(os.path.join(lbl_folder, 'classes.txt'), 'r') as f: class_names = f.read().splitlines()Since annotations in YOLO are typically stored in text files, it's useful to write a helper function that parses the annotation file and returns numpy arrays with the bounding box coordinates and bounding box classes.def read_yolo_boxes(fn:str): ''' Function reads a label.txt YOLO file and returns a numpy array of yolo_boxes for the box geometry and yolo_labels for the corresponding box labels. ''' box_f = open(fn) lines = box_f.read() box_f.close() # Split each box into a separate lines lines_split = lines.splitlines() yolo_boxes = np.zeros((len(lines_split),4)) yolo_labels = np.zeros(len(lines_split)) # Go through each line and parse data for l, line in enumerate(lines_split): line_split = line.split() yolo_boxes[l,:]=np.array((float(line_split[1]), float(line_split[2]), float(line_split[3]), float(line_split[4]))) yolo_labels[l]=int(line_split[0]) return yolo_boxes, yolo_labelsFinally, let's create the tensors and iterate through all the images in the dataset in order to upload the data in Deep Lake. Boxes and their labels will be stored in separate tensors, and for a given sample, the first axis of the boxes array corresponds to the first-and-only axis of the labels array (i.e. if there are 3 boxes in an image, the labels array is 3x1 and the boxes array is 3x4).with ds: ds.create_tensor('images', htype='image', sample_compression = 'jpeg') ds.create_tensor('labels', htype='class_label', class_names = class_names) ds.create_tensor('boxes', htype='bbox') # Define the format of the bounding boxes ds.boxes.info.update(coords = {'type': 'fractional', 'mode': 'LTWH'}) for fn_img in fn_imgs: img_name = os.path.splitext(fn_img)[0] fn_box = img_name+'.txt' # Get the arrays for the bounding boxes and their classes yolo_boxes, yolo_labels = read_yolo_boxes(os.path.join(lbl_folder,fn_box)) # Append data to tensors ds.append({'images': deeplake.read(os.path.join(img_folder, fn_img)), 'labels': yolo_labels.astype(np.uint32), 'boxes': yolo_boxes.astype(np.float32) })In order for Activeloop Platform to correctly visualize the labels, class_names must be a list of strings, where the numerical labels correspond to the index of the label in the list.


Inspect the Deep Lake Dataset

Let's check out the third sample from this dataset, which contains two bounding boxes.# Draw bounding boxes for the fourth imageind = 3img = Image.fromarray(ds.images[ind ].numpy())draw = ImageDraw.Draw(img)(w,h) = img.sizeboxes = ds.boxes[ind].numpy()for b in range(boxes.shape[0]): (xc,yc) = (int(boxes[b][0]*w), int(boxes[b][1]*h)) (x1,y1) = (int(xc-boxes[b][2]*w/2), int(yc-boxes[b][3]*h/2)) (x2,y2) = (int(xc+boxes[b][2]*w/2), int(yc+boxes[b][3]*h/2)) draw.rectangle([x1,y1,x2,y2], width=2) draw.text((x1,y1), ds.labels.info.class_names[ds.labels[ind].numpy()[b]])# Display the image and its bounding boxesimgCongrats! You just created a beautiful object detection dataset! 🎉Note: For optimal object detection model performance, it is often important for datasets to contain images with no annotations (See the 4th sample in the dataset above). Empty samples can be appended using:or by specifying an empty array whose len(shape) is equal to that of the other samples in the tensor: #len(sample.shape) == 2


Deep Lake is intuitive format for storing large time-series datasets and it offers compression for reducing storage costs. This tutorial demonstrates how to convert a time-series data to Deep Lake format and load the data for plotting.


Create the Deep Lake Dataset

The first step is to download the small dataset below called sensor data.This is a subset of a dataset available on kaggle, and it contains the iPhone x,y,z acceleration for 24 users (subjects) under conditions of walking and jogging. The dataset has the folder structure below. subjects_info.csv contains metadata such as height, weight, etc. for each subject, and the sub_n.csv files contains the time-series acceleration data for the nth subject.data_dir|_subjects_into.csv|_motion_data |_walk |_sub_1.csv |_sub_2.csv ... ... |_jog |_sub_1.csv |_sub_2.csv ... ...Now that you have the data, let's create a Deep Lake Dataset in the ./sensor_data_deeplake folder by running: import deeplakeimport pandas as pdimport osfrom tqdm import tqdmimport numpy as npimport matplotlib.pyplot as pltds = deeplake.empty('./sensor_data_deeplake') # Create the dataset locallyNext, let's specify the folder path containing the existing dataset, load the subjects metadata to a Pandas DataFrame, and create a list of all of the time-series files that should be converted to Deep Lake format.dataset_path= './sensor_data'subjects_info = pd.read_csv(os.path.join(dataset_path, 'subjects_info.csv'))fns_series = []for dirpath, dirnames, filenames in os.walk(os.path.join(dataset_path, 'motion_data')): for filename in filenames: fns_series .append(os.path.join(dirpath, filename))Next, let's create the tensors and add relevant metadata, such as the dataset source, the tensor units, and other information. We leverage groups to separate out the primary acceleration data from other user data such as the weight and height of the subjects. with ds: #Update dataset metadata ds.info.update(source = 'https://www.kaggle.com/malekzadeh/motionsense-dataset', notes = 'This is a small subset of the data in the source link') #Create tensors. Setting chunk_compression is optional and it defaults to None ds.create_tensor('acceleration_x', chunk_compression = 'lz4') ds.create_tensor('acceleration_y', chunk_compression = 'lz4') # Save the sampling rate as tensor metadata. Alternatively, # you could also create a 'time' tensor. ds.acceleration_x.info.update(sampling_rate_s = 0.1) ds.acceleration_y.info.update(sampling_rate_s = 0.1) # Encode activity as text ds.create_tensor('activity', htype = 'text') # Encode 'activity' as numeric labels and convert to text via class_names # ds.create_tensor('activity', htype = 'class_label', class_names = ['xyz']) ds.create_group('subjects_info') ds.subjects_info.create_tensor('age') ds.subjects_info.create_tensor('weight') ds.subjects_info.create_tensor('height') # Save the units of weight as tensor metadata ds.subjects_info.weight.info.update(units = 'kg') ds.subjects_info.height.info.update(units = 'cm')Finally, let's iterate through all the time-series data and upload it to the Deep Lake dataset. with ds: # Iterate through the time series and append data for fn in tqdm(fns_series): # Read the data in the time series df_data = pd.read_csv(fn) # Parse the 'activity' from the file name activity = os.path.basename(os.path.dirname(fn)) # Parse the subject code from the filename and pull the subject info from 'subjects_info' subject_code = int(os.path.splitext(os.path.basename(fn))[0].split('_')[1]) subject_info = subjects_info[subjects_info['code']==subject_code] # Append data to tensors ds.activity.append(activity) ds.subjects_info.age.append(subject_info['age'].values) ds.subjects_info.weight.append(subject_info['weight'].values) ds.subjects_info.height.append(subject_info['height'].values) ds.acceleration_x.append(df_data['userAcceleration.x'].values) ds.acceleration_y.append(df_data['userAcceleration.y'].values)


Inspect the Deep Lake Dataset

Let's check out the first sample from this dataset and plot the acceleration time-series.It is noteworthy that the Deep Lake dataset takes 36% less memory than the original dataset due to lz4 chunk compression for the acceleration tensors.s_ind = 0 # Plot the first time seriest_ind = 100 # Plot the first 100 indices in the time series#Plot the x accelerationx_data = ds.acceleration_x[s_ind].numpy()[:t_ind]sampling_rate_x = ds.acceleration_x.info.sampling_rate_splt.plot(np.arange(0, x_data.size)*sampling_rate_x, x_data, label='acceleration_x')#Plot the y accelerationy_data = ds.acceleration_y[s_ind].numpy()[:t_ind]sampling_rate_y = ds.acceleration_y.info.sampling_rate_splt.plot(np.arange(0, y_data.size)*sampling_rate_y, y_data, label='acceleration_y')plt.legend()plt.xlabel('time [s]', fontweight = 'bold')plt.ylabel('acceleration [g]', fontweight = 'bold')plt.title('Weight: {} {}, Height: {} {}'.format(ds.subjects_info.weight[s_ind].numpy()[0], ds.subjects_info.weight.info.units, ds.subjects_info.height[s_ind].numpy()[0], ds.subjects_info.height.info.units), fontweight = 'bold')plt.xlim([0, 10])plt.grid()plt.gcf().set_size_inches(8, 5)plt.show()Congrats! You just converted a time-series dataset to Deep Lake format! 🎉


Deep learning with computer vision is increasingly moving in a direction of temporal data, where video frames and their labels are stored as sequences, rather than independent images. Models trained on this data directly account for the temporal information content, rather than making predictions frame-by-frame and then fusing them with non-deep-learning techniques.


Create the Deep Lake Dataset

The first step is to download the dataset Multiple Object Tracking Benchmark. Additional information about this data and its format is in this GitHub Repo.The dataset has the following folder structure:data_dir|_train |_MOT16_N (Folder with sequence N) |_det |_gt (Folder with ground truth annotations) |_img1 (Folder with images the sequence) |_00000n.jpg (image of n-th frame in sequence) |_MOT16_M ....|_test (same structure as _train)The annotations in gt.txt have the format below, and the last 4 items (conf->z) are not used in the Deep Lake dataset:frame, id, bb_left, bb_top, bb_width, bb_height, conf, x, y, zNow we're ready to create a Deep Lake Dataset in the ./mot_2016_train folder by running:import deeplakeimport osimport pandas as pdimport numpy as npfrom PIL import Image, ImageDrawds = deeplake.empty('./mot_2015_train') # Create the dataset locallyNext, let's write code to inspect the folder structure for the downloaded dataset and create a list of folders containing the sequences:dataset_folder = '/MOT16/train'sequences = [ item for item in os.listdir(dataset_folder) if os.path.isdir(os.path.join(dataset_folder, item)) ]Finally, let's create the tensors by using the sequence[...] htype, iterate through each sequence, and iterate through each frame within the sequence, one-by-one. Data is appended to sequence[...] htypes using lists. The list contains the whole sample, and the individual elements of the list are the individual data points, such as the image frame, the bounding boxes in a particular frame, etc. See end of code block below.with ds: # Define tensors ds.create_tensor('frames', htype = 'sequence[image]', sample_compression = 'jpg') ds.create_tensor('boxes', htype = 'sequence[bbox]') ds.create_tensor('ids', htype = 'sequence[]', dtype = 'uint32') # Ids are not uploaded as htype = 'class_labels' because they don't contain information about the class of an object. ds.boxes.info.update(coords = {'type': 'pixel', 'mode': 'LTWH'}) # Bounding box format is left, top, width, height # Iterate through each sequence for sequence in sequences: # Define root directory for that sequence root_local = os.path.join(dataset_folder,sequence, 'img1') # Get a list of all the image paths img_paths = [os.path.join(root_local, item) for item in sorted(os.listdir(root_local))] # Read the annotations and convert to dataframe with open(os.path.join(dataset_folder,sequence, 'gt', 'gt.txt')) as f: anns = [line.rstrip('\n') for line in f] anns_df = pd.read_csv(os.path.join(dataset_folder, sequence, 'gt', 'gt.txt'), header = None) # Get the frames from the annotations and make sure they're of equal length as the images frames = pd.unique(anns_df[0]) assert len(frames) == len(img_paths) # Iterate through each frame and add data to sequence boxes_seq = [] ids_seq = [] for frame in frames: ann_df = anns_df[anns_df[0] == frame] # Find annotations in the specific frame boxes_seq.append(ann_df.loc[:, [2, 3, 4, 5]].to_numpy().astype('float32')) # Box coordinates are in the 3rd-6th column ids_seq.append(ann_df.loc[:, 1].to_numpy().astype('uint32')) # ids are in the second column # Append the sequences to the deeplake dataset ds.append({ 'frames': [deeplake.read(path) for path in img_paths], 'boxes': boxes_seq, 'ids': ids_seq})This dataset identifies objects by id, where each id represents an instance of an object. However, the id does not identify the class of the object, such person, car, truck, etc. Therefore, the ids were not uploaded as htype = 'class_label'.


Inspect the Deep Lake Dataset

Let's check out the 10th frame in the 6th sequence in this dataset. A complete visualization of this dataset is available in Activeloop Platform.# Draw bounding boxes for the 10th frame in the 6th sequenceseq_ind = 5frame_ind = 9img = Image.fromarray(ds.frames[seq_ind][frame_ind].numpy())draw = ImageDraw.Draw(img)(w,h) = img.sizeboxes = ds.boxes[seq_ind][frame_ind].numpy()for b in range(boxes.shape[0]): (x1,y1) = (int(boxes[b][0]), int(boxes[b][1])) (x2,y2) = (int(boxes[b][0]+boxes[b][2]), int(boxes[b][1]+boxes[b][3])) draw.rectangle([x1,y1,x2,y2], width=2, outline = 'red')# Display the frame and its bounding boxesimgCongrats! You just created a dataset using sequences! 🎉


Azure-Specific Information for Connecting Data to Deep Lake

Provisioning Federated CredentialsEnabling CORS


AWS-Specific Information for Connecting Data to Deep Lake

Provisioning Role-Based AccessEnabling CORS


Setting up Role-Based Access for AWS S3

The most secure method for connecting data from your AWS account to Deep Lake is using Federated Credentials and Role-Based Access, which are set up using the steps below:


Step 1: Create the AWS IAM Policy

1. Login to the AWS account where the IAM Role will be created and where the data is stored.2. Go to the IAM page in the AWS UI, which can be done by searching 'IAM' in the console and locating the IAM page under Services.3. In the left nav, open the Policies under Access management and on Create policy on the right.5. Select the JSON tab instead of Visual editor.6. Replace the code in the editor with the code below. Replace BUCKET_NAME with the bucket names for which you want to grant role-based access:{ 'Version': '2012-10-17', 'Statement': [ { 'Effect': 'Allow', 'Action': [ 's3:ListBucket', 's3:GetBucketLocation', 's3:GetObject', 's3:PutObject', 's3:DeleteObject' ], 'Resource': [ 'arn:aws:s3:::BUCKET_NAME', 'arn:aws:s3:::BUCKET_NAME/*' ] } ]}7. On the bottom right, click Next: Tags (create tags if needed) and Next: Preview, enter the policy name and description, and click Create policy


Step 2: Create the AWS IAM Role

1. On the IAM page, in the left nav, open the Roles under Access management, and click Create role on the right.3. Select Custom trust policy from the list of options.4. Replace the policy definition with the code below and click Next{ 'Version': '2012-10-17', 'Statement': [ { 'Sid': 'AllowAssumeRoleFromActiveloopSaaS', 'Effect': 'Allow', 'Principal': { 'AWS': [ 'arn:aws:iam::574987031486:role/activeloop_backend_dev', 'arn:aws:iam::574987031486:role/activeloop_backend', 'arn:aws:iam::597713067985:role/activeloop_backend' ] }, 'Action': 'sts:AssumeRole' } ]}5. From the provided policy list, select the previously created policy from Step 1 and click Next6. Set the name and description for the role and click Create role at the bottom.


Step 3: Grant Access to AWS KMS Key (only for buckets that are encrypted with customer managed KMS keys)

1. Navigate to the bucket in the AWS S3 UI2. Open the bucket Properties3. Scroll down to Default encryption and copy the AWS KMS key ARN 4. In the Policy creation step (Step 1, Sub-step 6), use the JSON below in the policy statement, and replace YOUR_KMS_KEY_ARN with the copied Key ARN for the encrypted bucket.{ 'Version': '2012-10-17', 'Statement': [ { 'Effect': 'Allow', 'Action': [ 's3:ListBucket', 's3:GetBucketLocation', 's3:*Object*' ], 'Resource': [ 'arn:aws:s3:::BUCKET_NAME', 'arn:aws:s3:::BUCKET_NAME/*' ] }, { 'Effect': 'Allow', 'Action': [ 'kms:Encrypt', 'kms:Decrypt', 'kms:ReEncrypt*', 'kms:GenerateDataKey*', 'kms:DescribeKey' ], 'Resource': [ 'YOUR_KMS_KEY_ARN' ] } ]}


Step 4: Enter the created AWS Role ARN (Step 2) into the Activeloop UI

See the first video in the link below:Storing Deep Lake Data in Your Own Cloud


Deep Lake API Reference

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Vector Store Creating a Deep Lake Vector Store Vector Store Operations Vector Store Properties VectorStore.DeepMemory Creating a Deep Memory Deep Memory Operations Deep Memory Properties Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype 3D Bounding Box Htype Intrinsics Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Nifti Htype Point Cloud Htype Mesh Htype Embedding Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read MMDetection High-Performance Features Dataloader Sampler Tensor Query Language Random Split Deep Memory API Reference deeplake deeplake.VectorStore deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log deeplake.core.transform deeplake.core.vectorstore.deep_memory deeplake.random.seed


Indices and tables

Index Module Index Search Page


Deep Lake API Reference¶

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Point Cloud Htype Mesh Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Enterprise Features Dataloader Sampler Tensor Query Language Random Split API Reference deeplake deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log deeplake.core.transform


Indices and tables¶

Index Module Index Search Page


Deep Lake API Reference¶

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Point Cloud Htype Mesh Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Enterprise Features Dataloader Sampler Tensor Query Language API Reference deeplake deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log deeplake.core.transform


deeplake.core.index¶

class deeplake.core.index.IndexEntry(value: Union[int, slice, Tuple[int]] = slice(None, None, None))¶ __getitem__(item: Union[int, slice, Tuple[int]])¶ Combines the given item and this IndexEntry.  Returns a new IndexEntry representing the composition of the two.  Examples >>> IndexEntry()[0:100] IndexEntry(slice(0, 100, None)) Copy to clipboard >>> IndexEntry()[100:200][5] IndexEntry(105) Copy to clipboard >>> IndexEntry()[(0, 1, 2, 3)] IndexEntry((0, 1, 2, 3)) Copy to clipboard >>> IndexEntry()[1, 2, 3] IndexEntry((0, 1, 2, 3)) Copy to clipboard Parameters: item – The desired sub-index to be composed with this IndexEntry.  Can be an int, a slice, or a tuple of ints.  Returns: The new IndexEntry object.  Return type: IndexEntry Raises: TypeError – An integer IndexEntry should not be indexed further.  __init__(value: Union[int, slice, Tuple[int]] = slice(None, None, None))¶ __str__()¶ Return str(self).  __weakref__¶ list of weak references to the object (if defined) indices(length: int)¶ Generates the sequence of integer indices for a target of a given length.  is_trivial()¶ Checks if an IndexEntry represents the entire slice length(parent_length: int) → int¶ Returns the length of an IndexEntry given the length of the parent it is indexing.  Examples >>> IndexEntry(slice(5, 10)).length(100) 5 >>> len(list(range(100))[5:10]) 5 >>> IndexEntry(slice(5, 100)).length(50) 45 >>> len(list(range(50))[5:100]) 45 >>> IndexEntry(0).length(10) 1 Copy to clipboard Parameters: parent_length (int) – The length of the target that this IndexEntry is indexing.  Returns: The length of the index if it were applied to a parent of the given length.  Return type: int subscriptable()¶ Returns whether an IndexEntry can be further subscripted.  validate(parent_length: int)¶ Checks that the index is not accessing values outside the range of the parent.  class deeplake.core.index.Index(item: Union[int, slice, Tuple[int], Index, List[IndexEntry]] = slice(None, None, None))¶ __getitem__(item: Union[int, slice, List[int], Tuple[Union[int, slice, Tuple[int]]], Index])¶ Returns a new Index representing a subscripting with the given item.  Modeled after NumPy's advanced integer indexing.  See: https://numpy.org/doc/stable/reference/arrays.indexing.html Examples >>> Index([5, slice(None)])[5] Index([5, 5]) Copy to clipboard >>> Index([5])[5:6] Index([5, slice(5, 6)]) Copy to clipboard >>> Index()[0, 1, 2:5, 3] Index([0, 1, slice(2, 5), 3]) Copy to clipboard >>> Index([slice(5, 6)])[(0, 1, 2:5, 3),] Index([(5, 1, slice(2, 5), 3)]) Copy to clipboard Parameters: item – The contents of the subscript expression to add to this Index.  Returns: The Index representing the result of the subscript operation.  Return type: Index Raises: TypeError – Given item should be another Index, or compatible with NumPy's advanced integer indexing.  __init__(item: Union[int, slice, Tuple[int], Index, List[IndexEntry]] = slice(None, None, None))¶ Initializes an Index from an IndexValue, another Index, or the values from another Index.  Represents a list of IndexEntry objects corresponding to indexes into each axis of an ndarray.  __repr__()¶ Return repr(self).  __str__()¶ Return str(self).  __weakref__¶ list of weak references to the object (if defined) apply(samples: List[ndarray])¶ Applies an Index to a list of ndarray samples with the same number of entries as the first entry in the Index.  apply_squeeze(samples: List[ndarray])¶ Applies the primary axis of an Index to a list of ndarray samples


deeplake.core.index¶

Will either return the list as given, or return the first sample .  compose_at(item: Union[int, slice, Tuple[int]], i: Optional[int] = None)¶ Returns a new Index representing the addition of an IndexValue, or the composition with a given axis.  Examples >>> Index([slice(None), slice(None)]).compose_at(5) Index([slice(None), slice(None), 5]) Copy to clipboard >>> Index([slice(None), slice(5, 10), slice(None)]).compose_at(3, 1) Index([slice(None), 8, slice(None)]) Copy to clipboard Parameters: item (IndexValue) – The value to append or compose with the Index.  i (int, optional) – The axis to compose with the given item.  Defaults to None, meaning that the item will be appended instead.  Returns: The result of the addition or composition.  Return type: Index find_axis(offset: int = 0)¶ Returns the index for the nth subscriptable axis in the values of an Index.  Parameters: offset (int) – The number of subscriptable axes to skip before returning.  Defaults to 0, meaning that the first valid axis is returned.  Returns: The index of the found axis, or None if no match is found.  Return type: int is_trivial()¶ Checks if an Index is equivalent to the trivial slice [:], aka slice(None).  length(parent_length: int)¶ Returns the primary length of an Index given the length of the parent it is indexing.  See: IndexEntry.length() validate(parent_length)¶ Checks that the index is not accessing values outside the range of the parent.  deeplake.core.index.merge_slices(existing_slice: slice, new_slice: slice) → slice¶ Compose two slice objects Given an iterable x, the following should be equivalent: x[existing_slice][new_slice] == x[merge_slices(existing_slice, new_slice)] Parameters: existing_slice (slice) – The existing slice to be restricted.  new_slice (slice) – The new slice to be applied to the existing slice.  Returns: the composition of the given slices Return type: slice Raises: NotImplementedError – Composing slices with negative values is not supported.  Negative indexing for slices is only supported for the first slice.  deeplake.core.index.slice_at_int(s: slice, i: int)¶ Returns the i th element of a slice s.  Examples >>> slice_at_int(slice(None), 10) 10 Copy to clipboard >>> slice_at_int(slice(10, 20, 2), 3) 16 Copy to clipboard Parameters: s (slice) – The slice to index into.  i (int) – The integer offset into the slice.  Returns: The index corresponding to the offset into the slice.  Return type: int Raises: NotImplementedError – Nontrivial slices should not be indexed with negative integers.  IndexError – If step is negative and start is not greater than stop.  deeplake.core.index.slice_length(s: slice, parent_length: int) → int¶ Returns the length of a slice given the length of its parent.


deeplake.api.dataset¶

class deeplake.api.dataset.dataset¶ static exists(path: Union[str, pathlib.Path], creds: Optional[dict] = None, token: Optional[str] = None) → bool¶ See deeplake.exists(). static empty(path: Union[str, pathlib.Path], overwrite: bool = False, public: bool = False, memory_cache_size: int = DEFAULT_MEMORY_CACHE_SIZE, local_cache_size: int = DEFAULT_LOCAL_CACHE_SIZE, creds: Optional[dict] = None, token: Optional[str] = None) → Dataset¶ See deeplake.empty(). static load(path: Union[str, pathlib.Path], read_only: Optional[bool] = None, memory_cache_size: int = DEFAULT_MEMORY_CACHE_SIZE, local_cache_size: int = DEFAULT_LOCAL_CACHE_SIZE, creds: Optional[dict] = None, token: Optional[str] = None, verbose: bool = True, access_method: str = 'stream') → Dataset¶ See deeplake.load(). static rename(old_path: Union[str, pathlib.Path], new_path: Union[str, pathlib.Path], creds: Optional[dict] = None, token: Optional[str] = None) → Dataset¶ See deeplake.rename(). static delete(path: Union[str, pathlib.Path], force: bool = False, large_ok: bool = False, creds: Optional[dict] = None, token: Optional[str] = None, verbose: bool = False) → None¶ See deeplake.delete(). static like(dest: Union[str, pathlib.Path], src: Union[str, Dataset, pathlib.Path], tensors: Optional[List[str]] = None, overwrite: bool = False, creds: Optional[dict] = None, token: Optional[str] = None, public: bool = False) → Dataset¶ See deeplake.like(). static copy(src: Union[str, pathlib.Path, Dataset], dest: Union[str, pathlib.Path], tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, src_token=None, dest_creds=None, dest_token=None, num_workers: int = 0, scheduler='threaded', progressbar=True)¶ See deeplake.copy(). static deepcopy(src: Union[str, pathlib.Path], dest: Union[str, pathlib.Path], tensors: Optional[List[str]] = None, overwrite: bool = False, src_creds=None, src_token=None, dest_creds=None, dest_token=None, num_workers: int = 0, scheduler='threaded', progressbar=True, public: bool = False, verbose: bool = True)¶ See deeplake.deepcopy(). static connect(src_path: str, creds_key: str, dest_path: Optional[str], org_id: Optional[str], ds_name: Optional[str], token: Optional[str])¶ See deeplake.connect(). static ingest(src: Union[str, pathlib.Path], dest: Union[str, pathlib.Path], images_compression: str = 'auto', dest_creds: dict = None, progressbar: bool = True, summary: bool = True, **dataset_kwargs) → Dataset¶ See deeplake.ingest(). static ingest_kaggle(tag: str, src: Union[str, pathlib.Path], dest: Union[str, pathlib.Path], exist_ok: bool = False, images_compression: str = 'auto', dest_creds: dict = None, kaggle_credentials: dict = None, progressbar: bool = True, summary: bool = True, **dataset_kwargs) → Dataset¶ See deeplake.ingest_kaggle(). static ingest_dataframe(src, dest: Union[str, pathlib.Path], dest_creds: Optional[Dict] = None, progressbar: bool = True, **dataset_kwargs)¶ See deeplake.ingest_dataframe(). static list(workspace: str = '', token: Optional[str] = None) → None¶ See deeplake.list().


deeplake.api.info¶

class deeplake.api.info.Info¶ Contains optional key-value pairs that can be stored for datasets/tensors. clear()¶ Clear info. get(key, default=None)¶ Get value for key from info. items()¶ Return all items in info. keys()¶ Return all keys in info. property nbytes¶ Returns size of info stored in bytes. pop(key, default=None)¶ Pop item from info by key. popitem()¶ Pop item from info. replace_with(d)¶ Replace info with another dictionary. setdefault(key, default=None)¶ Set default value for a key in info. update(*args, **kwargs)¶ Update info. values()¶ Return all values in info.


deeplake.api.link¶

deeplake.api.link.link(path: str, creds_key: Optional[str] = None) → LinkedSample¶ See deeplake.link().


deeplake.api.read¶

deeplake.api.read.read(path: str, verify: bool = False, creds: Optional[Dict] = None, compression: Optional[str] = None, storage: Optional[StorageProvider] = None) → Sample¶ See deeplake.read().


deeplake.api.tiled¶

deeplake.api.tiled.tiled(sample_shape: Tuple[int, ...], tile_shape: Optional[Tuple[int, ...]] = None, dtype: Union[str, np.dtype] = np.dtype('uint8'))¶ See deeplake.tiled().


deeplake.util.shape_interval¶

class deeplake.util.shape_interval.ShapeInterval¶ __init__(lower: Sequence[int], upper: Optional[Sequence[int]] = None)¶ Shapes in Deep Lake are best represented as intervals, this is to support dynamic tensors. Instead of having a single tuple of integers representing shape, we use 2 tuples of integers to represent the lower and upper bounds of the representing shape. If lower == upper for all cases, the shape is considered 'fixed'. If lower != upper for any cases, the shape is considered 'dynamic'. Parameters: lower (sequence) – Sequence of integers that represent the lower-bound shape. upper (sequence) – Sequence of integers that represent the upper-bound shape. If None is provided, lower is used as upper (implicitly fixed-shape). Raises: InvalidShapeIntervalError – If the provided lower/upper bounds are incompatible to represent a shape.


deeplake.util.remove_cache¶

deeplake.util.remove_cache.remove_memory_cache(storage: StorageProvider)¶ Removes the memory cache. deeplake.util.remove_cache.get_base_storage(storage: StorageProvider)¶ Removes all layers of caching and returns the underlying storage. deeplake.util.remove_cache.get_dataset_with_zero_size_cache(ds)¶ Returns a dataset with same storage but cache size set to zero. deeplake.util.remove_cache.create_read_copy_dataset(dataset, commit_id: Optional[str] = None)¶ Creates a read-only copy of the given dataset object, without copying underlying data. Parameters: dataset – The Dataset object to copy. commit_id – The commit id to checkout the new read-only copy to. Returns: A new Dataset object in read-only mode.


deeplake.util.notebook¶

deeplake.util.notebook.is_notebook()¶ Whether running in a notebook. deeplake.util.notebook.is_jupyter()¶ Whether running in a Jupyter notebook. deeplake.util.notebook.is_colab()¶ Whether running in a colab notebook.


deeplake.util.exceptions¶

Check if the name or id is correct.')¶ Bases: Exception class deeplake.util.exceptions.BadRequestException(message)¶ Bases: Exception class deeplake.util.exceptions.OverLimitException(message='You are over the allowed limits for this operation.')¶ Bases: Exception class deeplake.util.exceptions.ServerException(message='Internal Activeloop server error.')¶ Bases: Exception class deeplake.util.exceptions.BadGatewayException(message='Invalid response from Activeloop server.')¶ Bases: Exception class deeplake.util.exceptions.GatewayTimeoutException(message='Activeloop server took too long to respond.')¶ Bases: Exception class deeplake.util.exceptions.WaitTimeoutException(message='Timeout waiting for server state update.')¶ Bases: Exception class deeplake.util.exceptions.LockedException(message='The resource is currently locked.')¶ Bases: Exception class deeplake.util.exceptions.UnexpectedStatusCodeException(message)¶ Bases: Exception class deeplake.util.exceptions.EmptyTokenException(message='The authentication token is empty.')¶ Bases: Exception class deeplake.util.exceptions.S3Error¶ Bases: Exception Catchall for all errors encountered while working with S3 class deeplake.util.exceptions.S3GetError¶ Bases: S3Error Catchall for all errors encountered while working getting an object from S3 class deeplake.util.exceptions.S3SetError¶ Bases: S3Error Catchall for all errors encountered while working setting an object in S3 class deeplake.util.exceptions.S3DeletionError¶ Bases: S3Error Catchall for all errors encountered while working deleting an object in S3 class deeplake.util.exceptions.S3ListError¶ Bases: S3Error Catchall for all errors encountered while retrieving a list of objects present in S3 class deeplake.util.exceptions.UnsupportedCompressionError(compression: Optional[str], htype: Optional[str] = None)¶ Bases: CompressionError class deeplake.util.exceptions.SampleCompressionError(sample_shape: Tuple[int, ...], compression_format: Optional[str], message: str)¶ Bases: CompressionError class deeplake.util.exceptions.SampleDecompressionError¶ Bases: CompressionError class deeplake.util.exceptions.InvalidImageDimensions(actual_dims, expected_dims)¶ Bases: Exception class deeplake.util.exceptions.TensorUnsupportedSampleType¶ Bases: Exception class deeplake.util.exceptions.MetaError¶ Bases: Exception class deeplake.util.exceptions.MetaDoesNotExistError(key: str)¶ Bases: MetaError class deeplake.util.exceptions.MetaAlreadyExistsError(key: str, required_meta: dict)¶ Bases: MetaError class deeplake.util.exceptions.MetaInvalidKey(name: str, available_keys: List[str])¶ Bases: MetaError class deeplake.util.exceptions.MetaInvalidRequiredMetaKey(key: str, subclass_name: str)¶ Bases: MetaError class deeplake.util.exceptions.TensorMetaInvalidHtype(htype: str, available_htypes: Sequence[str])¶ Bases: MetaError class deeplake.util.exceptions.TensorMetaInvalidHtypeOverwriteValue(key: str, value: Any, explanation: str = '')¶ Bases: MetaError class deeplake.util.exceptions.TensorMetaMissingRequiredValue(htype: str, key: Union[str, List[str]])¶ Bases: MetaError class deeplake.util.exceptions.TensorMetaInvalidHtypeOverwriteKey(htype: str, key: str, available_keys: Sequence[str])¶ Bases: MetaError class deeplake.util.exceptions.TensorDtypeMismatchError(expected: Union[dtype, str], actual: str, htype: str)¶ Bases: MetaError class deeplake.util.exceptions.InvalidTensorLinkError(msg='Invalid tensor link.')¶ Bases: MetaError class deeplake.util.exceptions.TensorMetaMutuallyExclusiveKeysError(keys: Optional[List[str]] = None, custom_message: Optional[str] = None)¶ Bases: MetaError class deeplake.util.exceptions.ReadOnlyModeError(custom_message: Optional[str] = None)¶ Bases: Exception class deeplake.util.exceptions.TransformError¶ Bases: Exception class deeplake.util.exceptions.FilterError¶ Bases: Exception class deeplake.util.exceptions.InvalidInputDataError(operation)¶ Bases: TransformError class deeplake.util.exceptions.UnsupportedSchedulerError(scheduler)¶ Bases: TransformError class deeplake.util.exceptions.TensorMismatchError(tensors, output_keys, skip_ok=False)¶ Bases: TransformError class deeplake.util.exceptions.InvalidOutputDatasetError(message='The output Dataset to transform should not be `read_only`.')¶ Bases: TransformError class deeplake.util.exceptions.InvalidTransformDataset(message='The TransformDataset (2nd argument to transform function) of one of the functions is invalid


deeplake.util.exceptions¶

class deeplake.util.exceptions.ExternalCommandError(command: str, status: int)¶ Bases: Exception class deeplake.util.exceptions.KaggleError¶ Bases: Exception class deeplake.util.exceptions.KaggleMissingCredentialsError(env_var_name: str)¶ Bases: KaggleError class deeplake.util.exceptions.KaggleDatasetAlreadyDownloadedError(tag: str, path: str)¶ Bases: KaggleError class deeplake.util.exceptions.InvalidPathException(directory)¶ Bases: Exception class deeplake.util.exceptions.AutoCompressionError(directory)¶ Bases: Exception class deeplake.util.exceptions.InvalidFileExtension(directory)¶ Bases: Exception class deeplake.util.exceptions.SamePathException(directory)¶ Bases: Exception class deeplake.util.exceptions.TensorInvalidSampleShapeError(shape: Sequence[int], expected_dims: int)¶ Bases: Exception class deeplake.util.exceptions.TensorMetaMissingKey(key: str, meta: dict)¶ Bases: Exception class deeplake.util.exceptions.TensorDoesNotExistError(tensor_name: str)¶ Bases: KeyError, AttributeError class deeplake.util.exceptions.TensorAlreadyExistsError(key: str)¶ Bases: Exception class deeplake.util.exceptions.TensorGroupDoesNotExistError(group_name: str)¶ Bases: KeyError class deeplake.util.exceptions.TensorGroupAlreadyExistsError(key: str)¶ Bases: Exception class deeplake.util.exceptions.InvalidTensorNameError(name: str)¶ Bases: Exception class deeplake.util.exceptions.InvalidTensorGroupNameError(name: str)¶ Bases: Exception class deeplake.util.exceptions.DynamicTensorNumpyError(key: str, index, property_key: str)¶ Bases: Exception class deeplake.util.exceptions.InvalidShapeIntervalError(message: str, lower: Optional[Sequence[int]] = None, upper: Optional[Sequence[int]] = None)¶ Bases: Exception class deeplake.util.exceptions.InvalidKeyTypeError(item: Any)¶ Bases: TypeError class deeplake.util.exceptions.UnsupportedTensorTypeError(item: Any)¶ Bases: TypeError class deeplake.util.exceptions.InvalidBytesRequestedError¶ Bases: Exception class deeplake.util.exceptions.ProviderListEmptyError¶ Bases: Exception class deeplake.util.exceptions.DirectoryAtPathException¶ Bases: Exception class deeplake.util.exceptions.FileAtPathException(path)¶ Bases: Exception class deeplake.util.exceptions.ProviderSizeListMismatch¶ Bases: Exception class deeplake.util.exceptions.ModuleNotInstalledException(message)¶ Bases: Exception class deeplake.util.exceptions.LoginException(message='Error while logging in, invalid auth token.  Please try logging in again.')¶ Bases: Exception class deeplake.util.exceptions.UserNotLoggedInException(message='')¶ Bases: Exception class deeplake.util.exceptions.InvalidHubPathException(path)¶ Bases: Exception class deeplake.util.exceptions.PathNotEmptyException(use_hub=True)¶ Bases: Exception class deeplake.util.exceptions.AuthenticationException(message='Authentication failed.  Please try logging in again.')¶ Bases: Exception class deeplake.util.exceptions.AuthorizationException(message='You are not authorized to access this resource on Activeloop Server.', response=None)¶ Bases: Exception class deeplake.util.exceptions.InvalidPasswordException(message='The password you provided was invalid.')¶ Bases: AuthorizationException class deeplake.util.exceptions.CouldNotCreateNewDatasetException(path: str)¶ Bases: AuthorizationException class deeplake.util.exceptions.ResourceNotFoundException(message='The resource you are looking for was not found


deeplake.util.exceptions¶

All the tensors should have equal length for it to be valid.')¶ Bases: TransformError class deeplake.util.exceptions.HubComposeEmptyListError(message='Cannot deeplake.compose an empty list.')¶ Bases: TransformError class deeplake.util.exceptions.HubComposeIncompatibleFunction(index: int)¶ Bases: TransformError class deeplake.util.exceptions.DatasetUnsupportedPytorch(reason)¶ Bases: Exception class deeplake.util.exceptions.CorruptedMetaError¶ Bases: Exception class deeplake.util.exceptions.ChunkEngineError¶ Bases: Exception class deeplake.util.exceptions.FullChunkError¶ Bases: ChunkEngineError class deeplake.util.exceptions.ChunkIdEncoderError¶ Bases: ChunkEngineError class deeplake.util.exceptions.ChunkSizeTooSmallError(message='If the size of the last chunk is given, it must be smaller than the requested chunk size.')¶ Bases: ChunkEngineError class deeplake.util.exceptions.DatasetHandlerError(message)¶ Bases: Exception class deeplake.util.exceptions.MemoryDatasetCanNotBePickledError¶ Bases: Exception class deeplake.util.exceptions.CorruptedSampleError(compression)¶ Bases: Exception class deeplake.util.exceptions.VersionControlError¶ Bases: Exception class deeplake.util.exceptions.MergeError¶ Bases: Exception class deeplake.util.exceptions.MergeNotSupportedError¶ Bases: MergeError class deeplake.util.exceptions.MergeMismatchError(tensor_name, mismatch_type, original_value, target_value)¶ Bases: MergeError class deeplake.util.exceptions.MergeConflictError(conflict_tensors=None, message='')¶ Bases: MergeError class deeplake.util.exceptions.CheckoutError¶ Bases: VersionControlError class deeplake.util.exceptions.CommitError¶ Bases: VersionControlError class deeplake.util.exceptions.EmptyCommitError¶ Bases: CommitError class deeplake.util.exceptions.TensorModifiedError¶ Bases: Exception class deeplake.util.exceptions.GCSDefaultCredsNotFoundError¶ Bases: Exception class deeplake.util.exceptions.InvalidOperationError(method: str, type: str)¶ Bases: Exception class deeplake.util.exceptions.AgreementError¶ Bases: Exception class deeplake.util.exceptions.AgreementNotAcceptedError(agreements=None)¶ Bases: AgreementError class deeplake.util.exceptions.NotLoggedInError(msg=None)¶ Bases: AgreementError class deeplake.util.exceptions.RenameError(msg='Only name of the dataset can be different in new path.')¶ Bases: Exception class deeplake.util.exceptions.BufferError¶ Bases: Exception class deeplake.util.exceptions.InfoError¶ Bases: Exception class deeplake.util.exceptions.OutOfChunkCountError¶ Bases: Exception class deeplake.util.exceptions.OutOfSampleCountError¶ Bases: Exception class deeplake.util.exceptions.SampleHtypeMismatchError(htype, sample_type)¶ Bases: Exception class deeplake.util.exceptions.EmptyTensorError(message)¶ Bases: Exception class deeplake.util.exceptions.DatasetViewSavingError¶ Bases: Exception class deeplake.util.exceptions.ManagedCredentialsNotFoundError(org_id, creds_key)¶ Bases: Exception class deeplake.util.exceptions.UnableToReadFromUrlError(url, status_code)¶ Bases: Exception class deeplake.util.exceptions.InvalidTokenException¶ Bases: Exception class deeplake.util.exceptions.TokenPermissionError¶ Bases: Exception


deeplake.auto.unstructured.kaggle¶

deeplake.auto.unstructured.kaggle.download_kaggle_dataset(tag: str, local_path: str, kaggle_credentials: Optional[dict] = None, exist_ok: bool = False)¶ Calls the kaggle API (https://www.kaggle.com/docs/api) to download a kaggle dataset and unzip it's contents. Parameters: tag (str) – Kaggle dataset tag. Example: 'coloradokb/dandelionimages' points to https://www.kaggle.com/coloradokb/dandelionimages local_path (str) – Path where the kaggle dataset will be downloaded and unzipped. Only local path downloading is supported. kaggle_credentials (dict) – Credentials are gathered from the environment variables or ~/kaggle.json. If those don't exist, the kaggle_credentials argument will be used. exist_ok (bool) – If the kaggle dataset was already downloaded, and exist_ok is True, no error is thrown. Raises: KaggleMissingCredentialsError – If no kaggle credentials are found. KaggleDatasetAlreadyDownloadedError – If the dataset tag already exists in local_path.


Deep Lake API Reference¶

Deep Lake is an open-source database for AI.Getting Started Installation Key Concepts Datasets Creating Datasets Loading Datasets Deleting and Renaming Datasets Copying Datasets Dataset Operations Dataset Visualization Dataset Credentials Dataset Properties Dataset Version Control Dataset Views Tensors Creating Tensors Deleting and Renaming Tensors Adding and deleting samples Retrieving samples Tensor Properties Info Video features Htypes Image Htype Video Htype Audio Htype Class Label Htype Bounding Box Htype Segmentation Mask Htype Binary Mask Htype COCO Keypoints Htype Point Htype Polygon Htype Point Cloud Htype Mesh Htype Sequence htype Link htype Compressions Sample Compression Chunk Compression PyTorch and Tensorflow Support Utility Functions General Functions Making Deep Lake Samples Parallelism Integrations Weights and Biases Logging Dataset Creation Logging Dataset Read Experimental API Dataloader Tensor Query Language API Reference deeplake deeplake.core deeplake.core.dataset deeplake.core.tensor deeplake.api deeplake.auto deeplake.util deeplake.client.log


Indices and tables¶

Index Module Index Search Page


deeplake.core.sample¶

class deeplake.core.sample.Sample¶ __init__(path: str | None = None, array: ndarray | None = None, buffer: bytes | memoryview | None = None, compression: str | None = None, verify: bool = False, shape: Tuple[int] | None = None, dtype: str | None = None, creds: Dict | None = None, storage: StorageProvider | None = None)¶ Represents a single sample for a tensor. Provides all important meta information in one place. Note If self.is_lazy is True, this Sample doesn't actually have any data loaded. To read this data, simply try to read it into a numpy array (sample.array) Parameters: path (str) – Path to a sample stored on the local file system that represents a single sample. If path is provided, array should not be. Implicitly makes self.is_lazy == True. array (np.ndarray) – Array that represents a single sample. If array is provided, path should not be. Implicitly makes self.is_lazy == False. buffer – (bytes): Byte buffer that represents a single sample. If compressed, compression argument should be provided. compression (str) – Specify in case of byte buffer. verify (bool) – If a path is provided, verifies the sample if True. shape (Tuple[int]) – Shape of the sample. dtype (optional, str) – Data type of the sample. creds (optional, Dict) – Credentials for s3, gcp and http urls. storage (optional, StorageProvider) – Storage provider. Raises: ValueError – Cannot create a sample from both a path and array. property array: ndarray¶ Return numpy array corresponding to the sample. Decompresses the sample if necessary. Example >>> sample = deeplake.read('./images/dog.jpg') >>> arr = sample.array >>> arr.shape (323, 480, 3) Copy to clipboard compressed_bytes(compression: str | None) → bytes¶ Returns this sample as compressed bytes. Note If this sample is pointing to a path and the requested compression is the same as it's stored in, the data is returned without re-compressing. Parameters: compression (Optional[str]) – self.array will be compressed into this format. If compression is None, return uncompressed_bytes(). Returns: Bytes for the compressed sample. Contains all metadata required to decompress within these bytes. Return type: bytes Raises: ValueError – On recompression of unsupported formats. uncompressed_bytes() → bytes | None¶ Returns uncompressed bytes.


deeplake.core.linked_sample¶

class deeplake.core.linked_sample.LinkedSample(path: str, creds_key: str | None = None)¶ Represents a sample that is initialized using external links. See deeplake.link().


deeplake.core.partial_sample¶

class deeplake.core.partial_sample.PartialSample(sample_shape: Tuple[int, ...], tile_shape: Tuple[int, ...] | None = None, dtype: str | dtype = dtype('uint8'))¶ Represents a sample that is initialized by just shape and the data is updated later.


deeplake.core.storage¶

Called at the end of methods which write data, to ensure consistency as a default .  set_bytes(path: str, value: bytes, start_byte: int | None = None, overwrite: bool | None = False)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  start_byte (int, optional) – If only specific bytes starting from start_byte are to be assigned.  overwrite (boolean, optional) – If the value is True, if there is an object present at the path it is completely overwritten, without fetching it's data.  Raises: InvalidBytesRequestedError – If start_byte < 0.  ReadOnlyModeError – If the provider is in read-only mode.  LRU Cache¶ class deeplake.core.storage.LRUCache¶ Bases: StorageProvider LRU Cache that uses StorageProvider for caching __delitem__(path: str)¶ Deletes the object present at the path from the cache and the underlying storage.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ If item is in cache_storage, retrieves from there and returns.  If item isn't in cache_storage, retrieves from next storage, stores in cache_storage (if possible) and returns.  Parameters: path (str) – The path relative to the root of the underlying storage.  Raises: KeyError – if an object is not found at the path.  Returns: The bytes of the object present at the path.  Return type: bytes __getstate__() → Dict[str, Any]¶ Returns the state of the cache, for pickling __init__(cache_storage: StorageProvider, next_storage: StorageProvider | None, cache_size: int)¶ Initializes the LRUCache.  It can be chained with other LRUCache objects to create multilayer caches.  Parameters: cache_storage (StorageProvider) – The storage being used as the caching layer of the cache.  This should be a base provider such as MemoryProvider, LocalProvider or S3Provider but not another LRUCache.  next_storage (StorageProvider) – The next storage layer of the cache.  This can either be a base provider (i.e.  it is the final storage) or another LRUCache (i.e.  in case of chained cache).  While reading data, all misses from cache would be retrieved from here.  While writing data, the data will be written to the next_storage when cache_storage is full or flush is called.  cache_size (int) – The total space that can be used from the cache_storage in bytes.  This number may be less than the actual space available on the cache_storage.  Setting it to a higher value than actually available space may lead to unexpected behaviors.  __iter__()¶ Generator function that iterates over the keys of the cache and the underlying storage.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider


deeplake.core.storage¶

Base Storage Provider¶ class deeplake.core.storage.StorageProvider¶ abstract __delitem__(path: str)¶ Delete the object present at the path.  Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  abstract __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  abstract __iter__()¶ Generator function that iterates over the keys of the provider.  Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  abstract __len__()¶ Returns the number of files present inside the root of the provider.  Returns: the number of files present inside the root.  Return type: int abstract __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  __weakref__¶ list of weak references to the object (if defined) abstract _all_keys() → Set[str]¶ Generator function that iterates over the keys of the provider.  Returns: set of all keys present at the root of the provider.  Return type: set _is_hub_path = False¶ An abstract base class for implementing a storage provider.  To add a new provider using Provider, create a subclass and implement all 5 abstract methods below.  check_readonly()¶ Raises an exception if the provider is in read-only mode.  abstract clear(prefix='')¶ Delete the contents of the provider.  copy()¶ Returns a copy of the provider.  Returns: A copy of the provider.  Return type: StorageProvider disable_readonly()¶ Disables read-only mode for the provider.  enable_readonly()¶ Enables read-only mode for the provider.  flush()¶ Only needs to be implemented for caches.  Flushes the data to the next storage provider.  Should be a no op for Base Storage Providers like local, s3, azure, gcs, etc.  get_bytes(path: str, start_byte: int | None = None, end_byte: int | None = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  maybe_flush()¶ Flush cache if autoflush has been enabled


deeplake.core.storage¶

__len__()¶ Returns the number of files present in the cache and the underlying storage .  Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes | DeepLakeMemoryObject)¶ Puts the item in the cache_storage (if possible), else writes to next_storage.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  __setstate__(state: Dict[str, Any])¶ Recreates a cache with the same configuration as the state.  Parameters: state (dict) – The state to be used to recreate the cache.  Note While restoring the cache, we reset its contents.  In case the cache storage was local/s3 and is still accessible when unpickled (if same machine/s3 creds present respectively), the earlier cache contents are no longer accessible.  _all_keys()¶ Helper function that lists all the objects present in the cache and the underlying storage.  Returns: set of all the objects found in the cache and the underlying storage.  Return type: set _flush_if_not_read_only()¶ Flushes the cache if not in read-only mode.  _forward(path)¶ Forward the value at a given path to the next storage, and un-marks its key.  _forward_value(path, value)¶ Forwards a path-value pair to the next storage, and un-marks its key.  Parameters: path (str) – the path to the object relative to the root of the provider.  value (bytes, DeepLakeMemoryObject) – the value to send to the next storage.  _free_up_space(extra_size: int)¶ Helper function that frees up space the requred space in cache.No action is taken if there is sufficient space in the cache.  Parameters: extra_size (int) – the space that needs is required in bytes.  _insert_in_cache(path: str, value: bytes | DeepLakeMemoryObject)¶ Helper function that adds a key value pair to the cache.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  _pop_from_cache()¶ Helper function that pops the least recently used key, value pair from the cache clear(prefix='')¶ Deletes ALL the data from all the layers of the cache and the actual storage.  This is an IRREVERSIBLE operation.  Data once deleted can not be recovered.  clear_cache()¶ Flushes the content of all the cache layers if not in read mode and and then deletes contents of all the layers of it.  This doesn't delete data from the actual storage.  clear_deeplake_objects()¶ Removes all DeepLakeMemoryObjects from the cache.  flush()¶ Writes data from cache_storage to next_storage.  Only the dirty keys are written.  This is a cascading function and leads to data being written to the final storage in case of a chained cache