Htypes

Examples Appending polygons with 2-D points >>> poly1 = [(1, 2), (2, 3), (3, 4)] >>> poly2 = [(10, 12), (14, 19)] >>> poly3 = [(33, 32), (54, 67), (67, 43), (56, 98)] >>> sample = [poly1, poly2, poly3] >>> ds.polygons.append(sample) Copy to clipboard Appending polygons with 3-D points >>> poly1 = [(10, 2, 9), (12, 3, 8), (12, 10, 4)] >>> poly2 = [(10, 1, 8), (5, 17, 11)] >>> poly3 = [(33, 33, 31), (45, 76, 13), (60, 24, 17), (67, 87, 83)] >>> sample = [poly1, poly2, poly3] >>> ds.polygons.append(sample) Copy to clipboard Appending polygons with numpy arrays >>> import numpy as np >>> sample = np.random.randint(0, 10, (5, 7, 2)) # 5 polygons with 7 points >>> ds.polygons.append(sample) Copy to clipboard >>> import numpy as np >>> poly1 = np.random.randint(0, 10, (5, 2)) >>> poly2 = np.random.randint(0, 10, (8, 2)) >>> poly3 = np.random.randint(0, 10, (3, 2)) >>> sample = [poly1, poly2, poly3] >>> ds.polygons.append(sample) Copy to clipboard Nifti Htype Sample dimensions: (# height, # width, # slices) or (# height, # width, # slices, # time unit) in case of time-series data .  Creating a nifti tensor A nifti tensor can be created using >>> ds.create_tensor('patients', htype='nifti', sample_compression='nii.gz') Copy to clipboard Supported compressions: >>> ['nii.gz', 'nii', None] Copy to clipboard Appending nifti data Nifti samples can be of type np.ndarray or Sample which is returned by deeplake.read().  Deep Lake does not support compression of raw nifti data.  Therefore, array of raw frames can only be appended to tensors with None compression.  Examples >>> ds.patients.append(deeplake.read('data/patient0.nii.gz')) Copy to clipboard >>> ds.patients.extend([deeplake.read(f'data/patient{i}.nii.gz') for i in range(10)]) Copy to clipboard Point Cloud Htype Sample dimensions: (# num_points, 3) Point cloud samples can be of type np.ndarray or Sample which is returned by deeplake.read().  Each point cloud is a list / array of points.  All points in a sample should have the same number of co-ordinates.  Different point clouds can have different number of points.  Creating a point cloud tensor A point cloud tensor can be created using >>> ds.create_tensor('point_clouds', htype='point_cloud', sample_compression='las') Copy to clipboard Optional args: sample_compression Supported compressions: >>> [None, 'las'] Copy to clipboard Appending point clouds Point clouds can be appended as a np.ndarray.  Examples Appending point clouds with numpy arrays >>> import numpy as np >>> point_cloud1 = np.random.randint(0, 10, (5, 3)) >>> ds.point_clouds.append(point_cloud1) >>> point_cloud2 = np.random.randint(0, 10, (15, 3)) >>> ds.point_clouds.append(point_cloud2) >>> ds.point_clouds.shape >>> (2, None, 3) Copy to clipboard Or we can use deeplake.read() method to add samples >>> import deeplake as dp >>> sample = dp.read('example.las') # point cloud with 100 points >>> ds.point_cloud.append(sample) >>> ds.point_cloud.shape >>> (1, 100, 3) Copy to clipboard Mesh Htype Sample dimensions: (# num_points, 3) Mesh samples can be of type np.ndarray or Sample which is returned by deeplake.read().  Each sample in a tensor of mesh htype is a mesh array (3-D object data).  Each mesh is a list / array of points


Htypes

Examples >>> ds = deeplake.dataset('......') Copy to clipboard Adding credentials to the dataset You can add the names of the credentials you want to use (not needed for http/local urls) >>> ds.add_creds_key('MY_S3_KEY') >>> ds.add_creds_key('GCS_KEY') Copy to clipboard and populate the added names with credentials dictionaries >>> ds.populate_creds('MY_S3_KEY', {}) # add creds here >>> ds.populate_creds('GCS_KEY', {}) # add creds here Copy to clipboard These creds are only present temporarily and will have to be repopulated on every reload .  For datasets connected to Activeloop Platform, you can store your credentials on the platform as Managed Credentials and use them just by adding the keys to your dataset.  For example if you have managed credentials with names 'my_s3_creds', 'my_gcs_creds', you can add them to your dataset using Dataset.add_creds_key without having to populate them.  >>> ds.add_creds_key('my_s3_creds', managed=True) >>> ds.add_creds_key('my_gcs_creds', managed=True) Copy to clipboard Create a link tensor >>> ds.create_tensor('img', htype='link[image]', sample_compression='jpg') Copy to clipboard Populate the tensor with links >>> ds.img.append(deeplake.link('s3://abc/def.jpeg', creds_key='my_s3_key')) >>> ds.img.append(deeplake.link('gcs://ghi/jkl.png', creds_key='GCS_KEY')) >>> ds.img.append(deeplake.link('https://picsum.photos/200/300')) # http path doesn't need creds >>> ds.img.append(deeplake.link('./path/to/cat.jpeg')) # local path doesn't need creds >>> ds.img.append(deeplake.link('s3://abc/def.jpeg')) # this will throw an exception as cloud paths always need creds_key :bluebold:`Accessing the data` Copy to clipboard >>> for i in range(5): ...  ds.img[i].numpy() ...  Copy to clipboard Updating a sample >>> ds.img[0] = deeplake.link('./data/cat.jpeg') Copy to clipboard


Image Htype

Sample dimensions: (height, width, # channels) or (height, width). Images can be stored in Deep Lake as compressed bytes or as raw arrays. Due to the high compression ratio for most image formats, it is highly recommended to store compressed images using the sample_compression input to the create_tensor method. Creating an image tensor An image tensor can be created using >>> ds.create_tensor('images', htype='image', sample_compression='jpg') Copy to clipboard OR >>> ds.create_tensor('images', htype='image', chunk_compression='jpg') Copy to clipboard Optional args: dtype: Defaults to uint8. Supported compressions: >>> [None, 'bmp', 'dib', 'gif', 'ico', 'jpeg', 'jpeg2000', 'pcx', 'png', 'ppm', 'sgi', 'tga', 'tiff', ... 'webp', 'wmf', 'xbm', 'eps', 'fli', 'im', 'msp', 'mpo'] Copy to clipboard Appending image samples Image samples can be of type np.ndarray or Deep Lake Sample which can be created using deeplake.read(). Examples Appending pixel data with array >>> ds.images.append(np.zeros((5, 5, 3), dtype=np.uint8)) Copy to clipboard Appening Deep Lake image sample >>> ds.images.append(deeplake.read('images/0001.jpg')) Copy to clipboard You can append multiple samples at the same time using extend(). >>> ds.images.extend([deeplake.read(f'images/000{i}.jpg') for i in range(10)]) Copy to clipboard Note If the compression format of the input sample does not match the sample_compression of the tensor, Deep Lake will decompress and recompress the image for storage, which may significantly slow down the upload process. The upload process is fastest when the image compression matches the sample_compression. image.rgb and image.gray htypes image.rgb and image.gray htypes can be used to force your samples to be of RGB or grayscale type. i.e., if RGB images are appended to an image.gray tensor, Deep Lake will convert them to grayscale and if grayscale images are appended to an image.rgb tensor, Deep Lake will convert them to RGB format. image.rgb and image.gray tensors can be created using >>> ds.create_tensor('rgb_images', htype='image.rgb', sample_compression='...') Copy to clipboard >>> ds.create_tensor('gray_images', htype='image.gray', sample_compression='...') Copy to clipboard


Creating an image tensor

An image tensor can be created using Copy to clipboard OR Copy to clipboard Optional args: dtype: Defaults to uint8. Supported compressions: Copy to clipboard


Video Htype

Sample dimensions: (# frames, height, width, # channels) or (# frames, height, width) Creating a video tensor A video tensor can be created using >>> ds.create_tensor('videos', htype='video', sample_compression='mp4') Copy to clipboard Optional args: dtype: Defaults to uint8. Supported compressions: >>> [None, 'mp4', 'mkv', 'avi'] Copy to clipboard Appending video samples Video samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Deep Lake does not support compression of raw video frames. Therefore, array of raw frames can only be appended to tensors with None compression. Recompression of samples read with deeplake.read is also not supported. Examples Appending Deep Lake video sample >>> ds.videos.append(deeplake.read('videos/0012.mp4')) Copy to clipboard Extending with multiple videos >>> ds.videos.extend([deeplake.read(f'videos/00{i}.mp4') for i in range(10)]) Copy to clipboard


Appending image samples

Image samples can be of type np.ndarray or Deep Lake Sample which can be created using deeplake.read(). ExamplesAppending pixel data with array Copy to clipboard Appening Deep Lake image sample Copy to clipboard You can append multiple samples at the same time using extend(). Copy to clipboard NoteIf the compression format of the input sample does not match the sample_compression of the tensor, Deep Lake will decompress and recompress the image for storage, which may significantly slow down the upload process. The upload process is fastest when the image compression matches the sample_compression.


image.rgb and image.gray htypes

image.rgb and image.gray htypes can be used to force your samples to be of RGB or grayscale type. i.e., if RGB images are appended to an image.gray tensor, Deep Lake will convert them to grayscale and if grayscale images are appended to an image.rgb tensor, Deep Lake will convert them to RGB format.image.rgb and image.gray tensors can be created using Copy to clipboard Copy to clipboard


Appending video samples

Video samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Deep Lake does not support compression of raw video frames. Therefore, array of raw frames can only be appended to tensors with None compression. Recompression of samples read with deeplake.read is also not supported. ExamplesAppending Deep Lake video sample Copy to clipboard Extending with multiple videos Copy to clipboard


Audio Htype

Sample dimensions: (# samples in audio, # channels) or (# samples in audio,) Creating an audio tensor An audio tensor can be created using >>> ds.create_tensor('audios', htype='audio', sample_compression='mp3') Copy to clipboard Optional args: dtype: Defaults to float64. Supported compressions: >>> [None, 'mp3', 'wav', 'flac'] Copy to clipboard Appending audio samples Audio samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Like videos, Deep Lake does not support compression or recompression of input audio samples. Thus, samples of type np.ndarray can only be appended to tensors with None compression. Examples Appending Deep Lake audio sample >>> ds.audios.append(deeplake.read('audios/001.mp3')) Copy to clipboard Extending with Deep Lake audio samples >>> ds.audios.extend([deeplake.read(f'videos/00{i}.mp3') for i in range(10)]) Copy to clipboard


Creating a video tensor

A video tensor can be created using Copy to clipboard Optional args: dtype: Defaults to uint8. Supported compressions: Copy to clipboard


Appending audio samples

Audio samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Like videos, Deep Lake does not support compression or recompression of input audio samples. Thus, samples of type np.ndarray can only be appended to tensors with None compression. ExamplesAppending Deep Lake audio sample Copy to clipboard Extending with Deep Lake audio samples Copy to clipboard


Class Label Htype

Sample dimensions: (# labels,) Class labels are stored as numerical values in tensors, which are indices of the list tensor.info.class_names. Creating a class label tensor A class label tensor can be created using >>> classes = ['airplanes', 'cars', 'birds', 'cats', 'deer', 'dogs', 'frogs', 'horses', 'ships', 'trucks'] >>> ds.create_tensor('labels', htype='class_label', class_names=classes, chunk_compression='lz4') Copy to clipboard Optional args: class_names: This must be a list of strings. tensor.info.class_names will be set to this list. sample_compression or chunk_compression. dtype: Defaults to uint32. Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set the class names after tensor creation. >>> ds.labels.info.update(class_names = ['airplanes', 'cars', 'birds', 'cats', 'deer', 'dogs', 'frogs', 'horses', 'ships', 'trucks']) Copy to clipboard Note If specifying compression, since the number of labels in one sample will be too low, chunk_compression would be the better option to use. Appending class labels Class labels can be appended as int, str, np.ndarray or list of int or str. In case of strings, tensor.info.class_names is updated automatically. Examples Appending index >>> ds.labels.append(0) >>> ds.labels.append(np.zeros((5,), dtype=np.uint32)) Copy to clipboard Extending with list of indices >>> ds.labels.extend([[0, 1, 2], [1, 3]]) Copy to clipboard Appending text labels >>> ds.labels.append(['cars', 'airplanes']) Copy to clipboard


Creating a class label tensor

A class label tensor can be created using Copy to clipboard Optional args: class_names: This must be a list of strings. tensor.info.class_names will be set to this list. sample_compression or chunk_compression. dtype: Defaults to uint32. Supported compressions: Copy to clipboard You can also choose to set the class names after tensor creation. Copy to clipboard NoteIf specifying compression, since the number of labels in one sample will be too low, chunk_compression would be the better option to use.


Creating an audio tensor

An audio tensor can be created using Copy to clipboard Optional args: dtype: Defaults to float64. Supported compressions: Copy to clipboard


Bounding Box Htype

Sample dimensions: (# bounding boxes, 4) Bounding boxes have a variety of conventions such as those used in YOLO, COCO, Pascal-VOC and others. In order for bounding boxes to be correctly displayed by the visualizer, the format of the bounding box must be specified in the coords key in tensor meta information mentioned below. Creating a bbox tensor A bbox tensor can be created using >>> ds.create_tensor('boxes', htype='bbox', coords={'type': 'fractional', 'mode': 'CCWH'}) Copy to clipboard Optional args: coords: A dictionary with keys 'type' and 'mode'. type: Specifies the units of bounding box coordinates. 'pixel': is in unit of pixels. 'fractional': is in units relative to the width and height of the image, such as in YOLO format. mode: Specifies the convention for the 4 coordinates 'LTRB': left_x, top_y, right_x, bottom_y 'LTWH': left_x, top_y, width, height 'CCWH': center_x, center_y, width, height dtype: Defaults to float32. sample_compression or chunk_compression. Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set the class names after tensor creation. >>> ds.boxes.info.update(coords = {'type': 'pixel', 'mode': 'LTRB'}) Copy to clipboard Note If the bounding box format is not specified, the visualizer will assume a YOLO format (fractional + CCWH) if the box coordinates are < 1 on average. Otherwise, it will assume the COCO format (pixel + LTWH). Appending bounding boxes Bounding boxes can be appended as np.ndarrays or list or lists of arrays. Examples Appending one bounding box >>> box array([[462, 123, 238, 98]]) >>> ds.boxes.append(box) Copy to clipboard Appending sample with 3 bounding boxes >>> boxes array([[965, 110, 262, 77], [462, 123, 238, 98], [688, 108, 279, 116]]) >>> boxes.shape (3, 4) >>> ds.boxes.append(boxes) Copy to clipboard


Creating a bbox tensor

A bbox tensor can be created using Copy to clipboard Optional args: coords: A dictionary with keys 'type' and 'mode'. type: Specifies the units of bounding box coordinates. 'pixel': is in unit of pixels. 'fractional': is in units relative to the width and height of the image, such as in YOLO format. mode: Specifies the convention for the 4 coordinates 'LTRB': left_x, top_y, right_x, bottom_y 'LTWH': left_x, top_y, width, height 'CCWH': center_x, center_y, width, height dtype: Defaults to float32. sample_compression or chunk_compression. Supported compressions: Copy to clipboard You can also choose to set the class names after tensor creation. Copy to clipboard NoteIf the bounding box format is not specified, the visualizer will assume a YOLO format (fractional + CCWH) if the box coordinates are < 1 on average. Otherwise, it will assume the COCO format (pixel + LTWH).


Appending bounding boxes

Bounding boxes can be appended as np.ndarrays or list or lists of arrays. ExamplesAppending one bounding box Copy to clipboard Appending sample with 3 bounding boxes Copy to clipboard


Appending class labels

Class labels can be appended as int, str, np.ndarray or list of int or str. In case of strings, tensor.info.class_names is updated automatically. ExamplesAppending index Copy to clipboard Extending with list of indices Copy to clipboard Appending text labels Copy to clipboard


3D Bounding Box Htype

In order for 3D bounding boxes to be correctly displayed by the visualizer, the format of the bounding box must be specified in the coords key in tensor meta information mentioned below. Creating a 3d bbox tensor Note In order for 3D bounding boxes to be correctly displayed by the visualizer, the format of the bounding box must be specified in the coords key in tensor meta information mentioned below. In addition, for projecting 3D bounding boxes onto 2D data (such as an image), the intrinsics tensor must exist in the dataset, or the intrinsics matrix must be specified in the ds.img_tensor.info dictionary, where the key is 'intrinsics' and the value is the matrix. A 3d bbox tensor can be created using >>> ds.create_tensor('3d_boxes', htype='bbox.3d', coords={'mode': 'center'}) Copy to clipboard Optional args: coords: A dictionary with key 'mode'. mode: Specifies the convention for the bbox coordinates. 'center': [center_x, center_y, center_z, size_x, size_y, size_z, rot_x, rot_y, rot_z] Sample dimensions: (# bounding boxes, 9) size_x - is the length of the bounding box along x direction size_y - is the width of the bounding box along y direction size_z - is the height of the bounding box along z direction rot_x - rotation angle along x axis, given in degrees rot_y - rotation angle along y axis, given in degrees rot_z - rotation angle along z axis, given in degrees 'vertex': 8 3D vertices - [[x0, y0, z0], [x1, y1, z1], [x2, y2, z2], ….., [x7, y7, z7]] Sample dimensions: (# bounding boxes, 8, 3) The vertex order is of the following form: 4_____________________ 5 /| /| / | / | / | / | /___|_________________/ | 0| | | 1 | | | | | | | | | | | | | | |_________________|___| | / 7 | / 6 | / | / | / | / |/_____________________|/ 3 2 Copy to clipboard dtype: Defaults to float32. sample_compression or chunk_compression. Supported compressions: >>> ['lz4'] Copy to clipboard Note rotation angles are specified in degrees, not radians Appending 3d bounding boxes Bounding boxes can be appended as np.ndarrays or list or lists of arrays. Examples Appending one bounding box >>> box array([[462, 123, 238, 98, 22, 36, 44, 18, 0, 36, 0]]) >>> ds.3d_boxes.append(box) Copy to clipboard Appending sample with 3 bounding boxes >>> boxes array([[965, 110, 262, 77, 22, 36, 44, 18, 0, 28, 0], [462, 123, 238, 98, 26, 34, 24, 19, 0, -50, 0], [688, 108, 279, 116, 12, 32, 14, 38, 0, 30, 0]]) >>> boxes.shape (9, 4) >>> ds.3d_boxes.append(boxes) Copy to clipboard


Creating a 3d bbox tensor

NoteIn order for 3D bounding boxes to be correctly displayed by the visualizer, the format of the bounding box must be specified in the coords key in tensor meta information mentioned below. In addition, for projecting 3D bounding boxes onto 2D data (such as an image), the intrinsics tensor must exist in the dataset, or the intrinsics matrix must be specified in the ds.img_tensor.info dictionary, where the key is 'intrinsics' and the value is the matrix.A 3d bbox tensor can be created using Copy to clipboard Optional args: coords: A dictionary with key 'mode'. mode: Specifies the convention for the bbox coordinates. 'center': [center_x, center_y, center_z, size_x, size_y, size_z, rot_x, rot_y, rot_z] Sample dimensions: (# bounding boxes, 9) size_x - is the length of the bounding box along x direction size_y - is the width of the bounding box along y direction size_z - is the height of the bounding box along z direction rot_x - rotation angle along x axis, given in degrees rot_y - rotation angle along y axis, given in degrees rot_z - rotation angle along z axis, given in degrees 'vertex': 8 3D vertices - [[x0, y0, z0], [x1, y1, z1], [x2, y2, z2], ….., [x7, y7, z7]] Sample dimensions: (# bounding boxes, 8, 3) The vertex order is of the following form: 4_____________________ 5 /| /| / | / | / | / | /___|_________________/ | 0| | | 1 | | | | | | | | | | | | | | |_________________|___| | / 7 | / 6 | / | / | / | / |/_____________________|/ 3 2 Copy to clipboard dtype: Defaults to float32. sample_compression or chunk_compression. Supported compressions: Copy to clipboard Noterotation angles are specified in degrees, not radians


Creating an intrinsics tensor

An intrinsics tensor can be created using Copy to clipboard Optional args: sample_compression or chunk_compression. dtype: Defaults to float32. Supported compressions: Copy to clipboard


Intrinsics Htype

Sample dimensions: (# intrinsics matrices, 3, 3) The intrinsic matrix represents a projective transformation from the 3-D camera's coordinates into the 2-D image coordinates. The intrinsic parameters include the focal length, the optical center, also known as the principal point. The camera intrinsic matrix, K, is defined as:[fx0cx0fycy001] [cx,cy] - Optical center (the principal point), in pixels. [fx,fy] - Focal length in pixels. fx=F/px fy=F/py F - Focal length in world units, typically expressed in millimeters. (px,py) - Size of the pixel in world units. Creating an intrinsics tensor An intrinsics tensor can be created using >>> ds.create_tensor('intrinsics', htype='intrinsics') Copy to clipboard Optional args: sample_compression or chunk_compression. dtype: Defaults to float32. Supported compressions: >>> ['lz4'] Copy to clipboard Appending intrinsics matrices >>> intrinsic_params = np.zeros((3, 3)) >>> ds.intrinsics.append(intrinsic_params) Copy to clipboard


Segmentation Mask Htype

Sample dimensions: (height, width) Segmentation masks are 2D representations of class labels where the numerical label data is encoded in an array of same shape as the image. The numerical values are indices of the list tensor.info.class_names. Creating a segment_mask tensor A segment_mask tensor can be created using >>> classes = ['background', 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle'] >>> ds.create_tensor('masks', htype='segment_mask', class_names=classes, sample_compression='lz4') Copy to clipboard Optional args: class_names: This must be a list of strings. tensor.info.class_names will be set to this list. sample_compression or chunk_compression dtype: Defaults to uint32. Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set the class names after tensor creation. >>> ds.labels.info.update(class_names = ['background', 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle']) Copy to clipboard Note Since segmentation masks often contain large amounts of data, it is recommended to compress them using lz4. Appending segmentation masks Segmentation masks can be appended as np.ndarray. Examples >>> ds.masks.append(np.zeros((512, 512))) Copy to clipboard Note Since each pixel can only be labeled once, segmentation masks are not appropriate for datasets where objects might overlap, or where multiple objects within the same class must be distinguished. For these use cases, please use htype = 'binary_mask'.


Creating a segment_mask tensor

A segment_mask tensor can be created using Copy to clipboard Optional args: class_names: This must be a list of strings. tensor.info.class_names will be set to this list. sample_compression or chunk_compression dtype: Defaults to uint32. Supported compressions: Copy to clipboard You can also choose to set the class names after tensor creation. Copy to clipboard NoteSince segmentation masks often contain large amounts of data, it is recommended to compress them using lz4.


Appending intrinsics matrices

Copy to clipboard


Appending segmentation masks

Segmentation masks can be appended as np.ndarray. Examples Copy to clipboard NoteSince each pixel can only be labeled once, segmentation masks are not appropriate for datasets where objects might overlap, or where multiple objects within the same class must be distinguished. For these use cases, please use htype = 'binary_mask'.


Binary Mask Htype

Sample dimensions: (height, width, # objects in a sample) Binary masks are similar to segmentation masks, except that each object is represented by a channel in the mask. Each channel in the mask encodes values for a single object. A pixel in a mask channel should have a value of 1 if the pixel of the image belongs to this object and 0 otherwise. The labels corresponding to the channels should be stored in an adjacent tensor of htype class_label, in which the number of labels at a given index is equal to the number of objects (number of channels) in the binary mask. Creating a binary_mask tensor A binary_mask tensor can be created using >>> ds.create_tensor('masks', htype='binary_mask', sample_compression='lz4') Copy to clipboard Optional args: ref:sample_compression <sample_compression> or chunk_compression dtype: Defaults to bool. Supported compressions: >>> ['lz4'] Copy to clipboard Note Since segmentation masks often contain large amounts of data, it is recommended to compress them using lz4. Appending binary masks Binary masks can be appended as np.ndarray. Examples Appending a binary mask with 5 objects >>> ds.masks.append(np.zeros((512, 512, 5), dtype='bool')) >>> ds.labels.append(['aeroplane', 'aeroplane', 'bottle', 'bottle', 'bird']) Copy to clipboard


Appending binary masks

Binary masks can be appended as np.ndarray. ExamplesAppending a binary mask with 5 objects Copy to clipboard


Creating a binary_mask tensor

A binary_mask tensor can be created using Copy to clipboard Optional args: ref:sample_compression <sample_compression> or chunk_compression dtype: Defaults to bool. Supported compressions: Copy to clipboard NoteSince segmentation masks often contain large amounts of data, it is recommended to compress them using lz4.


COCO Keypoints Htype

Sample dimensions: (3 x # keypoints, # objects in a sample) COCO keypoints are a convention for storing points of interest in an image. Each keypoint consists of 3 values: x - coordinate, y - coordinate and v - visibility. A set of K keypoints of an object is represented as:[x1, y1, v1, x2, y2, v2, …, xk, yk, vk]The visibility v can be one of three values: 0 keypoint not in image. 1 keypoint in image but not visible. 2 keypoint in image and visible. Creating a keypoints_coco tensor A keypoints_coco tensor can be created using >>> ds.create_tensor('keypoints', htype='keypoints_coco', keypoints=['knee', 'elbow', 'head'], connections=[[0, 1], [1, 2]]) Copy to clipboard Optional args: keypoints: List of strings describing the i th keypoint. tensor.info.keypoints will be set to this list. connections: List of strings describing which points should be connected by lines in the visualizer. sample_compression or chunk_compression dtype: Defaults to int32. Supported compressions: >>> ['lz4'] Copy to clipboard You can also choose to set keypoints and / or connections after tensor creation. >>> ds.keypoints.info.update(keypoints = ['knee', 'elbow',...]) >>> ds.keypoints.info.update(connections = [[0,1], [2,3], ...]) Copy to clipboard Appending keypoints Keypoints can be appended as np.ndarray or list. Examples Appending keypoints sample with 3 keypoints and 4 objects >>> ds.keypoints.update(keypoints = ['left ear', 'right ear', 'nose']) >>> ds.keypoints.update(connections = [[0, 2], [1, 2]]) >>> kp_arr array([[465, 398, 684, 469], [178, 363, 177, 177], [ 2, 2, 2, 1], [454, 387, 646, 478], [177, 322, 137, 161], [ 2, 2, 2, 2], [407, 379, 536, 492], [271, 335, 150, 143], [ 2, 1, 2, 2]]) >>> kp_arr.shape (9, 4) >>> ds.keypoints.append(kp_arr) Copy to clipboard Warning In order to correctly use the keypoints and connections metadata, it is critical that all objects in every sample have the same number of K keypoints in the same order. For keypoints that are not present in an image, they can be stored with dummy coordinates of x = 0, y = 0, and v = 0, and the visibility will prevent them from being drawn in the visualizer.


Appending 3d bounding boxes


Point Htype

Sample dimensions: (# points, 2) in case of 2-D (X, Y) co-ordinates or (# points, 3) in case of 3-D (X, Y, Z) co-ordinates of the point. Points does not contain a fixed mapping across samples between the point order and real-world objects (i.e., point 0 is an elbow, point 1 is a knee, etc.). If you require such a mapping, use COCO Keypoints Htype. Creating a point tensor A point tensor can be created using >>> ds.create_tensor('points', htype='point', sample_compression=None) Copy to clipboard Optional args: sample_compression or chunk_compression dtype: Defaults to int32. Supported compressions: >>> ['lz4'] Copy to clipboard Appending point samples Points can be appended as np.ndarray or list. Examples Appending 2 2-D points >>> ds.points.append([[0, 1], [1, 3]]) Copy to clipboard Appending 2 3-D points >>> ds.points.append(np.zeros((2, 3))) Copy to clipboard


Creating a point tensor

A point tensor can be created using Copy to clipboard Optional args: sample_compression or chunk_compression dtype: Defaults to int32. Supported compressions: Copy to clipboard


Appending point samples

Points can be appended as np.ndarray or list. ExamplesAppending 2 2-D points Copy to clipboard Appending 2 3-D points Copy to clipboard


Creating a keypoints_coco tensor

A keypoints_coco tensor can be created using Copy to clipboard Optional args: keypoints: List of strings describing the i th keypoint. tensor.info.keypoints will be set to this list. connections: List of strings describing which points should be connected by lines in the visualizer. sample_compression or chunk_compression dtype: Defaults to int32. Supported compressions: Copy to clipboard You can also choose to set keypoints and / or connections after tensor creation. Copy to clipboard


Polygon Htype

Sample dimensions: (# polygons, # points per polygon, # co-ordinates per point) Each sample in a tensor of polygon htype is a list of polygons. Each polygon is a list / array of points. All points in a sample should have the same number of co-ordinates (eg., cannot mix 2-D points with 3-D points). Different samples can have different number of polygons. Different polygons can have different number of points. Creating a polygon tensor A polygon tensor can be created using >>> ds.create_tensor('polygons', htype='polygon', sample_compression=None) Copy to clipboard Optional args: sample_compression or chunk_compression dtype: Defaults to float32. Supported compressions: >>> ['lz4'] Copy to clipboard Appending polygons Polygons can be appended as a list of list of tuples or np.ndarray. Examples Appending polygons with 2-D points >>> poly1 = [(1, 2), (2, 3), (3, 4)] >>> poly2 = [(10, 12), (14, 19)] >>> poly3 = [(33, 32), (54, 67), (67, 43), (56, 98)] >>> sample = [poly1, poly2, poly3] >>> ds.polygons.append(sample) Copy to clipboard Appending polygons with 3-D points >>> poly1 = [(10, 2, 9), (12, 3, 8), (12, 10, 4)] >>> poly2 = [(10, 1, 8), (5, 17, 11)] >>> poly3 = [(33, 33, 31), (45, 76, 13), (60, 24, 17), (67, 87, 83)] >>> sample = [poly1, poly2, poly3] >>> ds.polygons.append(sample) Copy to clipboard Appending polygons with numpy arrays >>> import numpy as np >>> sample = np.random.randint(0, 10, (5, 7, 2)) # 5 polygons with 7 points >>> ds.polygons.append(sample) Copy to clipboard >>> import numpy as np >>> poly1 = np.random.randint(0, 10, (5, 2)) >>> poly2 = np.random.randint(0, 10, (8, 2)) >>> poly3 = np.random.randint(0, 10, (3, 2)) >>> sample = [poly1, poly2, poly3] >>> ds.polygons.append(sample) Copy to clipboard


Appending polygons

Polygons can be appended as a list of list of tuples or np.ndarray. ExamplesAppending polygons with 2-D points Copy to clipboard Appending polygons with 3-D points Copy to clipboard Appending polygons with numpy arrays Copy to clipboard Copy to clipboard


Nifti Htype

Sample dimensions: (# height, # width, # slices) or (# height, # width, # slices, # time unit) in case of time-series data. Creating a nifti tensor A nifti tensor can be created using >>> ds.create_tensor('patients', htype='nifti', sample_compression='nii.gz') Copy to clipboard Supported compressions: >>> ['nii.gz', 'nii', None] Copy to clipboard Appending nifti data Nifti samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Deep Lake does not support compression of raw nifti data. Therefore, array of raw frames can only be appended to tensors with None compression. Examples >>> ds.patients.append(deeplake.read('data/patient0.nii.gz')) Copy to clipboard >>> ds.patients.extend([deeplake.read(f'data/patient{i}.nii.gz') for i in range(10)]) Copy to clipboard


Creating a nifti tensor

A nifti tensor can be created using Copy to clipboard Supported compressions: Copy to clipboard


Appending nifti data

Nifti samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Deep Lake does not support compression of raw nifti data. Therefore, array of raw frames can only be appended to tensors with None compression. Examples Copy to clipboard Copy to clipboard


Point Cloud Htype

Sample dimensions: (# num_points, 3) Point cloud samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Each point cloud is a list / array of points. All points in a sample should have the same number of co-ordinates. Different point clouds can have different number of points. Creating a point cloud tensor A point cloud tensor can be created using >>> ds.create_tensor('point_clouds', htype='point_cloud', sample_compression='las') Copy to clipboard Optional args: sample_compression Supported compressions: >>> [None, 'las'] Copy to clipboard Appending point clouds Point clouds can be appended as a np.ndarray. Examples Appending point clouds with numpy arrays >>> import numpy as np >>> point_cloud1 = np.random.randint(0, 10, (5, 3)) >>> ds.point_clouds.append(point_cloud1) >>> point_cloud2 = np.random.randint(0, 10, (15, 3)) >>> ds.point_clouds.append(point_cloud2) >>> ds.point_clouds.shape >>> (2, None, 3) Copy to clipboard Or we can use deeplake.read() method to add samples >>> import deeplake as dp >>> sample = dp.read('example.las') # point cloud with 100 points >>> ds.point_cloud.append(sample) >>> ds.point_cloud.shape >>> (1, 100, 3) Copy to clipboard


Creating a point cloud tensor

A point cloud tensor can be created using Copy to clipboard Optional args: sample_compression Supported compressions: Copy to clipboard


Appending point clouds

Point clouds can be appended as a np.ndarray. ExamplesAppending point clouds with numpy arrays Copy to clipboard Or we can use deeplake.read() method to add samples Copy to clipboard


Mesh Htype

Sample dimensions: (# num_points, 3) Mesh samples can be of type np.ndarray or Sample which is returned by deeplake.read(). Each sample in a tensor of mesh htype is a mesh array (3-D object data). Each mesh is a list / array of points. Different meshes can have different number of points. Creating a mesh tensor A mesh tensor can be created using >>> ds.create_tensor('mesh', htype='mesh', sample_compression='ply') Copy to clipboard Optional args: sample_compression Supported compressions: >>> ['ply'] Copy to clipboard Appending meshes Examples Appending a ply file containing a mesh data to tensor >>> import deeplake as dp >>> sample = dp.read('example.ply') # mesh with 100 points and 200 faces >>> ds.mesh.append(sample) Copy to clipboard >>> ds.mesh.shape >>> (1, 100, 3) Copy to clipboard


Creating a mesh tensor

A mesh tensor can be created using Copy to clipboard Optional args: sample_compression Supported compressions: Copy to clipboard


Creating a polygon tensor

A polygon tensor can be created using Copy to clipboard Optional args: sample_compression or chunk_compression dtype: Defaults to float32. Supported compressions: Copy to clipboard


Appending keypoints

Keypoints can be appended as np.ndarray or list. ExamplesAppending keypoints sample with 3 keypoints and 4 objects Copy to clipboard WarningIn order to correctly use the keypoints and connections metadata, it is critical that all objects in every sample have the same number of K keypoints in the same order. For keypoints that are not present in an image, they can be stored with dummy coordinates of x = 0, y = 0, and v = 0, and the visibility will prevent them from being drawn in the visualizer.


Embedding Htype

Sample dimensions: (# elements in the embedding,) Creating an embedding tensor An embedding tensor can be created using >>> ds.create_tensor('embedding', htype='embedding') Copy to clipboard Supported compressions: >>> ['lz4', None] Copy to clipboard Appending embedding samples Embedding samples can be of type np.ndarray. Examples Appending Deep Lake embedding sample >>> ds.embedding.append(np.random.uniform(low=-1, high=1, size=(1024))) Copy to clipboard Extending with Deep Lake embeddding samples >>> ds.embedding.extend([np.random.uniform(low=-1, high=1, size=(1024)) for i in range(10)]) Copy to clipboard


Creating an embedding tensor

An embedding tensor can be created using Copy to clipboard Supported compressions: Copy to clipboard


Appending embedding samples

Embedding samples can be of type np.ndarray. ExamplesAppending Deep Lake embedding sample Copy to clipboard Extending with Deep Lake embeddding samples Copy to clipboard


Sequence htype

A special meta htype for tensors where each sample is a sequence. The items in the sequence are samples of another htype. It is a wrapper htype that can wrap other htypes like sequence[image], sequence[video], sequence[text], etc. Examples Copy to clipboard Copy to clipboard


Link htype

Link htype is a special meta htype that allows linking of external data (files) to the dataset, without storing the data in the dataset itself. Moreover, there can be variations in this htype, such as link[image], link[video], link[audio], etc. that would enable the activeloop visualizer to correctly display the data. No data is actually loaded until you try to read the sample from a dataset. There are a few exceptions to this:- If create_shape_tensor=True was specified during create_tensor of the tensor to which this is being added, the shape of the sample is read. This is True by default. If create_sample_info_tensor=True was specified during create_tensor of the tensor to which this is being added, the sample info is read. This is True by default. If verify=True was specified during create_tensor of the tensor to which this is being added, some metadata is read from them to verify the integrity of the link samples. This is True by default. If you do not want to verify your links, all three of verify, create_shape_tensor and create_sample_info_tensor have to be set to False. Examples Copy to clipboard Adding credentials to the datasetYou can add the names of the credentials you want to use (not needed for http/local urls) Copy to clipboard and populate the added names with credentials dictionaries Copy to clipboard These creds are only present temporarily and will have to be repopulated on every reload.For datasets connected to Activeloop Platform, you can store your credentials on the platform as Managed Credentials and use them just by adding the keys to your dataset. For example if you have managed credentials with names 'my_s3_creds', 'my_gcs_creds', you can add them to your dataset using Dataset.add_creds_key without having to populate them. Copy to clipboard Create a link tensor Copy to clipboard Populate the tensor with links Copy to clipboard Copy to clipboard Updating a sample Copy to clipboard


Appending meshes

ExamplesAppending a ply file containing a mesh data to tensor Copy to clipboard Copy to clipboard


Compressions

Deep Lake can read, compress, decompress and recompress data to different formats. The supported htype-compression configurations are given below. Sample Type Htype Compressions Image image bmp, dib, gif, ico, jpeg, jpeg2000, pcx, png, ppm, sgi, tga, tiff, webp, wmf, xbm, eps, fli, im, msp, mpo, apng Video video mp4, mkv, avi Audio audio flac, mp3, wav Dicom dicom dcm Point Cloud point_cloud las Mesh mesh ply Other bbox, text, list, json, generic, etc. lz4 Sample Compression If sample compression is specified when creating tensors, samples will be compressed to the given format if possible. If given data is already compressed and matches the provided sample_compression, it will be stored as is. If left as None, given samples are uncompressed. Note For audio and video, we don't support compressing raw frames but only reading compressed audio and video data. Examples: >>> ds.create_tensor('images', htype='image', sample_compression='jpg') >>> ds.create_tensor('videos', htype='video', sample_compression='mp4') >>> ds.create_tensor('point_clouds', htype='point_cloud', sample_compression='las') Copy to clipboard Structure of sample-wise compressed tensor. Chunk Compression If chunk compression is specified when creating tensors, added samples will be clubbed together and compressed to the given format chunk-wise. If given data is already compressed, it will be uncompressed and then recompressed chunk-wise. Note Chunk-wise compression is not supported for audio, video and point_cloud htypes. Examples: >>> ds.create_tensor('images', htype='image', chunk_compression='jpg') >>> ds.create_tensor('boxes', htype='bbox', chunk_compression='lz4') Copy to clipboard Structure of chunk-wise compressed tensor. Note See deeplake.read() to learn how to read data from files and populate these tensors.


Sample Compression

If sample compression is specified when creating tensors, samples will be compressed to the given format if possible. If given data is already compressed and matches the provided sample_compression, it will be stored as is. If left as None, given samples are uncompressed.NoteFor audio and video, we don't support compressing raw frames but only reading compressed audio and video data.Examples: Copy to clipboard Structure of sample-wise compressed tensor.


Chunk Compression

If chunk compression is specified when creating tensors, added samples will be clubbed together and compressed to the given format chunk-wise. If given data is already compressed, it will be uncompressed and then recompressed chunk-wise.NoteChunk-wise compression is not supported for audio, video and point_cloud htypes.Examples: Copy to clipboard Structure of chunk-wise compressed tensor. NoteSee deeplake.read() to learn how to read data from files and populate these tensors.


PyTorch and Tensorflow Support

Deep Lake datasets can be easily converted to Torch dataloaders or Tensorflow datasets for training. Dataset.pytorch Converts the dataset into a pytorch Dataloader. Dataset.tensorflow Converts the dataset into a tensorflow compatible format.


Utility Functions

General Functions exists Checks if a dataset exists at the given path. Making Deep Lake Samples read Utility that reads raw data from supported files into Deep Lake format. link Utility that stores a link to raw data. link_tiled Utility that stores links to multiple images that act as tiles and together form a big image. Parallelism compute Compute is a decorator for functions. compose Takes a list of functions decorated using deeplake.compute() and creates a pipeline that can be evaluated using .eval Transform pipelines returned by compute() and compose() are evaluated using eval: eval Evaluates the pipeline on data_in to produce an output dataset ds_out.


Making Deep Lake Samples

read Utility that reads raw data from supported files into Deep Lake format. link Utility that stores a link to raw data. link_tiled Utility that stores links to multiple images that act as tiles and together form a big image.


Parallelism

compute Compute is a decorator for functions. compose Takes a list of functions decorated using deeplake.compute() and creates a pipeline that can be evaluated using .eval Transform pipelines returned by compute() and compose() are evaluated using eval: eval Evaluates the pipeline on data_in to produce an output dataset ds_out.


General Functions

exists Checks if a dataset exists at the given path.


Weights and Biases

Deep Lake's Weights and Biases integration allows you to track and improve reproducibility of your machine learning experiments. Deep Lake will automatically push all information required to reproduce the snapshot of the data like your dataset's URI, commit ID, and view IDs of any views that you have used in your training workflow.Learn more about Weights and Biases here. Logging Dataset Creation If you create a Deep Lake dataset using any of the functions mentioned in Creating Datasets, just perform a commit on the dataset to log its creation on W&B. >>> run = wandb.init(project='deeplake_wandb', job_type='dataset_upload') >>> ds = deeplake.empty('hub://fayazrahman4u/my_dataset') # create dataset >>> ds.create_tensor('images', htype='image', sample_compression='jpg') # create a tensor >>> ds.images.append(deeplake.read('files/images/dog.jpg')) # add a sample >>> ds.commit('creation') # commit -> trigger logging >>> run.finish() Copy to clipboard Note If you created your dataset using deeplake.deepcopy(), perform the commit only if you have head changes. Note If you make changes to an existing dataset, commit the changes with an active Weights and Biases run to log it's state. Logging Dataset Read A dataset read will be logged if you iterate over a dataset or call Dataset.pytorch() or Tensor.numpy() on its tensors. >>> run = wandb.init(project='deeplake_wandb', job_type='torch dataloader') >>> train_loader = ds.pytorch() >>> run.finish() Copy to clipboard >>> run = wandb.init(project='deeplake_wandb', job_type='iteration') >>> for sample in ds: >>> print(sample['images'].shape) >>> run.finish() Copy to clipboard


Logging Dataset Creation

If you create a Deep Lake dataset using any of the functions mentioned in Creating Datasets, just perform a commit on the dataset to log its creation on W&B. Copy to clipboard NoteIf you created your dataset using deeplake.deepcopy(), perform the commit only if you have head changes.NoteIf you make changes to an existing dataset, commit the changes with an active Weights and Biases run to log it's state.


Logging Dataset Read

A dataset read will be logged if you iterate over a dataset or call Dataset.pytorch() or Tensor.numpy() on its tensors. Copy to clipboard Copy to clipboard


Dataloader

Train your models using the new high performance C++ dataloader.  See the dataloader method on how to create dataloaders from your datasets: Dataset.dataloader Returns a DeepLakeDataLoader object.  DeepLakeDataLoader class deeplake.enterprise.DeepLakeDataLoader batch(batch_size: int, drop_last: bool = False) Returns a batched DeepLakeDataLoader object.  Parameters batch_size (int) – Number of samples in each batch.  drop_last (bool) – If True, the last batch will be dropped if its size is less than batch_size.  Defaults to False.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .batch() has already been called.  close() Shuts down the workers and releases the resources.  numpy(num_workers: int = 0, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, decode_method: Optional[Dict[str, str]] = None, persistent_workers: bool = False) Returns a DeepLakeDataLoader object.  Parameters num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  persistent_workers (bool) – If True, the data loader will not shutdown the worker processes after a dataset has been consumed once.  Defaults to False.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are:- ‘numpy' Default behaviour.  Returns samples as numpy arrays.  'tobytes' Returns raw bytes of the samples.  'pil' Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or ‘png'.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .pytorch() or .tensorflow() or .numpy() has already been called.  offset(off: int = 0) Returns a shifted DeepLakeDataLoader object.  Parameters off (int) – index that the dataloadee will start to iterate.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .offset() has already been called.  pytorch(num_workers: int = 0, collate_fn: Optional[Callable] = None, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, distributed: bool = False, return_index: bool = True, decode_method: Optional[Dict[str, str]] = None, persistent_workers: bool = False) Returns a DeepLakeDataLoader object.  Parameters num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  collate_fn (Callable, Optional) – merges a list of samples to form a mini-batch of Tensor(s).  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data


Dataloader

If None, the number of threads is automatically determined .  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  distributed (bool) – Used for DDP training.  Distributes different sections of the dataset to different ranks.  Defaults to False.  return_index (bool) – Used to idnetify where loader needs to retur sample index or not.  Defaults to True.  persistent_workers (bool) – If True, the data loader will not shutdown the worker processes after a dataset has been consumed once.  Defaults to False.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are: 'numpy' Default behaviour.  Returns samples as numpy arrays.  'tobytes' Returns raw bytes of the samples.  'pil' Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or 'png'.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .pytorch() or .tensorflow() or .numpy() has already been called.  Examples >>> import deeplake >>> from torchvision import transforms >>> ds_train = deeplake.load('hub://activeloop/fashion-mnist-train') >>> tform = transforms.Compose([ ...  transforms.RandomRotation(20), # Image augmentation ...  transforms.ToTensor(), # Must convert to pytorch tensor for subsequent operations to run ...  transforms.Normalize([0.5], [0.5]), ...  ]) ...  >>> batch_size = 32 >>> # create dataloader by chaining with transform function and batch size and returns batch of pytorch tensors >>> train_loader = ds_train.dataloader()\ ...  .transform({'images': tform, 'labels': None})\ ...  .batch(batch_size)\ ...  .shuffle()\ ...  .pytorch(decode_method={'images': 'pil'}) # return samples as PIL images for transforms ...  >>> # iterate over dataloader >>> for i, sample in enumerate(train_loader): ...  pass ...  Copy to clipboard query(query_string: str) Returns a sliced DeepLakeDataLoader object with given query results.  It allows to run SQL like queries on dataset and extract results.  See supported keywords and the Tensor Query Language documentation here.  Parameters query_string (str) – An SQL string adjusted with new functionalities to run on the dataset object Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Examples >>> import deeplake >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> query_ds_train = ds_train.dataloader().query('select * where labels != 5') Copy to clipboard >>> import deeplake >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> query_ds_train = ds_train.dataloader().query('(select * where contains(categories, 'car') limit 1000) union (select * where contains(categories, 'motorcycle') limit 1000)') Copy to clipboard sample_by(weights: Union[str, list, tuple, ndarray], replace: Optional[bool] = True, size: Optional[int] = None) Returns a sliced DeepLakeDataLoader with given weighted sampler applied Parameters weights – (Union[str, list, tuple, np.ndarray]): If it's string then tql will be run to calculate the weights based on the expression.  list, tuple and ndarray will be treated as the list of the weights per sample replace – Optional[bool] If true the samples can be repeated in the result view.  (default: True)


Dataloader

size – Optional[int] The length of the result view .  (default: len(dataset)) Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Examples Sample the dataloader with labels == 5 twice more than labels == 6 >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('max_weight(labels == 5: 10, labels == 6: 5)') Copy to clipboard Sample the dataloader treating labels tensor as weights.  >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('labels') Copy to clipboard Sample the dataloader with the given weights; >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> weights = list() >>> for i in range(0, len(ds_train)): ...  weights.append(i % 5) ...  >>> sampled_ds = ds.dataloader().sample_by(weights, replace=False) Copy to clipboard shuffle(shuffle: bool = True, buffer_size: int = 2048) Returns a shuffled DeepLakeDataLoader object.  Parameters shuffle (bool) – shows wheter we need to shuffle elements or not.  Defaults to True.  buffer_size (int) – The size of the buffer used to shuffle the data in MBs.  Defaults to 2048 MB.  Increasing the buffer_size will increase the extent of shuffling.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .shuffle() has already been called.  ValueError – If dataset is view and shuffle is True tensorflow(num_workers: int = 0, collate_fn: Optional[Callable] = None, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, return_index: bool = True, decode_method: Optional[Dict[str, str]] = None, persistent_workers: bool = False) Returns a DeepLakeDataLoader object.  Parameters num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  collate_fn (Callable, Optional) – merges a list of samples to form a mini-batch of Tensor(s).  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  return_index (bool) – Used to idnetify where loader needs to retur sample index or not.  Defaults to True.  persistent_workers (bool) – If True, the data loader will not shutdown the worker processes after a dataset has been consumed once.  Defaults to False.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are: 'numpy' Default behaviour.  Returns samples as numpy arrays.  'tobytes' Returns raw bytes of the samples.  'pil' Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or 'png'.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .pytorch() or .tensorflow() or .numpy() has already been called


Dataloader

Examples >>> import deeplake >>> from torchvision import transforms >>> ds_train = deeplake.load('hub://activeloop/fashion-mnist-train') >>> batch_size = 32 >>> # create dataloader by chaining with transform function and batch size and returns batch of pytorch tensors >>> train_loader = ds_train.dataloader()\ .. .  .batch(batch_size)\ ...  .shuffle()\ ...  .tensorflow() # return samples as PIL images for transforms ...  >>> # iterate over dataloader >>> for i, sample in enumerate(train_loader): ...  pass ...  Copy to clipboard transform(transform: Union[Callable, Dict[str, Optional[Callable]]], **kwargs: Dict) Returns a transformed DeepLakeDataLoader object.  Parameters transform (Callable or Dict[Callable]) – A function or dictionary of functions to apply to the data.  kwargs – Additional arguments to be passed to transform.  Only applicable if transform is a callable.  Ignored if transform is a dictionary.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .transform() has already been called.


DeepLakeDataLoader

class deeplake.enterprise.DeepLakeDataLoader batch(batch_size: int, drop_last: bool = False) Returns a batched DeepLakeDataLoader object.  Parameters batch_size (int) – Number of samples in each batch.  drop_last (bool) – If True, the last batch will be dropped if its size is less than batch_size.  Defaults to False.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .batch() has already been called.  close() Shuts down the workers and releases the resources.  numpy(num_workers: int = 0, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, decode_method: Optional[Dict[str, str]] = None, persistent_workers: bool = False) Returns a DeepLakeDataLoader object.  Parameters num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  persistent_workers (bool) – If True, the data loader will not shutdown the worker processes after a dataset has been consumed once.  Defaults to False.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are:- ‘numpy' Default behaviour.  Returns samples as numpy arrays.  'tobytes' Returns raw bytes of the samples.  'pil' Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or ‘png'.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .pytorch() or .tensorflow() or .numpy() has already been called.  offset(off: int = 0) Returns a shifted DeepLakeDataLoader object.  Parameters off (int) – index that the dataloadee will start to iterate.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .offset() has already been called.  pytorch(num_workers: int = 0, collate_fn: Optional[Callable] = None, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, distributed: bool = False, return_index: bool = True, decode_method: Optional[Dict[str, str]] = None, persistent_workers: bool = False) Returns a DeepLakeDataLoader object.  Parameters num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  collate_fn (Callable, Optional) – merges a list of samples to form a mini-batch of Tensor(s).  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker


DeepLakeDataLoader

Defaults to 2 .  distributed (bool) – Used for DDP training.  Distributes different sections of the dataset to different ranks.  Defaults to False.  return_index (bool) – Used to idnetify where loader needs to retur sample index or not.  Defaults to True.  persistent_workers (bool) – If True, the data loader will not shutdown the worker processes after a dataset has been consumed once.  Defaults to False.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are: 'numpy' Default behaviour.  Returns samples as numpy arrays.  'tobytes' Returns raw bytes of the samples.  'pil' Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or 'png'.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .pytorch() or .tensorflow() or .numpy() has already been called.  Examples >>> import deeplake >>> from torchvision import transforms >>> ds_train = deeplake.load('hub://activeloop/fashion-mnist-train') >>> tform = transforms.Compose([ ...  transforms.RandomRotation(20), # Image augmentation ...  transforms.ToTensor(), # Must convert to pytorch tensor for subsequent operations to run ...  transforms.Normalize([0.5], [0.5]), ...  ]) ...  >>> batch_size = 32 >>> # create dataloader by chaining with transform function and batch size and returns batch of pytorch tensors >>> train_loader = ds_train.dataloader()\ ...  .transform({'images': tform, 'labels': None})\ ...  .batch(batch_size)\ ...  .shuffle()\ ...  .pytorch(decode_method={'images': 'pil'}) # return samples as PIL images for transforms ...  >>> # iterate over dataloader >>> for i, sample in enumerate(train_loader): ...  pass ...  Copy to clipboard query(query_string: str) Returns a sliced DeepLakeDataLoader object with given query results.  It allows to run SQL like queries on dataset and extract results.  See supported keywords and the Tensor Query Language documentation here.  Parameters query_string (str) – An SQL string adjusted with new functionalities to run on the dataset object Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Examples >>> import deeplake >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> query_ds_train = ds_train.dataloader().query('select * where labels != 5') Copy to clipboard >>> import deeplake >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> query_ds_train = ds_train.dataloader().query('(select * where contains(categories, 'car') limit 1000) union (select * where contains(categories, 'motorcycle') limit 1000)') Copy to clipboard sample_by(weights: Union[str, list, tuple, ndarray], replace: Optional[bool] = True, size: Optional[int] = None) Returns a sliced DeepLakeDataLoader with given weighted sampler applied Parameters weights – (Union[str, list, tuple, np.ndarray]): If it's string then tql will be run to calculate the weights based on the expression.  list, tuple and ndarray will be treated as the list of the weights per sample replace – Optional[bool] If true the samples can be repeated in the result view.  (default: True).  size – Optional[int] The length of the result view.  (default: len(dataset)) Returns A DeepLakeDataLoader object


DeepLakeDataLoader


DeepLakeDataLoader

Return type DeepLakeDataLoader Examples Sample the dataloader with labels == 5 twice more than labels == 6 >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('max_weight(labels == 5: 10, labels == 6: 5)') Copy to clipboard Sample the dataloader treating labels tensor as weights .  >>> ds = deeplake.load('hub://activeloop/fashion-mnist-train') >>> sampled_ds = ds.dataloader().sample_by('labels') Copy to clipboard Sample the dataloader with the given weights; >>> ds_train = deeplake.load('hub://activeloop/coco-train') >>> weights = list() >>> for i in range(0, len(ds_train)): ...  weights.append(i % 5) ...  >>> sampled_ds = ds.dataloader().sample_by(weights, replace=False) Copy to clipboard shuffle(shuffle: bool = True, buffer_size: int = 2048) Returns a shuffled DeepLakeDataLoader object.  Parameters shuffle (bool) – shows wheter we need to shuffle elements or not.  Defaults to True.  buffer_size (int) – The size of the buffer used to shuffle the data in MBs.  Defaults to 2048 MB.  Increasing the buffer_size will increase the extent of shuffling.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .shuffle() has already been called.  ValueError – If dataset is view and shuffle is True tensorflow(num_workers: int = 0, collate_fn: Optional[Callable] = None, tensors: Optional[List[str]] = None, num_threads: Optional[int] = None, prefetch_factor: int = 2, return_index: bool = True, decode_method: Optional[Dict[str, str]] = None, persistent_workers: bool = False) Returns a DeepLakeDataLoader object.  Parameters num_workers (int) – Number of workers to use for transforming and processing the data.  Defaults to 0.  collate_fn (Callable, Optional) – merges a list of samples to form a mini-batch of Tensor(s).  tensors (List[str], Optional) – List of tensors to load.  If None, all tensors are loaded.  Defaults to None.  num_threads (int, Optional) – Number of threads to use for fetching and decompressing the data.  If None, the number of threads is automatically determined.  Defaults to None.  prefetch_factor (int) – Number of batches to transform and collate in advance per worker.  Defaults to 2.  return_index (bool) – Used to idnetify where loader needs to retur sample index or not.  Defaults to True.  persistent_workers (bool) – If True, the data loader will not shutdown the worker processes after a dataset has been consumed once.  Defaults to False.  decode_method (Dict[str, str], Optional) – A dictionary of decode methods for each tensor.  Defaults to None.  Supported decode methods are: 'numpy' Default behaviour.  Returns samples as numpy arrays.  'tobytes' Returns raw bytes of the samples.  'pil' Returns samples as PIL images.  Especially useful when transformation use torchvision transforms, that require PIL images as input.  Only supported for tensors with sample_compression='jpeg' or 'png'.  Returns A DeepLakeDataLoader object.  Return type DeepLakeDataLoader Raises ValueError – If .pytorch() or .tensorflow() or .numpy() has already been called


Sampler

The sampler applies weighted sampling on the dataset and returns the sampled view. It creates a discrete distribution with given weights and randomly picks samples based on it. The resulting view is generated in such a way that when creating a dataloader from the view and training on it, the performance impact is minimal. See the sample_by method on how to use this feature: Dataset.sample_by Returns a sliced Dataset with given weighted sampler applied.


Tensor Query Language

For example, labels == 0 OR labels == 1 Copy to clipboard From SQL we also support the following two keywords: BETWEEN labels BETWEEN 0 and 5 Copy to clipboard IN labels in ARRAY[0, 2, 4, 6, 8] Copy to clipboard Functions There are predefined functions which can be used in WHERE expression as well as in ORDER BY expressions: CONTAINS - checks if the given tensor contains given value - CONTAINS(categories, 'person') RANDOM - returns random number .  May be used in ORDER BY to shuffle the output - ORDER BY RANDOM() SHAPE - returns the shape array of the given tensor - SHAPE(boxes) ALL - takes an array of booleans and returns single boolean, True if all elements of the input array are True ALL_STRICT - same as ALL with one difference.  ALL returns True on empty array, while ALL_STRICT return False ANY - takes an array of booleans and returns single boolean, True if any of the elements int the input array is True LOGICAL_AND - takes two boolean arrays, does element wise logical and, returns the result array.  This will return False if the input arrays have different sizes.  LOGICAL_OR - takes two boolean arrays, does element wise logical or, returns the result array.  This will return False if the input arrays have different sizes.  UNION, INTERSECT, EXCEPT Query can contain multiple SELECT statements, combined by one of the set operations - UNION, INTERSECT and EXCEPT.  Examples Querying for images containing 0 in MNIST Train Dataset with ds.query.  >>> import deeplake >>> ds = deeplake.load('hub://activeloop/mnist-train') >>> result = ds.query('select * where labels == 0') >>> len(result) 5923 Copy to clipboard Querying for samples with car or motorcycle in categories of COCO Train Dataset.  >>> import deeplake >>> ds = deeplake.load('hub://activeloop/coco-train') >>> result = ds.query('(select * where contains(categories, 'car')) union (select * where contains(categories, 'motorcycle'))') >>> len(result) 14376 Copy to clipboard


Tensor Query Language

This page describes the Tensor Query Language (TQL), an SQL-like language used for Querying in Activeloop Platform as well as in ds.query in our Python API.  To use queries, install deeplake with pip install deeplake[enterprise].  Syntax SELECT TQL supports only SELECT statement.  Every TQL expression starts with SELECT *.  TQL supports only * which means to select all tensors.  The common syntax for select statement is the following: SELECT * [FROM string] [WHERE expression] [LIMIT number [OFFSET number]] [ORDER BY expression [ASC/DESC]] Copy to clipboard Each part of the SELECT statement can be omitted.  FROM expression is allowed, but it does not have any effect on the query, because for now TQL queries are run on a specific dataset, so the FROM is known from the context WHERE WHERE expression is used to filter the samples in the dataset by conditions.  The conditions should be convertible to boolean.  Any expression which outputs a number will be converted to boolean with non-zero values taken as True.  If the expression is not convertible to boolean, such as strings, json objects and arrays, the query will print the corresponding error.  ORDER BY ORDER BY expression orders the output of the query by the given criteria.  The criteria can be any expression output of which can be ordered.  The ordered outputs are either scalar numbers or strings.  In addition it can also be json, which contains number or string.  ORDER BY statement optionally accepts ASC/DESC keywords specifying whether the ordering should be ascending or descending.  It is ascending by default.  LIMIT OFFSET LIMIT and OFFSET expressions are used to limit the output of the query by index, as in SQL.  Expressions TQL supports any comparison operator (==, !=, <, <=, >=) where the left side is a tensor and the right side is a known value.  The value can be numeric scalar or array as well as string value.  String literal should be provided within single quotes (') and can be used on class_label, json and text tensors.  For class labels it will get corresponding numeric value from the class_names list and do numeric comparison.  For json and text it will do string comparison.  The left side of the expression can be indexed (subscripted) if the tensor is multidimensional array or json.  Jsons support indexing by string, e.g.  index_meta['id'] == 'some_id'.  Jsons can also be indexed by number if the underlying data is array.  Numeric multidimensional tensors can be indexed by numbers, e.g.  categories[0] == 1 as well as Python style slicing and multidimensional indexing, such as boxes[:2].  This last expression returns array containing the third elements of the initial two dimensional array boxes.  TQL supports logical operators - AND, OR and NOT.  These operators can be used to combine boolean expressions


Syntax

SELECT TQL supports only SELECT statement.  Every TQL expression starts with SELECT *.  TQL supports only * which means to select all tensors.  The common syntax for select statement is the following: SELECT * [FROM string] [WHERE expression] [LIMIT number [OFFSET number]] [ORDER BY expression [ASC/DESC]] Copy to clipboard Each part of the SELECT statement can be omitted.  FROM expression is allowed, but it does not have any effect on the query, because for now TQL queries are run on a specific dataset, so the FROM is known from the context WHERE WHERE expression is used to filter the samples in the dataset by conditions.  The conditions should be convertible to boolean.  Any expression which outputs a number will be converted to boolean with non-zero values taken as True.  If the expression is not convertible to boolean, such as strings, json objects and arrays, the query will print the corresponding error.  ORDER BY ORDER BY expression orders the output of the query by the given criteria.  The criteria can be any expression output of which can be ordered.  The ordered outputs are either scalar numbers or strings.  In addition it can also be json, which contains number or string.  ORDER BY statement optionally accepts ASC/DESC keywords specifying whether the ordering should be ascending or descending.  It is ascending by default.  LIMIT OFFSET LIMIT and OFFSET expressions are used to limit the output of the query by index, as in SQL.  Expressions TQL supports any comparison operator (==, !=, <, <=, >=) where the left side is a tensor and the right side is a known value.  The value can be numeric scalar or array as well as string value.  String literal should be provided within single quotes (') and can be used on class_label, json and text tensors.  For class labels it will get corresponding numeric value from the class_names list and do numeric comparison.  For json and text it will do string comparison.  The left side of the expression can be indexed (subscripted) if the tensor is multidimensional array or json.  Jsons support indexing by string, e.g.  index_meta['id'] == 'some_id'.  Jsons can also be indexed by number if the underlying data is array.  Numeric multidimensional tensors can be indexed by numbers, e.g.  categories[0] == 1 as well as Python style slicing and multidimensional indexing, such as boxes[:2].  This last expression returns array containing the third elements of the initial two dimensional array boxes.  TQL supports logical operators - AND, OR and NOT.  These operators can be used to combine boolean expressions


SELECT

TQL supports only SELECT statement. Every TQL expression starts with SELECT *. TQL supports only * which means to select all tensors. The common syntax for select statement is the following: Copy to clipboard Each part of the SELECT statement can be omitted.FROM expression is allowed, but it does not have any effect on the query, because for now TQL queries are run on a specific dataset, so the FROM is known from the context


Syntax

For example, labels == 0 OR labels == 1 Copy to clipboard From SQL we also support the following two keywords: BETWEEN labels BETWEEN 0 and 5 Copy to clipboard IN labels in ARRAY[0, 2, 4, 6, 8] Copy to clipboard Functions There are predefined functions which can be used in WHERE expression as well as in ORDER BY expressions: CONTAINS - checks if the given tensor contains given value - CONTAINS(categories, 'person') RANDOM - returns random number .  May be used in ORDER BY to shuffle the output - ORDER BY RANDOM() SHAPE - returns the shape array of the given tensor - SHAPE(boxes) ALL - takes an array of booleans and returns single boolean, True if all elements of the input array are True ALL_STRICT - same as ALL with one difference.  ALL returns True on empty array, while ALL_STRICT return False ANY - takes an array of booleans and returns single boolean, True if any of the elements int the input array is True LOGICAL_AND - takes two boolean arrays, does element wise logical and, returns the result array.  This will return False if the input arrays have different sizes.  LOGICAL_OR - takes two boolean arrays, does element wise logical or, returns the result array.  This will return False if the input arrays have different sizes.  UNION, INTERSECT, EXCEPT Query can contain multiple SELECT statements, combined by one of the set operations - UNION, INTERSECT and EXCEPT.


WHERE

WHERE expression is used to filter the samples in the dataset by conditions. The conditions should be convertible to boolean. Any expression which outputs a number will be converted to boolean with non-zero values taken as True. If the expression is not convertible to boolean, such as strings, json objects and arrays, the query will print the corresponding error.


Expressions

TQL supports any comparison operator (==, !=, <, <=, >=) where the left side is a tensor and the right side is a known value.The value can be numeric scalar or array as well as string value.String literal should be provided within single quotes (') and can be used on class_label, json and text tensors.For class labels it will get corresponding numeric value from the class_names list and do numeric comparison.For json and text it will do string comparison. The left side of the expression can be indexed (subscripted) if the tensor is multidimensional array or json. Jsons support indexing by string, e.g. index_meta['id'] == 'some_id'. Jsons can also be indexed by number if the underlying data is array.Numeric multidimensional tensors can be indexed by numbers, e.g. categories[0] == 1 as well as Python style slicing and multidimensional indexing, such as boxes[:2]. This last expression returns array containing the third elements of the initial two dimensional array boxes.TQL supports logical operators - AND, OR and NOT. These operators can be used to combine boolean expressions. For example, Copy to clipboard From SQL we also support the following two keywords: BETWEEN Copy to clipboard IN Copy to clipboard


Functions

There are predefined functions which can be used in WHERE expression as well as in ORDER BY expressions: CONTAINS - checks if the given tensor contains given value - CONTAINS(categories, 'person') RANDOM - returns random number. May be used in ORDER BY to shuffle the output - ORDER BY RANDOM() SHAPE - returns the shape array of the given tensor - SHAPE(boxes) ALL - takes an array of booleans and returns single boolean, True if all elements of the input array are True ALL_STRICT - same as ALL with one difference. ALL returns True on empty array, while ALL_STRICT return False ANY - takes an array of booleans and returns single boolean, True if any of the elements int the input array is True LOGICAL_AND - takes two boolean arrays, does element wise logical and, returns the result array. This will return False if the input arrays have different sizes. LOGICAL_OR - takes two boolean arrays, does element wise logical or, returns the result array. This will return False if the input arrays have different sizes.


ORDER BY

ORDER BY expression orders the output of the query by the given criteria. The criteria can be any expression output of which can be ordered. The ordered outputs are either scalar numbers or strings. In addition it can also be json, which contains number or string.ORDER BY statement optionally accepts ASC/DESC keywords specifying whether the ordering should be ascending or descending. It is ascending by default.


LIMIT OFFSET

LIMIT and OFFSET expressions are used to limit the output of the query by index, as in SQL.


Examples

Querying for images containing 0 in MNIST Train Dataset with ds.query. Copy to clipboard Querying for samples with car or motorcycle in categories of COCO Train Dataset. Copy to clipboard


UNION, INTERSECT, EXCEPT

Query can contain multiple SELECT statements, combined by one of the set operations - UNION, INTERSECT and EXCEPT.


Random Split

Splits the dataset into non overlapping new datasets of given lengths. The resulting datasets are generated in such a way that when creating a dataloader from the view and training on it, the performance impact is minimal. Using the outputs of this function with .pytorch method of dataset (instead of .dataloader) may result in poor performance. See the random_split method on how to use this feature: Dataset.random_split Splits the dataset into non-overlapping Dataset objects of given lengths.


Deep Memory API

class deeplake.core.vectorstore.deep_memory.DeepMemory __init__(dataset: Dataset, client: DeepMemoryBackendClient, logger: Logger, embedding_function: Optional[Any] = None, token: Optional[str] = None, creds: Optional[Dict[str, Any]] = None) Based Deep Memory class to train and evaluate models on DeepMemory managed service.  Parameters dataset (Dataset) – deeplake dataset object.  client (DeepMemoryBackendClient) – Client to interact with the DeepMemory managed service.  Defaults to None.  logger (logging.Logger) – Logger object.  embedding_function (Optional[Any], optional) – Embedding funtion class used to convert queries/documents to embeddings.  Defaults to None.  token (Optional[str], optional) – API token for the DeepMemory managed service.  Defaults to None.  creds (Optional[Dict[str, Any]], optional) – Credentials to access the dataset.  Defaults to None.  Raises ImportError – if indra is not installed cancel(job_id: str) Cancel a training job on DeepMemory managed service.  Examples >>> cancelled: bool = vectorstore.deep_memory.cancel(job_id) Copy to clipboard Parameters job_id (str) – job_id of the training job.  Returns True if job was cancelled successfully, False otherwise.  Return type bool delete(job_id: str) Delete a training job on DeepMemory managed service.  Examples >>> deleted: bool = vectorstore.deep_memory.delete(job_id) Copy to clipboard Parameters job_id (str) – job_id of the training job.  Returns True if job was deleted successfully, False otherwise.  Return type bool evaluate(relevance: List[List[Tuple[str, int]]], queries: List[str], embedding_function: Optional[Callable[[...], List[ndarray]]] = None, embedding: Optional[Union[List[ndarray], List[List[float]]]] = None, top_k: List[int] = [1, 3, 5, 10, 50, 100], qvs_params: Optional[Dict[str, Any]] = None) → Dict[str, Dict[str, float]] Evaluate a model using the DeepMemory managed service.  Examples # 1.  Evaluate a model using an embedding function: relevance = [[('doc_id_1', 1), ('doc_id_2', 1)], [('doc_id_3', 1)]] queries = ['What is the capital of India?', 'What is the capital of France?'] embedding_function = openai_embedding.embed_documents vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding_function=embedding_function, ) # 2.  Evaluate a model with precomputed embeddings: embeddings = [[-1.2, 12, …], …] vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, ) # 3.  Evaluate a model with precomputed embeddings and log queries: vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, qvs_params={'log_queries': True}, ) # 4.  Evaluate with precomputed embeddings, log queries, and a custom branch: vectorstore.deep_memory.evaluate( relevance=relevance, queries=queries, embedding=embeddings, qvs_params={ 'log_queries': True, 'branch': 'queries', } ) Parameters queries (List[str]) – Queries for model evaluation.  relevance (List[List[Tuple[str, int]]]) – Relevant documents and scores for each query.  - Outer list: matches the queries.  - Inner list: pairs of doc_id and relevance score.  - doc_id: Document ID from the corpus dataset, found in the id tensor.  - relevance_score: Between 0 (not relevant) and 1 (relevant).  embedding (Optional[np.ndarray], optional) – Query embeddings.  Defaults to None.  embedding_function (Optional[Callable[..., List[np.ndarray]]], optional) – Function to convert queries into embeddings.  Defaults to None.  top_k (List[int], optional) – Ranks for model evaluation.  Defaults to [1, 3, 5, 10, 50, 100].  qvs_params (Optional[Dict], optional) – Parameters to initialize the queries vectorstore.  When specified, creates a new vectorstore to track evaluation queries, the Deep Memory response, and the naive vector search results.  Defaults to None


Deep Memory API

Returns Recalls for each rank .  Return type Dict[str, Dict[str, float]] Raises ImportError – If indra is not installed.  ValueError – If no embedding_function is provided either during initialization or evaluation.  list_jobs(debug=False) List all training jobs on DeepMemory managed service.  status(job_id: str) Get the status of a training job on DeepMemory managed service.  Examples >>> vectorstore.deep_memory.status(job_id) -------------------------------------------------------------- | 6508464cd80cab681bfcfff3 | -------------------------------------------------------------- | status | pending | -------------------------------------------------------------- | progress | None | -------------------------------------------------------------- | results | not available yet | -------------------------------------------------------------- Copy to clipboard Parameters job_id (str) – job_id of the training job.  train(queries: List[str], relevance: List[List[Tuple[str, int]]], embedding_function: Optional[Callable[[str], ndarray]] = None, token: Optional[str] = None) → str Train a model on DeepMemory managed service.  Examples >>> queries: List[str] = ['What is the capital of India?', 'What is the capital of France?'] >>> relevance: List[List[Tuple[str, int]]] = [[('doc_id_1', 1), ('doc_id_2', 1)], [('doc_id_3', 1)]] >>> # doc_id_1, doc_id_2, doc_id_3 are the ids of the documents in the corpus dataset that is relevant to the queries.  It is stored in the `id` tensor of the corpus dataset.  >>> job_id: str = vectorstore.deep_memory.train(queries, relevance) Copy to clipboard Parameters queries (List[str]) – List of queries to train the model on.  relevance (List[List[Tuple[str, int]]]) – List of relevant documents for each query with their respective relevance score.  The outer list corresponds to the queries and the inner list corresponds to the doc_id, relevence_score pair for each query.  doc_id is the document id in the corpus dataset.  It is stored in the id tensor of the corpus dataset.  relevence_score is the relevance score of the document for the query.  The range is between 0 and 1, where 0 stands for not relevant and 1 stands for relevant.  embedding_function (Optional[Callable[[str], np.ndarray]], optional) – Embedding funtion used to convert queries to embeddings.  Defaults to None.  token (str, optional) – API token for the DeepMemory managed service.  Defaults to None.  Returns job_id of the training job.  Return type str Raises ValueError – if embedding_function is not specified either during initialization or during training.  Syntax This page describes ds.query.  DeepMemory is a deep learning model that is trained on the dataset to improve the search results, by aligning queries with the corpus dataset.  It gives up to +22% of recall improvement on an eval dataset.  To use deep_memory, please subscribe to our waitlist.  Training To start training you should first create a vectostore object, and then preprocess the data and use deep memory with it: >>> from deeplake import VectorStore >>> db = VectorStore( ...  path='hub://{$ORG_ID}/{$DATASET_ID}', ...  token=token, # or you can be signed in with CLI ...  runtime={'tensor_db': True}, ...  embedding_function=embedding_function, # function that takes converts texts into embeddings, it is optional and can be provided later ..


Deep Memory API

) Copy to clipboard To train a deepmemory model you need to preprocess the dataset so that, corpus, will become a list of list of tuples, where outer list corresponds to the query and inner list to the relevant documents .  Each tuple should contain the document id (id tensor from the corpus dataset) and the relevance score (range is 0-1, where 0 represents unrelated document and 1 related).  queries should be a list of strings.  >>> job_id = db.deep_memory.train( ...  corpus: List[List[Tuple[str, float]]] = corpus, ...  queries: List[str] = queries, ...  embedding_function = embedding_function, # function that takes converts texts into embeddings, it is optional and can be skipped if provided during initialization ...  ) Copy to clipboard Tracking the training progress job_id is string, which can be used to track the training progress.  You can use db.deep_memory.status(job_id) to get the status of the job.  when the model is still in pending state (not started yet) you will see the following: >>> db.deep_memory.status(job_id) ————————————————————– | 6508464cd80cab681bfcfff3 | ————————————————————– | status | pending | ————————————————————– | progress | None | ————————————————————– | results | not available yet | ————————————————————– After some time the model will start training and you will see the following: >>> db.deep_memory.status(job_id) -------------------------------------------------------------- | 6508464cd80cab681bfcfff3 | -------------------------------------------------------------- | status | training | -------------------------------------------------------------- | progress | eta: 2.5 seconds | | | recall@10: 0.62% (+0.62%) | -------------------------------------------------------------- | results | not available yet | -------------------------------------------------------------- Copy to clipboard If you want to get all training jobs you can use db.deep_memory.list_jobs() which will show all jobs that happened on this dataset.  >>> db.deep_memory.list_jobs() ID STATUS RESULTS PROGRESS 65198efcd28df3238c49a849 completed recall@10: 0.62% (+0.62%) eta: 2.5 seconds recall@10: 0.62% (+0.62%) 651a4d41d05a21a5a6a15f67 completed recall@10: 0.62% (+0.62%) eta: 2.5 seconds recall@10: 0.62% (+0.62%) Copy to clipboard Deep Memory Evaluation Once the training is completed, you can use db.deep_memory.evaluate to evaluate the model performance on the custom dataset.  Once again you would need to preprocess the dataset so that, corpus, will become a list of list of tuples, where outer list corresponds to the query and inner list to the relevant documents.  Each tuple should contain the document id (id tensor from the corpus dataset) and the relevance score (range is 0-1, where 0 represents unrelated document and 1 related).  queries should be a list of strings.  >>> recalls = db.deep_memory.evaluate( ...  corpus: List[List[Tuple[str, float]]] = corpus, ...  queries: List[str] = queries, ...  embedding_function = embedding_function, # function that takes converts texts into embeddings, it is optional and can be skipped if provided during initialization ...  qvs_params = {'enbabled': True} ..


Syntax

This page describes ds.query.  DeepMemory is a deep learning model that is trained on the dataset to improve the search results, by aligning queries with the corpus dataset.  It gives up to +22% of recall improvement on an eval dataset.  To use deep_memory, please subscribe to our waitlist.  Training To start training you should first create a vectostore object, and then preprocess the data and use deep memory with it: >>> from deeplake import VectorStore >>> db = VectorStore( ...  path='hub://{$ORG_ID}/{$DATASET_ID}', ...  token=token, # or you can be signed in with CLI ...  runtime={'tensor_db': True}, ...  embedding_function=embedding_function, # function that takes converts texts into embeddings, it is optional and can be provided later ...  ) Copy to clipboard To train a deepmemory model you need to preprocess the dataset so that, corpus, will become a list of list of tuples, where outer list corresponds to the query and inner list to the relevant documents.  Each tuple should contain the document id (id tensor from the corpus dataset) and the relevance score (range is 0-1, where 0 represents unrelated document and 1 related).  queries should be a list of strings.  >>> job_id = db.deep_memory.train( ...  corpus: List[List[Tuple[str, float]]] = corpus, ...  queries: List[str] = queries, ...  embedding_function = embedding_function, # function that takes converts texts into embeddings, it is optional and can be skipped if provided during initialization ...  ) Copy to clipboard Tracking the training progress job_id is string, which can be used to track the training progress.  You can use db.deep_memory.status(job_id) to get the status of the job.  when the model is still in pending state (not started yet) you will see the following: >>> db.deep_memory.status(job_id) ————————————————————– | 6508464cd80cab681bfcfff3 | ————————————————————– | status | pending | ————————————————————– | progress | None | ————————————————————– | results | not available yet | ————————————————————– After some time the model will start training and you will see the following: >>> db.deep_memory.status(job_id) -------------------------------------------------------------- | 6508464cd80cab681bfcfff3 | -------------------------------------------------------------- | status | training | -------------------------------------------------------------- | progress | eta: 2.5 seconds | | | recall@10: 0.62% (+0.62%) | -------------------------------------------------------------- | results | not available yet | -------------------------------------------------------------- Copy to clipboard If you want to get all training jobs you can use db.deep_memory.list_jobs() which will show all jobs that happened on this dataset.  >>> db.deep_memory.list_jobs() ID STATUS RESULTS PROGRESS 65198efcd28df3238c49a849 completed recall@10: 0.62% (+0.62%) eta: 2.5 seconds recall@10: 0.62% (+0.62%) 651a4d41d05a21a5a6a15f67 completed recall@10: 0.62% (+0.62%) eta: 2.5 seconds recall@10: 0.62% (+0.62%) Copy to clipboard Deep Memory Evaluation Once the training is completed, you can use db.deep_memory.evaluate to evaluate the model performance on the custom dataset.  Once again you would need to preprocess the dataset so that, corpus, will become a list of list of tuples, where outer list corresponds to the query and inner list to the relevant documents


Syntax

Each tuple should contain the document id (id tensor from the corpus dataset) and the relevance score (range is 0-1, where 0 represents unrelated document and 1 related) .  queries should be a list of strings.  >>> recalls = db.deep_memory.evaluate( ...  corpus: List[List[Tuple[str, float]]] = corpus, ...  queries: List[str] = queries, ...  embedding_function = embedding_function, # function that takes converts texts into embeddings, it is optional and can be skipped if provided during initialization ...  qvs_params = {'enbabled': True} ...  ) Copy to clipboard recalls is a dictionary with the following keys: with_model contains a dictionary with recall metrics for the naive vector search on the custom dataset for different k values without_model contains a dictionary with recall metrics for the naive vector search on the custom dataset for different k values qvs_params when specified creates a separate vectorstore that tracks all evaluation queries and documents, so that you can use it to compare the performance of deep_memory to naive vector search.  By default, it is turned off.  If enabled the dataset will be created at hub://{$ORG_ID}/{$DATASET_ID}_eval_queries Deep Memory Search After the model is trained you also can search using it: >>> results = db.search( ...  embedding_data: List[str] = queries, ...  embedding_function = embedding_function, # function that takes converts texts into embeddings, it is optional and can be skipped if provided during initialization ...  k = 4, # number of results to return ...  deep_memory = True, # use deep memory model ...  ) Copy to clipboard


Deep Memory API

) Copy to clipboard recalls is a dictionary with the following keys: with_model contains a dictionary with recall metrics for the naive vector search on the custom dataset for different k values without_model contains a dictionary with recall metrics for the naive vector search on the custom dataset for different k values qvs_params when specified creates a separate vectorstore that tracks all evaluation queries and documents, so that you can use it to compare the performance of deep_memory to naive vector search .  By default, it is turned off.  If enabled the dataset will be created at hub://{$ORG_ID}/{$DATASET_ID}_eval_queries Deep Memory Search After the model is trained you also can search using it: >>> results = db.search( ...  embedding_data: List[str] = queries, ...  embedding_function = embedding_function, # function that takes converts texts into embeddings, it is optional and can be skipped if provided during initialization ...  k = 4, # number of results to return ...  deep_memory = True, # use deep memory model ...  ) Copy to clipboard


Deep Memory Evaluation

Once the training is completed, you can use db.deep_memory.evaluate to evaluate the model performance on the custom dataset. Once again you would need to preprocess the dataset so that, corpus, will become a list of list of tuples, where outer list corresponds to the query and inner list to the relevant documents. Each tuple should contain the document id (id tensor from the corpus dataset) and the relevance score (range is 0-1, where 0 represents unrelated document and 1 related). queries should be a list of strings. Copy to clipboard recalls is a dictionary with the following keys: with_model contains a dictionary with recall metrics for the naive vector search on the custom dataset for different k values without_model contains a dictionary with recall metrics for the naive vector search on the custom dataset for different k values qvs_params when specified creates a separate vectorstore that tracks all evaluation queries and documents, so that you can use it to compare the performance of deep_memory to naive vector search. By default, it is turned off. If enabled the dataset will be created at hub://{$ORG_ID}/{$DATASET_ID}_eval_queries