LRU Cache¶

In case the cache storage was local/s3 and is still accessible when unpickled (if same machine/s3 creds present respectively), the earlier cache contents are no longer accessible .  _all_keys()¶ Helper function that lists all the objects present in the cache and the underlying storage.  Returns: set of all the objects found in the cache and the underlying storage.  Return type: set _flush_if_not_read_only()¶ Flushes the cache if not in read-only mode.  _forward(path)¶ Forward the value at a given path to the next storage, and un-marks its key.  _forward_value(path, value)¶ Forwards a path-value pair to the next storage, and un-marks its key.  Parameters: path (str) – the path to the object relative to the root of the provider.  value (bytes, DeepLakeMemoryObject) – the value to send to the next storage.  _free_up_space(extra_size: int)¶ Helper function that frees up space the requred space in cache.No action is taken if there is sufficient space in the cache.  Parameters: extra_size (int) – the space that needs is required in bytes.  _insert_in_cache(path: str, value: Union[bytes, DeepLakeMemoryObject])¶ Helper function that adds a key value pair to the cache.  Parameters: path (str) – the path relative to the root of the underlying storage.  value (bytes) – the value to be assigned at the path.  Raises: ReadOnlyError – If the provider is in read-only mode.  _pop_from_cache()¶ Helper function that pops the least recently used key, value pair from the cache clear(prefix='')¶ Deletes ALL the data from all the layers of the cache and the actual storage.  This is an IRREVERSIBLE operation.  Data once deleted can not be recovered.  clear_cache()¶ Flushes the content of all the cache layers if not in read mode and and then deletes contents of all the layers of it.  This doesn't delete data from the actual storage.  clear_deeplake_objects()¶ Removes all DeepLakeMemoryObjects from the cache.  flush()¶ Writes data from cache_storage to next_storage.  Only the dirty keys are written.  This is a cascading function and leads to data being written to the final storage in case of a chained cache.  get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path


Google Drive Storage Provider¶

class deeplake.core.storage.GDriveProvider¶ Bases: StorageProvider Provider class for using Google Drive storage. __delitem__(path)¶ Delete the object present at the path. Parameters: path (str) – the path to the object relative to the root of the provider. Raises: KeyError – If an object is not found at the path. __getitem__(path)¶ Gets the object present at the path within the given byte range. Parameters: path (str) – The path relative to the root of the provider. Returns: The bytes of the object present at the path. Return type: bytes Raises: KeyError – If an object is not found at the path. __init__(root: str, token: Optional[Union[str, Dict]] = None, makemap: bool = True)¶ Initializes the GDriveProvider Example >>> gdrive_provider = GDriveProvider('gdrive://folder_name/folder_name') Copy to clipboard Parameters: root (str) – The root of the provider. All read/write request keys will be appended to root. token (dict, str, optional) – Google Drive token. Can be path to the token file or the actual credentials dictionary. makemap (bool) – Creates path to id map if True. Note Requires client_secrets.json in working directory if token is not provided. Due to limits on requests per 100 seconds on google drive api, continuous requests such as uploading many small files can be slow. Users can request to increse their quotas on their google cloud platform. __iter__()¶ Generator function that iterates over the keys of the provider. Yields: str – the path of the object that it is iterating over, relative to the root of the provider. __len__()¶ Returns the number of files present inside the root of the provider. Returns: the number of files present inside the root. Return type: int __setitem__(path, content)¶ Sets the object present at the path with the value Parameters: path (str) – the path relative to the root of the provider. value (bytes) – the value to be assigned at the path. _all_keys()¶ Generator function that iterates over the keys of the provider. Returns: set of all keys present at the root of the provider. Return type: set clear(prefix='')¶ Delete the contents of the provider. sync()¶ Sync provider keys with actual storage


Google Cloud Storage Provider¶

class deeplake.core.storage.GCSProvider¶ Bases: StorageProvider Provider class for using GC storage. __contains__(key)¶ Checks if key exists in mapping. __delitem__(key)¶ Remove key. __getitem__(key)¶ Retrieve data. __init__(root: str, token: Optional[Union[str, Dict]] = None, project: Optional[str] = None)¶ Initializes the GCSProvider. Example >>> gcs_provider = GCSProvider('gcs://my-bucket/gcs_ds') Copy to clipboard Parameters: root (str) – The root of the provider. All read/write request keys will be appended to root. token (str/Dict) – GCP token, used for fetching credentials for storage). Can be a path to the credentials file, actual credential dictionary or one of the folowing: - google_default: Tries to load default credentials for the specified project. - cache: Retrieves the previously used credentials from cache if exist. - anon: Sets credentials=None. - browser: Generates and stores new token file using cli. project (str) – Name of the project from GCloud. Raises: ModuleNotFoundError – If google cloud packages aren't installed. __iter__()¶ Iterating over the structure. __len__()¶ Returns length of the structure. __setitem__(key, value)¶ Store value in key. _all_keys()¶ Generator function that iterates over the keys of the provider. Returns: set of all keys present at the root of the provider. Return type: set _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials. These are only relevant to datasets using Deep Lake storage. This info is used to fetch new credentials when the temporary 12 hour credentials expire. Parameters: hub_path (str) – The deeplake cloud path to the dataset. expiration (str) – The time at which the credentials expire. clear(prefix='')¶ Remove all keys with given prefix below root - empties out mapping. Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range. Parameters: path (str) – The path relative to the root of the provider. start_byte (int, optional) – If only specific bytes starting from start_byte are required. end_byte (int, optional) – If only specific bytes up to end_byte are required. Returns: The bytes of the object present at the path within the given byte range. Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0. KeyError – If an object is not found at the path. rename(root)¶ Rename root folder.


Local Storage Provider¶

class deeplake.core.storage.LocalProvider¶ Bases: StorageProvider Provider class for using the local filesystem.  __delitem__(path: str)¶ Delete the object present at the path.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> del local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider.  Raises: KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  ReadOnlyError – If the provider is in read-only mode.  __getitem__(path: str)¶ Gets the object present at the path within the given byte range.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> my_data = local_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider.  Returns: The bytes of the object present at the path.  Return type: bytes Raises: KeyError – If an object is not found at the path.  DirectoryAtPathException – If a directory is found at the path.  Exception – Any other exception encountered while trying to fetch the object.  __init__(root: str)¶ Initializes the LocalProvider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') Copy to clipboard Parameters: root (str) – The root of the provider.  All read/write request keys will be appended to root.' Raises: FileAtPathException – If the root is a file instead of a directory.  __iter__()¶ Generator function that iterates over the keys of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> for my_data in local_provider: ...  pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider.  __len__()¶ Returns the number of files present inside the root of the provider.  Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> len(local_provider) Copy to clipboard Returns: the number of files present inside the root.  Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> local_provider = LocalProvider('/home/ubuntu/Documents/') >>> local_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider.  value (bytes) – the value to be assigned at the path.  Raises: Exception – If unable to set item due to directory at path or permission or space issues.  FileAtPathException – If the directory to the path is a file instead of a directory.  ReadOnlyError – If the provider is in read-only mode.  _all_keys(refresh: bool = False) → Set[str]¶ Lists all the objects present at the root of the Provider.  Parameters: refresh (bool) – refresh keys Returns: set of all the objects found at the root of the Provider.  Return type: set _check_is_file(path: str)¶ Checks if the path is a file.  Returns the full_path to file if True.  Parameters: path (str) – the path to the object relative to the root of the provider


S3 Storage Provider¶

Returns: set of all the objects found at the root of the S3Provider .  Return type: set Raises: S3ListError – Any S3 error encountered while listing the objects.  _check_update_creds(force=False)¶ If the client has an expiration time, check if creds are expired and fetch new ones.  This would only happen for datasets stored on Deep Lake storage for which temporary 12 hour credentials are generated.  _set_hub_creds_info(hub_path: str, expiration: str)¶ Sets the tag and expiration of the credentials.  These are only relevant to datasets using Deep Lake storage.  This info is used to fetch new credentials when the temporary 12 hour credentials expire.  Parameters: hub_path (str) – The Deep Lake cloud path to the dataset.  expiration (str) – The time at which the credentials expire.  _state_keys()¶ Keys used to store the state of the provider.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the s3 bucket (under self.root).  Warning Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  S3GetError – Any other error other than KeyError while retrieving the object.  need_to_reload_creds(err: ClientError) → bool¶ Checks if the credentials need to be reloaded.  This happens if the credentials were loaded from the environment and have now expired.  rename(root)¶ Rename root folder.


Memory Provider¶

class deeplake.core.storage.MemoryProvider¶ Bases: StorageProvider Provider class for using the memory. __delitem__(path: str)¶ Delete the object present at the path. Example >>> memory_provider = MemoryProvider('xyz') >>> del memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – the path to the object relative to the root of the provider. Raises: KeyError – If an object is not found at the path. ReadOnlyError – If the provider is in read-only mode. __getitem__(path: str)¶ Gets the object present at the path within the given byte range. Example >>> memory_provider = MemoryProvider('xyz') >>> my_data = memory_provider['abc.txt'] Copy to clipboard Parameters: path (str) – The path relative to the root of the provider. Returns: The bytes of the object present at the path. Return type: bytes Raises: KeyError – If an object is not found at the path. __getstate__() → str¶ Does NOT save the in memory data in state. __init__(root: str = '')¶ __iter__()¶ Generator function that iterates over the keys of the provider. Example >>> memory_provider = MemoryProvider('xyz') >>> for my_data in memory_provider: ... pass Copy to clipboard Yields: str – the path of the object that it is iterating over, relative to the root of the provider. __len__()¶ Returns the number of files present inside the root of the provider. Example >>> memory_provider = MemoryProvider('xyz') >>> len(memory_provider) Copy to clipboard Returns: the number of files present inside the root. Return type: int __setitem__(path: str, value: bytes)¶ Sets the object present at the path with the value Example >>> memory_provider = MemoryProvider('xyz') >>> memory_provider['abc.txt'] = b'abcd' Copy to clipboard Parameters: path (str) – the path relative to the root of the provider. value (bytes) – the value to be assigned at the path. Raises: ReadOnlyError – If the provider is in read-only mode. _all_keys()¶ Lists all the objects present at the root of the Provider. Returns: set of all the objects found at the root of the Provider. Return type: set clear(prefix='')¶ Clears the provider.


Local Storage Provider¶

Returns: the full path to the requested file .  Return type: str Raises: DirectoryAtPathException – If a directory is found at the path.  clear(prefix='')¶ Deletes ALL data with keys having given prefix on the local machine (under self.root).  Exercise caution! get_bytes(path: str, start_byte: Optional[int] = None, end_byte: Optional[int] = None)¶ Gets the object present at the path within the given byte range.  Parameters: path (str) – The path relative to the root of the provider.  start_byte (int, optional) – If only specific bytes starting from start_byte are required.  end_byte (int, optional) – If only specific bytes up to end_byte are required.  Returns: The bytes of the object present at the path within the given byte range.  Return type: bytes Raises: InvalidBytesRequestedError – If start_byte > end_byte or start_byte < 0 or end_byte < 0.  KeyError – If an object is not found at the path.  rename(path)¶ Renames root folder


deeplake.auto.structured¶

deeplake.auto.structured.base StructuredDataset deeplake.auto.structured.dataframe DataFrame


deeplake.auto.unstructured¶

deeplake.auto.unstructured.base UnstructuredDataset deeplake.auto.unstructured.image_classification ImageClassification deeplake.auto.unstructured.kaggle download_kaggle_dataset()


deeplake.auto.unstructured.base¶

class deeplake.auto.unstructured.base.UnstructuredDataset(source: str)¶


deeplake.auto.unstructured.image_classification¶

class deeplake.auto.unstructured.image_classification.ImageClassification(source: str)¶ structure(ds: Dataset, use_progress_bar: bool = True, generate_summary: bool = True, image_tensor_args: dict = {}) → Dataset¶ Create a structured dataset. Parameters: ds (Dataset) – A Deep Lake dataset object. use_progress_bar (bool) – Defines if the method uses a progress bar. Defaults to True. generate_summary (bool) – Defines if the method generates ingestion summary. Defaults to True. image_tensor_args (dict) – Defines the sample compression of the dataset (jpeg or png). Returns: A Deep Lake dataset.


deeplake.auto.structured.base¶

class deeplake.auto.structured.base.StructuredDataset(source)¶ Initializes a structured dataset. Parameters: source (str) – The local path to folder or file containing a structured dataset and of the form ./path/to/dataset or ~/path/to/dataset or path/to/dataset.


deeplake.auto.structured.dataframe¶

class deeplake.auto.structured.dataframe.DataFrame(source)¶ fill_dataset(ds: Dataset, progressbar: bool = True) → Dataset¶ Fill dataset with data from the dataframe - one tensor per column Parameters: ds (Dataset) – A Deep Lake dataset object. progressbar (bool) – Defines if the method uses a progress bar. Defaults to True. Returns: A Deep Lake dataset.


Let's build the AI Image Search App!

We need four things: data, a way to generate embeddings, a vector database to store them, and an interactive app. Let's start with the data. You can also take a look at the article companion video below and fork the GitHub repo as well.


How to Conduct Multimodal Search with ImageBind & Deep Lake?

If you've ever thought that Multimodality in AI is limited to generating images with Midjourney or Dall-E, think again. Multimodal use cases will increasingly become more prevalent, with each additional modality unlocking incremental business value.In this guide, we'll explore the creation of a search engine that retrieves AI-generated images using text, audio, or visual inputs, opening new doors for accessibility, user experience, and business intelligence.To achieve this, we will leverage ImageBind by Meta AI, a game-changer for multimodal AI applications. It captures diverse data modalities and maps them into a common vector space, making our search more powerful. This unlocks novel use cases beyond a vanilla image similarity search.Unlike anything else, Deep Lake by Activeloop enables the storage and querying of multimodal data (not only the embeddings but also the raw data!). With Deep Lake and ImageBind, the potential applications of this technology are vast. Whether improving product discovery in eCommerce, streamlining digital media libraries, enhancing accessibility in tech products, or powering intuitive search in digital archives, this innovation can drive user satisfaction and business growth.


Gathering AI-Generated Images from Lexica for AI Image Search

We were thinking about interesting data to search on, so we came across a the Lexica dataset containing images from Lexica - a website where you can search for AI generate (mostly Stable Diffusion) images. The image search works by exact match in the prompt, while we will create a semantic search.Since we want to display the images on the web app, we need to get them and store them somewhere (in our case, an S3 bucket - but since Deep Lake is serverless, you can do it wherever). So, first, we load the hugging face dataset1# pip install datasets 2from datasets import load_dataset 3 4dataset = load_dataset('xfh/lexica_6k', split='train') 5CopyThen we store each image on a disk.1for row in dataset: 2 image = row['image'] 3 name = row['text'] 4 image.save(f'{row[name]}.jpg') 5CopyThis is a simplified version. For speed, we used a thread pool on dataset batches. Then we stored the images in a S3 bucket to show them later on the app.


What is ImageBind?

In a nutshell, ImageBind is a transformer-based model trained on multiple pairs of modalities, e.g., text-image, and learns how to map all of them in the same vector space.  This means that a text query 'dog' will be mapped close to a dog image, allowing us to search in that space seamlessly.  The main advantage is that we don't need one model per modality, like in CLIP where you have one for text and one for image, but we can use the same weights for all of them.  The following image taken from the paper shows the idea.The model supports images, text, audio, depth, thermal, and IMU data.  We will limit ourselves to the first three.  The task of learning similar embeddings for similar concepts in different modalities, e.g., 'dog' and an image of a dog, is called alignment.  The ImageBind authors used a Vision Transformer (ViT) , a typical architecture these days.  Due to the number of different modalities, the preprocessing step is different.  For example, for videos, we need to consider the time dimension, the audio needs to be converted to a spectrogram, but the main weights are the same.To learn to align pairs of modalities (text, image), (audio, text), the Authors used contrastive learning and specifically the InfoNCE loss.  Using InfoNCE, the model is trained to identify a positive example from a batch of negative ones by maximizing the similarity between positive pairs and minimizing the similarity between negative ones.The most exciting thing is that even if the model was trained on pairs (text, image) and (audio, text), the model also learns (image, audio).  This is what the Authors called the 'Emergent alignment of unseen pairs of modalities'/Moreover, we can do Embedding space arithmetic, adding (or subtracting) multiple modalities embeddings to capture different semantic information.  We'll play with it later on.For the most curious reader, you can learn more by reading the paperOkay, let's get the image embeddings.  We need to load the model and store the embeddings for all the images, so we can, later on, read them and dump them in the vector database.Getting the embeddings is quite easy with the ImageBind code code.1import data 2import torch 3from models import imagebind_model 4from models.imagebind_model import ModalityType 5 6text_list=['A dog.', 'A car', 'A bird'] 7image_paths=['.assets/dog_image.jpg', '.assets/car_image.jpg', '.assets/bird_image.jpg'] 8audio_paths=['.assets/dog_audio.wav', '.assets/car_audio.wav', '.assets/bird_audio.wav'] 9 10device = 'cuda:0' if torch.cuda.is_available() else 'cpu' 11 12# Instantiate model 13model = imagebind_model.imagebind_huge(pretrained=True) 14model.eval() 15model.to(device) 16 17# Load data 18inputs = { 19 ModalityType.TEXT: data.load_and_transform_text(text_list, device), 20 ModalityType.VISION: data.load_and_transform_vision_data(image_paths, device), 21 ModalityType.AUDIO: data.load_and_transform_audio_data(audio_paths, device), 22} 23 24with torch.no_grad(): 25 embeddings = model(inputs) 26 27print(embeddings[ModalityType.VISION]) 28print(embeddings[ModalityType.AUDIO]) 29print(embeddings[ModalityType.TEXT]) 30CopyWe first store all the image embeddings as pth files on disk using a simple function to batch the images


How to Store Embeddings in a Vector Database?

After we have obtained our embeddings, load them into Deep Lake.  You can learn more about Deep Lake in Deep Lake docs.To start, we need to define the vector database.1import deeplake 2 3ds = deeplake.empty( 4 path='<hub://<YOUR_ACTIVELOOP_ORG_ID>/<DATASET_NAME>', 5 runtime={'db_engine': True}, 6 token='<YOUR_TOKEN>', 7 overwrite=overwrite, 8 ) 9CopyWe are setting db_engine=True, meaning we won't store the data on our disk, but we will use the managed Deep Lake database to store the data and run our queries.  This comes in handy when developing applications where you need to have compute and data storage separation while keeping data where it matters to you.  You can deploy the same setup entirely locally and not send your sensitive data anywhere it's not supposed to be.Next, we need to define the shape of the data.1with ds: 2 ds.create_tensor( 3 'metadata', 4 htype='json', 5 create_id_tensor=False, 6 create_sample_info_tensor=False, 7 create_shape_tensor=False, 8 chunk_compression='lz4', 9 ) 10 ds.create_tensor('images', htype='image', sample_compression='jpg') 11 ds.create_tensor( 12 'embeddings', 13 htype='embedding', 14 dtype=np.float32, 15 create_id_tensor=False, 16 create_sample_info_tensor=False, 17 max_chunk_size=64 * MB, 18 create_shape_tensor=True, 19 ) 20CopyHere we create three tensors, one to hold the metadata of each embedding, one to store the images (in our case, this is optional, but it's cool to showcase), and one to store the actual tensors of our embeddings.  Deep Lake stands out from the crowd with this feature.Then it's time to add our data.  We stored batched embeddings to disk as .pth files if you recall.1 2def add_torch_embeddings(ds: deeplake.Dataset, embeddings_data_path: Path): 3 embeddings_data = torch.load(embeddings_data_path) 4 for embedding_data in embeddings_data: 5 metadata = embedding_data['metadata'] 6 embedding = embedding_data['embedding'].cpu().float().numpy() 7 image = read_image(metadata['path']).permute(1, 2, 0).numpy() 8 metadata['path'] = Path(metadata['path']).name 9 ds.append({'embeddings': embedding, 'metadata': metadata, 'images': image}) 10 11embeddings_data_paths = embeddings_root.glob('*.pth') 12list( 13 tqdm( 14 map( 15 partial(add_torch_embeddings, ds), 16 embeddings_data_paths, 17 ) 18 ) 19) 20CopyHere we are just iterating all the embeddings file and adding everything within each one.  We can have a look at the data from activeloop dashboard - spoiler alert.  It is quite cool.  You can also visualize the 3D embedding space (and pick your preferred clustering algorithm).Cool!To run a query on Deep Lake we can1embedding = # getting the embeddings from ImageBind 2dataset_path = # our path to active loop dataset 3limit = # number of results we want 4query = f'select * from (select metadata, cosine_similarity(embeddings, ARRAY{embedding.tolist()}) as score from '{dataset_path}') order by score desc limit {limit}' 5query_res = ds.query(query, runtime={'tensor_db': True}) 6# query_res = Dataset(path='hub://zuppif/lexica-6k', read_only=True, index=Index([(1331, 1551)]), tensors=['embeddings', 'images', 'metadata']) 7CopyWe can access the metadata by1query_res.metadata.data(aslist=True)['value'] 2# [{'path': '5e3a7c9b-e890-4975-9342-4b6898fed2c6.jpeg'}, {'path': '7a961855-25af-4359-b869-5ae1cc8a4b95.jpeg'}] 3CopyIf you remember, these are the metadata we stored previously, so the image filename.  We wrapped all the vector store-related code into a VectorStore class inside vector_store.py.1class VectorStore(): 2 ..


Create Image Embeddings for Multimodal Retrieval

To search the images given a query, we need to encode the images in embeddings, then encode the query and perform cosine similarity to find the 'closets,' aka 'most similar' images. We want to search using multiple modalities, text, images, or audio. For this reason, we decided to use the new Meta model called ImageBind. If you want, you can learn more about generating image embeddings.


Developing an AI Image Search App with Gradio

We'll use Gradio to create a sleek UI for the app.  We'd need first to define the inputs and outputs of the app.1with gr.Blocks() as demo: 2 # a little description 3 with Path('docs/APP_README.md').open() as f: 4 gr.Markdown(f.read()) 5 # text input 6 text_query = gr.Text(label='Text') 7 with gr.Row(): 8 # image input 9 image_query = gr.Image(label='Image', type='pil') 10 with gr.Column(): 11 # audio input 12 audio_query = gr.Audio(label='Audio', source='microphone', type='filepath') 13 search_button = gr.Button('Search', label='Search', variant='primary') 14 # and a little section to change the settings 15 with gr.Accordion('Settings', open=False): 16 limit = gr.Slider( 17 minimum=1, 18 maximum=30, 19 value=15, 20 step=1, 21 label='search limit', 22 interactive=True, 23 ) 24 # This will show the images 25 gallery = gr.Gallery().style(columns=[3], object_fit='contain', height='auto') 26 27CopyThis results in the following UI.Then we need to link the search button to the actual search code.1...  2search_button.click( 3 search_button_handler, [text_query, image_query, audio_query, limit], [gallery] 4 ) 5CopyThis means text_query, image_query, audio_query, limit are the inputs to search_button_handler and gallery is the output.where search_button_handler is1...  2vs = VectorStore.from_env() 3model = get_model() 4...  5def search_button_handler( 6 text_query: Optional[str], 7 image_query: Optional[Image.Image], 8 audio_query: Optional[str], 9 limit: int = 15, 10): 11 if not text_query and not image_query and not audio_query: 12 logger.info('No inputs!') 13 return 14 # we have to pass a list for each query 15 if text_query == '' and len(text_query) <= 0: 16 text_query = None 17 if text_query is not None: 18 text_query = [text_query] 19 if image_query is not None: 20 image_query = [image_query] 21 if audio_query is not None: 22 audio_query = [audio_query] 23 start = perf_counter() 24 logger.info(f'Searching ...') 25 embeddings = get_embeddings(model, text_query, image_query, audio_query).values() 26 # if multiple inputs, we sum them 27 embedding = torch.stack(list(embeddings), dim=0).sum(0).squeeze() 28 logger.info(f'Model took {(perf_counter() - start) * 1000:.2f}') 29 images_paths, query_res = vs.retrieve(embedding.cpu().float(), limit) 30 return [f'{BUCKET_LINK}{image_path}' for image_path in images_paths] 31CopySo for each input, we check that they exist, if they do, we wrap them into a list.  This is needed for our internal implementation.  vs.retrieve is a function of VectorStore, just a utility class that wrap all the code in the same place.  Inside that function, we first compute the embeddings using the get_embeddings function shown before, and then we run a query against the vector db.We have stored all the images in S3, so we return a list of links to the images there; this is the input to gr.GalleryWhere that's it! Let's see it in action.Normal single modality works as expected.If we receive more than one input, we sum them up.  Basically,1embedding = torch.vstack(list(embeddings)).sum(0) 2CopyFor example, we can pass an image of a car and an audio of a f1 race.or the text + image.  We can also do text + image + audio


3 def retrieve(self, embedding: torch.Tensor, limit: int = 15) -> List[str]: 4 query = f'select * from (select metadata, cosine_similarity(embeddings, ARRAY{embedding.tolist()}) as score from '{self.dataset_path}') order by score desc limit {limit}' 5 query_res = self._ds.query(query, runtime={'tensor_db': True}) 6 images = [ 7 el['path'].split('.')[0] 8 for el in query_res.metadata.data(aslist=True)['value'] 9 ] 10 return images, query_res 11CopySo, since the model supports text, images, and audio we can also create a utility function to make our life easier.1@torch.no_grad() 2def get_embeddings( 3 model: torch.nn.Module, 4 texts: Optional[List[str]], 5 images: Optional[List[ImageLike]], 6 audio: Optional[List[str]], 7 dtype: torch.dtype = torch.float16 8) -> Dict[str, torch.Tensor]: 9 inputs = {} 10 if texts is not None: 11 # they need to be ints 12 inputs[ModalityType.TEXT] = load_and_transform_text(texts, device) 13 if images is not None: 14 inputs[ModalityType.VISION] = load_and_transform_vision_data(images, device, dtype) 15 if audio is not None: 16 inputs[ModalityType.AUDIO] = load_and_transform_audio_data(audio, device, dtype) 17 embeddings = model(inputs) 18 return embeddings 19CopyAlways remember the torch.no_grad decorator :) Next, we can easily do1vs = VectorStore(...) 2vs.retrieve(get_embeddings(texts=['A Dog'])) 3Copyqueryresults 'A Dog' 1vs = VectorStore(...) 2vs.retrieve(get_embeddings(images=['car.jpeg'])) 3Copyqueryresults


The Future of Multimodal Search

In conclusion, the fusion of ImageBind and Deep Lake provides a robust framework for developing an AI-based multimodal search engine. By leveraging AI-generated images and text, audio, and visual inputs, we've shown how it's possible to make strides toward more intuitive, efficient, and inclusive search experiences.For machine learning engineers, this exploration opens up new avenues for creating more user-centric applications. At the same time, business executives can see the transformative impact of AI on a wide array of industries.The future of search is here, and it's multimodal. Now it's your turn. Try the demo and share the results!


What Modalities are Supported by ImageBind?

ImageBind supports six different modalities - images, text, audio, depth, thermal, and IMU data. The modality encoders are based on a Transformer architecture, including the Vision Transformer (ViT) for images and videos. Audio is encoded by converting a 2-second audio sample into spectrograms, which are treated as 2D signals and encoded using a ViT. Thermal images and depth images are treated as one-channel images and also encoded using a ViT.


Is ImageBind Open Source?

ImageBind is an open-source project with a PyTorch implementation and pretrained models available. The model and accompanying weights are available for download and can be used to feed text, image, and audio data into ImageBind.


Is ImageBind Applications?

ImageBind enables novel applications ‘out-of-the-box' including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. It has several potential use cases, including information retrieval, zero-shot classification, and connecting the output of ImageBind to other models. ImageBind could play a crucial role in developing autonomous vehicles, helping them to perceive and interpret their surroundings more effectively.


Who Developed ImageBind?

ImageBind has been developed and open-sourced by researchers at Meta.


What is multimodal AI?

Multimodal AI is an AI category that integrates various types, or modalities, of data to reach more precise conclusions, make insightful deductions, or provide more accurate real-world problem predictions. Multimodal AI platforms use and learn from a variety of data including video, audio, speech, images, text, and numerous traditional structured datasets.


What are the benefits of multimodal AI?

Multimodal AI generally surpasses single modal AI in many real-world situations. Through the combination of different data types, multimodal AI can produce more accurate, human-like responses, thereby enhancing its versatility and adaptability in varying scenarios. Industries like healthcare, finance, and retail could significantly benefit from multimodal AI due to its ability to provide precise and customized responses.


What is multimodal machine learning?

Multimodal machine learning is an evolving multidisciplinary research domain aimed at developing computer agents with intelligent capabilities to process and connect information from multiple modalities. There has been considerable progress in this emerging field over recent years.


What are the challenges of multimodal AI?

Despite its potential and advantages, multimodal AI does have associated challenges, specifically related to data quality and interpretation for developers. Certain modalities may be excessively noisy, complicating the AI system's learning process. The complexity of multimodal AI systems necessitates substantial computational resources. Lastly, for these systems to be trusted, they need to be explainable.


What are the applications of multimodal AI?

Multimodal AI has extensive applications across various sectors including healthcare, finance, and retail. Multimodal conversational AI systems can answer queries, complete tasks, and mimic human conversations by comprehending and conveying information from multiple modalities. Complex recipe generation from images is another potential application for multimodal AI.


What is the difference between multimodal AI and single modal AI?

The key distinction between multimodal AI and conventional single modal AI lies in the data. Single modal AI is typically designed to handle a singular data source or type. For instance, a financial AI leverages business financial data, along with wider economic and industrial sector data, to conduct analyses, make financial predictions, or identify potential financial issues for the company. In other words, the single modal AI is specialized for a specific task.


What is ImageBind?

Note that we store a dictionary to add metadata; we are interested in the image_path and will use it later.1@torch.no_grad() 2def encode_images( 3 images_root: Path, 4 model: torch.nn.Module, 5 embeddings_out_dir: Path, 6 batch_size: int = 64, 7): 8 # not the best way, but the faster, best way would be to use a torch Dataset + Dataloader 9 images = images_root.glob('*.jpeg') 10 embeddings_out_dir.mkdir(exist_ok=True) 11 for batch_idx, chunk in tqdm(enumerate(chunks(images, batch_size))): 12 images_paths_str = [str(el) for el in chunk] 13 images_embeddings = get_images_embeddings(model, images_paths_str) 14 torch.save( 15 [ 16 {'metadata': {'path': image_path}, 'embedding': embedding} 17 for image_path, embedding in zip(images_paths_str, images_embeddings) 18 ], 19 f'{str(embeddings_out_dir)}/{batch_idx}.pth', 20 ) 21CopyNote that a better solution would have been using torch Dataset + Dataloader, and we dive into this in this image embedding tutorial.


Feel free to test it out!Some of the results were not too great, 'cartoon' + cat image.In our experiments, we've noticed that the text is more potent compared to image and audio when combined with the other modalities


The Problem in RAGs is Inaccurate Retrieval

Enterprises are currently building ‘chat with your data' applications for a variety of use cases ranging from documenting internal processes to adding automation for customer support. These solutions are typically implemented using Retrieval Augmented Generation (RAG) systems that provide context to Large Language Models (LLMs) like GPT-4. Retrieval Augmented Generation also infuses the model with strategically relevant bits of the latest information, reducing the need to fine-tune the model separately and removing the training dataset restriction. While truthfulness and faithfulness are important, the utility of the implementation ultimately depends on the retrieval accuracy. At best, RAG applications achieve a retrieval accuracy of 70%, so in 30% of cases, an LLM like GPT-4 cannot provide an accurate response to the user's requests. This level of inconsistency is obvious and perceptible by the end user, and it is unacceptable when applications play a crucial role in business-critical operations.


Existing Solutions Provide Incremental Improvements

There are several options for boosting the accuracy of RAG systems such as: adjusting the input with feature engineering, fine-tuning embeddings, employing hybrid or lexical search, reranking the final results with cross encoders, and context-aware fine-tuning your LLM. However, these techniques offer marginal improvements that do not fundamentally change the user-experience of using LLM apps.Another approach for increasing accuracy is to use a broader LLM context, but this often results in higher costs with only minor gains. Why? Answer quality decreases, and the risk of hallucination increases As shown in a recent study from Stanford, LLMs struggle to pick out key details from large contexts, especially if these details are in the middle. These models find it hard to focus on relevant info when given many documents, and the problem gets worse (up to 20%) with more documents. The bigger the context, the higher the cost for LLM execution. LLM providers bill based on data size, and adding more data to a query increases the cost. For self-deployments, the cost manifests though higher compute requirements and larger infrastructure.


Customer Spotlight: Munai Boosts Vector Search Accuracy by 41% with Deep Memory

Health tech startup Munai, backed by the Bill & Melinda Gates Foundation, provides intelligent patient management. 'Leveraging Deep Memory, we've achieved an astounding 18.6% boost in our vector search accuracy across medical documents. Such a transformation directly impacts the efficiency and efficacy of our solutions.' - Mateus Cichelero da Silva, Data Chapter Lead at Munai The company collaborates with one of the largest hospitals in Brazil to facilitate patient management for healthcare professionals and health insurance companies. 'Munai is one of the leading innovative generative AI solution providers in Healthcare. We are more than excited to partner with them to deploy mission critical AI workloads into hospitals' - Davit Buniatyan, CEO of Activeloop Hugo Morales e Cristian Rocha, founders of Munai, healthtech generative AI startup (Crédito: Divulgação) 'Deep Memory marks a groundbreaking advancement in constructing precision-focused RAG systems for medical applications. In contexts as delicate as healthcare environments, this is not just an improvement; it's a revolution' - Cristian Rocha CEO of Munai


Introducing Deep Memory: An Actual Solution That Works

Deep Memory significantly increases Deep Lake's vector search accuracy up to +22% by learning an index from labeled queries tailored to your application, without impacting search time. These results can be achieved with only a few hundred example pairs of prompt embeddings and most relevant answers from the vector store.Post training, vector search is used without modifications as if normal.Embeddings can still be computed using a model of your choice such as Open AI ada-002 or other OSS models BGE by BAAI. Furthermore, search results from Deep Memory can be further improved by combining them with lexical search or reranker.Recall comparison of lexical search, vector search and deep memory at top 10 retrieved documents


Higher Accuracy at Lower Cost for Your RAG Application

Deep Memory increases retrieval accuracy without altering your existing workflow. Additionally, by reducing the top_k input into the LLM, you can significantly cut inference costs via lower token usage.


How can you do it yourself?

Let's get hands-on. first, we load the indexed dataset, let's call it corpus. It has to be in managed database (runtime={'tensor_db': True}) to leverage Deep Memory training on the Deep Lake Managed Tensor Database.1from deeplake import VectorStore 2 3corpus = VectorStore( 4 'hub://activeloop-test/scifact-demo', 5 embedding_function = embeddings.embed_documents, 6 runtime={'tensor_db': True} 7) 8CopyThen, we construct a dataset of questions and relevance. Relevance is a set of pairs (corpus.id: str, significance: str) which provides information where is the answer inside the corpus. Sometimes an answer can be found in multiple locations or have different significance. Relevance enables Deep Memory training to correctly optimize the embedding space for higher accuracy. The goal here is to obtain a labelled query dataset similar to what your user would ask in production setting. To observe significant improvements, we would expect to have few hundred query pairs. If you don't have already labelled query dataset, you can also use GPT4 to generate synthetic pairs based on chunks from the corpus.Then, we kickstart the training job on the Activeloop platform.1questions = ['question 1', ...] 2relevance = [[(corpus.dataset.id[0], 1), ...], ...] 3 4job_id = corpus.deep_memory.train( 5 queries = questions, 6 relevance = relevance, 7 embedding_function = embeddings.embed_documents, 8) 9CopyWe can monitor training progress and observe that we received up to +21.5% on validation set in few minutes 🤯.1corpus.deep_memory.status(job_id) 2CopyWe can directly enable Deep Memory inside vector search without waiting the training to finish.1corpus.search( 2 embedding_data = 'Female carriers of the Apolipoprotein E4 (APOE4) allele have increased risk for dementia.', 3 embedding_function = embeddings.embed_query, 4 deep_memory = True 5) 6CopyA GPT4 answer based on naive vector search would produce the following result 'The provided context does not explicitly state that female carriers of the Apolipoprotein E4 (APOE4) allele have an increased risk for dementia.' however with deep memory enabled 'The Apolipoprotein E4 (APOE4) allele is a confirmed susceptibility locus for late-onset Alzheimer's disease.' While qualitative results are good for manual inspection, in order to evaluate in a production setting we would like to quantitatively measure the accuracy of the information retrieval on previously unseen queries. This provides objective metrics to compare against naive vector search, or even lexical search.1corpus.deep_memory.evaluate( 2 queries = test_questions, 3 relevance = test_relevance, 4 embedding_function = embeddings.embed_documents, 5 top_k=[1, 3, 5, 10, 50, 100] 6) 7CopyX Axis computes recall per top k. Higher recall % and lower K is better.As you increase k , the likelihood of correct answer appearing in the retrieved results increases. However, since the LLM API costs are proportional to the number of tokens in the context, your goal is to decrease top_k while preserving accuracy. Deep Memory lets you to decrease top_k from 10 to 3 while preserving the accuracy, resulting in up to 70% lower token usage, faster computations, and lower costs.


Get Started Today

A number of enterprise customers are already reaping the benefits of the Deep Memory-powered Retrieval Augmented Generation apps. Dive into the future with our hosted training service in our Managed Tensor Database.Deep Memory is now available by signing up for a waitlist on this link.


Benefits of Deep Memory

Higher Quality: Accuracy improves up to +22% learning from queries on average, with customers like Munai achieving up to 41% increase. Cost Reduction: Saving up to 50% GPT4 cost and execution time by reducing top k. Simple to Use: No change in Deep Lake Vector Search usage, same speed with higher accuracy. Natively integrated inside Langchain & LlamaIndex. Smaller, Faster, Cheaper: Use smaller text embeddings combined with Deep Memory such as BGE_small (384 dims) to beat OpenAI's ada-002 (1536 dims) and/or Elastic (BM25).


What is LangChain?

LangChain's rise has been nothing short of impressive.LangChain is a library that provides tools for working with LLMs. It streamlines the large language model development process by offering a chain of steps to generate text from a specific input prompt. The toolkit handles prompt input, text generation, and generated output manipulation. Effectively, it allows for developing conversational agents that use LLMs to generate natural language responses. These agents can be used for various applications, from supporting customers at scale to answering company-specifc questions based on your team's internal Notion database.


How does LangChain work?

LangChain provides a pipeline of steps that generate text from input prompts. The pipeline comprises seven main components: input (prompt templates), the large language model, agents, utilities, document loaders, chains, indexes, and memory.


Input - LangChain Prompt Templates

The input component specifies the input prompt or the initial input provided to the pipeline (check out the most complete prompt engineering resource on GitHub). This comes in the form of a template that defines the structure of the prompt, including the format and content (sadly, no more Sydney for the foreseeable future 🙁). A typical example, for instance, could be asking the LLM to act as a machine learning expert or writing a bulleted list of possible answers.Choosing the correct template is vital since a minor change in wording may significantly impact LLM performance - with up to 45% of improvement possible against the baseline with some methods.


Document Loaders

With Document Loaders module, you can ingest documents (e.g., pdfs of quarterly reports, powerpoints, etc., as we will do shortly!) into the LLM for further analysis (typically question answering).


Large Language Model

LangChain enables access to a range of pre-trained LLMs (e.g., GPT-3) trained on large datasets. The large language model component generates output (in this case, text) based on the prompt and input. These LLMs can further be fine-tuned to match the needs of specific conversational agents (e.g., if you are building a legal-specific chatbot). If you want to train your own LLM, check out how to train your Large Language Models (LLMs) efficiently.


Chains

LangChain Chains, arguably, are THE component that makes LangChain stand out. LangChain provides a simple interface to go beyond a single LLM call, but rather enables users to link up LLMs or calls to LLMs and other Utils. The LangChain chains can be generic - a chain made up of links, which can be either primitives (prompts, LLMs, Utils, or other chains) or additional chains. In addition, chains can be sequential, meaning that multiple chains are run one after another, with prior outputs serving as inputs to the next-in-line chain link.For instance, we used the Program-Aided Language Models in the previous example. Here, the chain takes user input (a reasoning question - 'I'm 6 feet tall. How many feet long is the circumference of the smallest moon of Jupiter compared to me?'), and decides that no additional Utils must be used to answer the question.Let's try asking another question. 'Alex has three times the number of balls as Ben. Ben has two more balls than Claire. If Claire has four balls, how many total balls do the three have together?'pal_chain = PALChain.from_math_prompt(llm, verbose=True) question = 'Alex has three times the number of balls as Ben. Ben has two more balls than Claire. If Claire has four balls, how many total balls do the three have together?' pal_chain.run(question) > Entering new PALChain chain... def solution(): '''Alex has three times the number of balls as Ben. Ben has two more balls than Claire. If Claire has four balls, how many total balls do the three have together?''' claire_balls = 4 ben_balls = claire_balls + 2 alex_balls = ben_balls * 3 total_balls = alex_balls + ben_balls + claire_balls result = total_balls return result > Finished chain. '28'


LangChain Agents

LangChain Agents make a call regarding which action should be taken next (e.g., a calculation should be handled by Wolfram Alpha, while a question about how old is the famous person might trigger a web search) to satisfy the directive given by the input. All the possible actions are chained together in a logical loop. In effect, you can use Tools (generic Utils, other chains, or even other agents) or a Toolkit (a group of tools) to minimize ChatGPT's peculiarity of being very bad at answering math questions, and asking sophisticated questions, like 'I'm 6 feet tall. How many feet longer is the circumference of the smallest moon of Jupiter compared to me?' (it's less than you think).Notably, Agents engage an LLM to decide which Action to take, execute that Action, see an Observation, and iterate until the goal is reached.For the previous query, the Agent would act as follows: Thought: Do I need to use a tool? Yes Action: PAL-MATH Action Input: The circumference of the smallest moon of Jupiter is 6 times longer than 6 feet. Observation: 36 Thought: Do I need to use a tool? No AI: The circumference of the smallest moon of Jupiter is 36 feet longer than 6 feet. The circumference of the smallest moon of Jupiter is 36 feet longer than 6 feet.


Indexes

Language models are more powerful and valuable when combined with your own (proprietary) data. LangChain provides common indices for working with data - embeddings or, most commonly, a data store, such as Deep Lake.


LangChain Utils

LangChain provides an extensive collection of common utilities (Utils) to use in your application, such as Python REPLs (LLM would generate code to calculate the answer, run that code to get the answer and print it out), bash commands (e.g., to interface with the local system) or search engines, as well as a requests wrapper (to link a URL published post-2021 that ChatGPT doesn't know about, for instance). Utils are employed to strengthen the potency of LLMs when interacting with other sources of knowledge or computation.


LangChain Memory

Finally, LangChain includes a memory component. By default, Chains and Agents are stateless, which means they react to each new input query independently of prior inputs. Memory thus allows agents to recall previous interactions with users (a great use case for chatbots) or remember entities (e.g., recall all the Horcruxes Lord Voldemort created and who destroyed which one after you feed the entirety of Pottervers into it). This enables agents to provide users with more personalized and contextualized responses as time passes.


What are the top LangChain use cases?

Agents: LangChain provides a dedicated module for building agents, which are systems that use a language model to interact with other tools. They can be used to create chatbots or power the next generation of personal assistants. Chatbots: Thanks to the Memory module, LangChain can be used to create chatbots, with the ability to remember what happened in previous interactions and the ability to recall distinct entities and answer questions based on them. Data Augmented Generation: LangChain provides tools for generating text based on specific data, such as summarization of a specific text or question answering over one particular or multiple pieces of text. Question Answering: As previously mentioned, LangChain can be utilized to perform question answering over your document data (powerpoints, pdfs, etc.), with the option to create an index over the data to save time and money (and, potentially, additional headcount on Research Assistants or Dedicated Knowledge teams - looking at you, consulting companies!). Code Understanding: You can use LangChain to parse large code repositories, understand how certain variables are defined or how the functions interact. More interestingly, you can also create demo scenarios and ask LangChain how the code would perform in that scenario. We've written a guide on using LangChain to Chat with GitHub Code Repos (we've used the Twitter recommendation algorithm). Check it out! Summarization: LangChain can synthesize an actionable summary of multiple longer documents, which helps distill long reports into the core pieces of information (or maybe even sifting through long and uninteresting email chains). Evaluation: LLMs are notoriously hard to evaluate with traditional 'hard' metrics. LangChain provides tools for assessing language models, specifically (data-augmented) question-answering systems. Model Comparison: LangChain provides a ModelLaboratory for testing and comparing various alternatives of prompts, models, and even chains to use.


Practical Example: Using LangChain and Deep Lake to Explore Amazon's Revenue Growth Pre- and Post-Pandemic

In this tutorial, we will first load Amazon's quarterly financial reports, embed using OpenAI's API, store the data in Deep Lake, and then explore it by asking questions.1!pip3 install langchain deeplake pypdf openai tiktoken 2 3Copy1from langchain.embeddings.openai import OpenAIEmbeddings 2from langchain.vectorstores import DeepLake 3from langchain.text_splitter import CharacterTextSplitter 4from langchain import OpenAI 5from langchain.chains import RetrievalQA 6from langchain.llms import OpenAIChat 7from langchain.document_loaders import PagedPDFSplitter 8CopyNext, we need to get API tokens from OpenAI and Activeloop (learn how to receive an API token for Activeloop after registering).1import os 2os.environ['OPENAI_API_KEY']='OPEN_AI_KEY_HERE' 3os.environ['ACTIVELOOP_TOKEN']='ACTIVELOOP_TOKEN' 4CopyNext, we need to get API tokens from OpenAI and Activeloop (learn how to receive an API token for Activeloop after registering).First, let's download financial reports from Amazon and load them into pages:1import requests 2import tqdm 3from typing import List 4 5# financial reports of amamzon, but can be replaced by any URLs of pdfs 6urls = ['https://s2.q4cdn.com/299287126/files/doc_financials/Q1_2018_-_8-K_Press_Release_FILED.pdf', 7 'https://s2.q4cdn.com/299287126/files/doc_financials/Q2_2018_Earnings_Release.pdf', 8 'https://s2.q4cdn.com/299287126/files/doc_news/archive/Q318-Amazon-Earnings-Press-Release.pdf', 9 'https://s2.q4cdn.com/299287126/files/doc_news/archive/AMAZON.COM-ANNOUNCES-FOURTH-QUARTER-SALES-UP-20-TO-$72.4-BILLION.pdf', 10 'https://s2.q4cdn.com/299287126/files/doc_financials/Q119_Amazon_Earnings_Press_Release_FINAL.pdf', 11 'https://s2.q4cdn.com/299287126/files/doc_news/archive/Amazon-Q2-2019-Earnings-Release.pdf', 12 'https://s2.q4cdn.com/299287126/files/doc_news/archive/Q3-2019-Amazon-Financial-Results.pdf', 13 'https://s2.q4cdn.com/299287126/files/doc_news/archive/Amazon-Q4-2019-Earnings-Release.pdf', 14 'https://s2.q4cdn.com/299287126/files/doc_financials/2020/Q1/AMZN-Q1-2020-Earnings-Release.pdf', 15 'https://s2.q4cdn.com/299287126/files/doc_financials/2020/q2/Q2-2020-Amazon-Earnings-Release.pdf', 16 'https://s2.q4cdn.com/299287126/files/doc_financials/2020/q4/Amazon-Q4-2020-Earnings-Release.pdf', 17 'https://s2.q4cdn.com/299287126/files/doc_financials/2021/q1/Amazon-Q1-2021-Earnings-Release.pdf', 18 'https://s2.q4cdn.com/299287126/files/doc_financials/2021/q2/AMZN-Q2-2021-Earnings-Release.pdf', 19 'https://s2.q4cdn.com/299287126/files/doc_financials/2021/q3/Q3-2021-Earnings-Release.pdf', 20 'https://s2.q4cdn.com/299287126/files/doc_financials/2021/q4/business_and_financial_update.pdf', 21 'https://s2.q4cdn.com/299287126/files/doc_financials/2022/q1/Q1-2022-Amazon-Earnings-Release.pdf', 22 'https://s2.q4cdn.com/299287126/files/doc_financials/2022/q2/Q2-2022-Amazon-Earnings-Release.pdf', 23 'https://s2.q4cdn.com/299287126/files/doc_financials/2022/q3/Q3-2022-Amazon-Earnings-Release.pdf', 24 'https://s2.q4cdn.com/299287126/files/doc_financials/2022/q4/Q4-2022-Amazon-Earnings-Release.pdf' 25 ] 26 27def load_reports(urls: List[str]) -> List[str]: 28 ''' Load pages from a list of urls''' 29 pages = [] 30 31 for url in tqdm.tqdm(urls): 32 r = requests.get(url) 33 path = url.split('/')[-1] 34 with open(path, 'wb') as f: 35 f.write(r.content) 36 loader = PagedPDFSplitter(path) 37 local_pages = loader.load_and_split() 38 pages.extend(local_pages) 39 return pages 40 41pages = load_reports(urls) 42CopyWe now use the Text Splitter Util to split documents into pages.1text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) 2texts = text_splitter.split_documents(pages) 3 4embeddings = OpenAIEmbeddings() 5 6db = DeepLake(dataset_path='hub://davitbun/amazon_earnings_6', embedding_function=embeddings, token=os.environ['ACTIVELOOP_TOKEN']) 7db.add_documents(texts) 8 9CopySince the dataset is stored on Activeloop, you can load it later without recomputing embeddings. This is a significant benefit cause it would save you time and computational resources. LangChain has a wrapper around Deep Lake, allowing you to use it as a Vector Store.1qa = RetrievalQA.from_chain_type(llm=OpenAIChat(model='gpt-3.5-turbo'), chain_type='stuff', retriever=db.as_retriever()) 2 3Copy


What are the main benefits of LangChain for large language models?

LangChain provides several benefits for developers working with LLMs. These comprise, but are not limited to: Simplified development: LangChain streamlines the development process by providing a simple pipeline for generating text from input prompts. This saves time and effort for developers who would otherwise have to handle complex NLP tasks manually. Improved accuracy: LangChain's 'chained' approach can enhance the accuracy of end result by ensuring consistency and coherence between the input prompts and the generated text. The model evaluation toolkit helps compare and improve the large language models you're working on. Customizability: LangChain's flexible pipeline allows developers to customize it to their specific needs. This includes choosing different LLMs and combining them with utilities such as search engines, modifying the prompt template, or handling the generated output in a custom way. But, it also can work out of the box, offering a multitude of pre-built prompts and chains for standard use cases, such as question answering, summarization, and chatbots, so that users can rapidly build and deploy language model applications. Interoperability: LangChain is built to integrate easily with various tools, data sources, and Python libraries, including Deep Lake, making it a stellar choice for building complex applications to solve multiple business problems.


How to answer questions using ML?

The state-of-the-art method of question answering is by leveraging LangChain and Activeloop Deep Lake. You can store your documents in the unified, streamable format, and connect it to LangChain to run a question-answering model such as GPT or BERT. These language models are able to understand the context of the question and generate a more accurate answer. You may also implement techniques like data augmentation, document retrieval, and summarization to enhance the system performance.Let's now explore Amazon's revenue change pre- and post-pandemic.1qa.run('Combine total revenue in 2020?') 2#Amazon's total revenue in 2020 was $386,064 million. 3CopyThis answer is obviously incorrect, as we didn't use any sophisticated methods for addition. We will explore further optimization for this use case to consistently get good answers by employing a chain of agents. For now, however, more straightforward queries can be answered very quickly:1qa.run('What is the revenue in 2021 Q3?') 2# The net sales for Q3 2021 was $110.8 billion, with a 15% increase to $127.1 billion in Q3 2022. 3Copy1qa.run('What is the revenue in 2023 Q1?') 2# The financial results for 2023 Q1 are not provided in the given context 3CopyObviously, the question answering works only on the given data, that is limited to 2022. To answer more current questions, we should connect LangChain to agents like GoogleSearchAPIWrapper. We will explore this in a further article.No more switching tabs between reports! Just feed the entirety of the information to LangChain and store the converted text in the Deep Lake format for immediate answers.You can also use the Colab Notebook to streamline the development.


Can I use LangChain to query financial data?

Yes. With Deep Lake as your vector store, you can store your financial pdfs or csv files and their embeddings and ask questions on top of your financial data.


Concluding Remarks: Use LangChain & Activeloop for efficient LLM training

In conclusion, LangChain is a robust tool for productizing large language models, providing a simple workflow to generate output by employing prompts, utilities, and language models in a linked fashion. It offers numerous benefits, including streamlined LLM development, improved accuracy of LLMs, better use-case customization, easy integration with data sources, and other Python libraries, including Activeloop Deep Lake. Deep Lake, in its turn, enables rapid LangChain prototyping, as you can immediately access data without the need to recompute the embeddings for the model finetuning. If you're working with LLMs and looking to streamline your development process, LangChain, powered by Deep Lake as the data store for LLM training, is worth exploring.


What are the main LangChain modules?

The main modules of LangChain include prompts, LLMs, document loaders, vector stores like Deep Lake, utils, chains, indexes, agents, memory, and chat.


How can I get started with LangChain?

Research: After reading this ultimate guide, familiarize yourself with the LangChain project, its goals, and its underlying technology. Read the documentation, and explore related LangChain articles to build a solid foundation. Join the Community: Connect with the LangChain community discord and follow them on Twitter to stay updated on the latest developments. Experiment with LangChain Sample Projects: Start by experimenting with sample projects for code understanding, building chatbots or answering questions over specific documents or webpages.


How does LangChain help with prompt management and optimization?

LangChain provides prompt templates and various agents that can use LLMs to decide what actions should be taken. In theory, you can use LangChain to iteratively optimize your prompts.


What is the use of LangChain?

LangChain allows users to build advanced AI applications using large language models (LLMs) for various tasks, such as chatbots, Generative Question Answering (GQA), code understanding or summarization. The library is designed to 'chain' together different components, including prompt templates, LLMs, agents, and memory, to create sophisticated use cases around LLMs. LangChain supports both OpenAI's GPT family of models, as well as open-source alternatives, offering flexibility and adaptability for developers.


How old is LangChain?

LangChain is was launched in October 2022. At the time of writing this article, it was approximately 7 months old.


Is LangChain open-source?

Yes, LangChain is is fully open-source and comprises Python and Javascript libraries.


Introduction: the History of Queries: from Papyrus Scrolls to Text to SQL, enabled by GPT-4

Querying datasets has been a crucial aspect of database management systems since their inception. It is the process of extracting data from a database based on specific criteria. Over the years, querying has undergone significant changes, from simple command-line interfaces to complex graphical user interfaces and, most recently, natural language processing. In this blog post, we will review each notable milestone in the evolution of sending commands to manipulate database data. While we've previously focused on the evolution of data storage, covering the development from Oracle to Databases for AI, this blog post will mainly focus on the querying side of things. We will also introduce the next step in the querying lifecycle - talking to the databases in natural language, as you would talk to your business intelligence analyst or data engineer.But first, let's get to the basics.


Retrieving Data pre-1960s

In 1945, magnetic tapes were developed as the first medium for searching data, replacing punch cards and paper tape. In the 1950s, with the increasing use of computers and the installation of the first commercial computer in 1957, the need for a more efficient and standardized way of accessing data became apparent. 1959 saw McGee propose generalized access to electronically stored data, and IBM introduced the Ramac system. As a result, analysts could read data in a non-sequential manner, and access to files became feasible.


The 1970s: Relational databases are born, and SEQUEL evolves

The 1970s saw the emergence of the relational model, proposed by Edgar F. Codd. The relational model allowed data to be stored in tables, making it easier to retrieve and manipulate data. Inspired by this, Donald D. Chamberlin and Raymond F. Boyce, at IBM's San Jose location (quite close to Activeloop's HQ), developed the Structured English Query Language (SEQUEL). After several attempts at creating a relational database language involving multiple experiments, puns on the query language of Ingres (QUEL), and a trademark issue with a British aircraft manufacturing company forcing them to drop the vowels, they settled on SQL. Next, IBM began developing commercial products based on their System R prototype. In 1979, Relational Software introduced Oracle V2, one of the first commercially available implementations of SQL.


The 1980s: Road to SQL becoming a global standard

With the rise of the development of personal computers, database management systems (DBMS) such as DBASE and PARADOX were created to allow PC users to manipulate and define data. However, these systems lacked insulation between programs and data and didn't support multiview or multiaccess capabilities. As computer sales rapidly increased, relational database systems emerged as a commercial success, leading to a decline in the popularity of network and hierarchical database models. IBM's DB2 became a flagship product, and the introduction of IBM PC led to the establishment of new database companies and the creation of various products, including PARADOX, RBASE 5000, RIM, Dbase III and IV, OS/2 Database Manager, and Watcom SQL. Finally, ANSI (1986) and ISO (1987) officially adopted the standard 'Database Language SQL' language definition, cementing SQL as the dominating query language for the years to come.


The 2010s: the rise of NoSQL, MongoDB, Snowflake, & Databricks

The 2010s saw the rise of big data and the need for more advanced querying techniques. NoSQL databases, such as MongoDB and Cassandra, emerged as viable alternatives to traditional relational databases. These databases allowed users to store and query large volumes of unstructured data. In addition, machine learning and artificial intelligence (AI) techniques were integrated into querying systems, allowing users to ask more complex questions and retrieve more relevant information. I've covered the differences between the options, including Lakehouse (and Photon) and Data Lake, in this article, in case you're interested.


The 1960s: The onset of computerization requires querying

In the 1960s, the advent of cost-effective computers enabled private organizations to begin storing data electronically, marking the start of computerized databases. Two dominant data models characterized this era: the network model known as CODASYL and the IMS hierarchical model. Among these models, the SABRE system emerged as a commercial success, allowing IBM to help American Airlines effectively manage its reservation data. The set was being staged for the first (truly) relational database systems to be designed in the next decade


The 2000s: Querying goes mainstream

During the 2000s, natural language processing (NLP) was increasingly used in querying, allowing non-technical users to retrieve data by asking questions in plain language. NLP-based systems such as Wolfram Alpha and Google's Knowledge Graph enabled users to retrieve relevant information from databases. Despite a dot-com bubble burst-driven decline in the early 2000s, database applications continued to grow, with new interactive applications developed for PDAs, point-of-sale transactions, and consolidation of vendors. The leading database companies in the western world were Microsoft, IBM, and Oracle. It wasn't until 2009 that NoSQL became more widespread by organizing an event to discuss non-relational databases (thanks, Johan Oskarsson!). Today, NoSQL databases enjoy wide adoption, which we will cover in the next section.


Hello, Text To SQL (or, in our case, Text to TQL)

What is Text to SQL? It is a technology that converts natural language queries into SQL code. It is rapidly gaining popularity due to its ability to simplify querying databases, particularly for non-technical users. Text to SQL was enabled with hard work from various researchers, including companies like Salesforce, that released the crowdsourced the WikiSQL dataset, containing queries in SQL and natural language prompts. If interested, check out their work, including the Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning.With Text to SQL, users can ask questions in plain English and get the desired results without having to write complex SQL queries. Naturally, this saves time and increases efficiency. Simplifying the process of querying databases, Text to SQL eliminates the need for extensive training or knowledge of complex SQL commands. In addition, Text to SQL can help bridge the gap between technical and non-technical teams, making it a valuable tool for businesses.Another benefit of Text to SQL is its ability to increase data accessibility. By allowing users to query databases using natural language, it reduces the need for manual data processing and interpretation, which can result in faster access to information. This can be particularly beneficial in industries where quick access to data is critical, such as finance, healthcare, and e-commerce.Having this in mind, we decided to ship Text to TQL to simplify querying complex datasets like Imagenet or COCO dataset and ease both technical and non-technical users into machine learning datasets exploration.


The 1990s: Object-oriented databases emerge

In the 1990s, the database industry saw the emergence of object-oriented databases (OODBs) that allowed for the storage of complex data structures and retrieval using object-oriented query languages like OQL and QBE. This decade saw the demand for extending DBMS capabilities to meet new applications, the emergence of commercial object-oriented DBMSs, and the demand for developing applications utilizing data from various sources and of different modalities (temporal, multimodal, etc). Furthermore, the decade saw the emergence of standards for data query and exchange (SQL2, PDES, STEP) and the extension of DBMS capabilities to heterogeneous and multi-database systems. The advent of the internet in the middle of the decade led to exponential growth in the database industry, with average desktop users accessing client-server database systems to access computer systems with legacy data. The introduction of GUI-based database management systems, such as Microsoft Access and Oracle Forms, made querying more accessible to non-technical users. These systems allowed users to create queries by selecting options from drop-down menus and forms. What is more, the decade saw increased investment in online businesses resulting in a rise in demand for internet database connectors such as Front Page, Active Server Pages, Java Servlets, and more.Finally, in 1998, Carlo Strozzi coined the term 'NoSQL' when naming his database Strozzi NoSQL. Although it was still de facto relational, it would significantly impact querying in the new millennium.


SQL Query Where Contains, but slightly different

So how does this work? Imagine you're working on the COCO training dataset, and need to build a simple classifier to discern between cats and dogs. Show me 100 images of dogs and 100 images of cats You type this text into the upper box and hit run. Almost instantly, we return the actual query:And we also run it for you, so you automatically have your data.Apart from labels for image data, you can also query other modalities, e.g., text data. Let's consider Stanford Question Answering Dataset (SQuaD dataset). The dataset consists of a set of questions and answers to them. show me data where the question contains France and the context contains Paris The UI will generate the query:1select * where contains(question, 'France') and contains(context, 'Paris') 2CopyText to TQL can also handle basic computational queries. Let's try a simple query on MNIST. find all images where the sample is smaller or equal to 3 Or,So, what is Behind Text to SQL?


Introducing Automated Tensor Query Language, powered by Large Language Models

What is TQL? Tensor Query Language (TQL) powerful dataset query engine built by Activeloop in C++, optimized for Deep Lake datasets, enables users to filter relevant data in data science workflows. Queries can be performed both in the UI and in the Python API. The resulting Dataset Views can be saved, loaded, and passed to machine learning frameworks for analysis. Dataset query syntax includes various operators like CONTAINS, ==, SHAPE, LIMIT, AND, OR, NOT, UNION, INTERSECT, ORDER BY, ANY, ALL, and SAMPLE BY, enabling flexible querying of datasets.In our most recent update, TQL leverages the power of machine learning models and algorithms to understand and interpret natural language queries and then provides relevant results based on the user's input. Thus with TQL, users can query datasets based on the labels, or tags, associated with the data. This makes it easier to find and retrieve specific data points that match certain criteria without needing to know the underlying structure or format of the dataset. Text to TQL is a powerful new tool for data scientists, machine learning engineers, and anyone working with large datasets who wants to quickly and efficiently find and analyze specific data points.


What is GPT-4?

GPT-4 is a large multimodal model (accepting image and text inputs, emitting text outputs - for now only for limited cases, though). It is a transformer-based model pre-trained to predict the next token in a document. Its post-training alignment process improves performance on measures of factuality and adherence to the desired behavior (API users can specify to a certain extent as 'system requirements').


The Future of Querying Datasets

Querying data has come a long way since its inception in the 1960s (or scrolling through papyrus rolls by Egyptians starting around 3000 BCE). The evolution of querying has been driven by advancements in computer technology, the rise of non-technical users, and the need for more advanced querying techniques. Today, querying is an integral part of database management systems, and with the advancements in AI and machine learning, it is set to become even more accessible and efficient.We are not fans of crystal balls, so instead of hypothesizing what the future of querying would look like, we attempted to build it. Enter querying for machine learning, with natural language as an input.


What are GPT-4 Advantages?

One advantage of GPT-4 is its capability to process both image and text inputs, expanding its potential use cases in dialogue systems, text summarization, as well as machine translation. Additionally, it exhibits improved performance on measures of factuality and adherence to the desired behavior.


How does GPT-4 compare to ChatGPT/GPT-3.5?

Compared to ChatGPT and GPT-3.5, GPT-4 demonstrates human-level performance across various professional and academic benchmarks, including passing a simulated LSAT test with a score around the top 10% of test takers (ChatGPT, in contrast, resides in the bottom 10% for the same test). It also outperforms previous large language models and most state-of-the-art systems on traditional NLP benchmarks and exhibits strong performance in non-English languages.'


GPT-4 Disadvantages

Nonetheless, like earlier GPT models, GPT-4 is not fully reliable and can suffer from 'hallucinations,' has a limited context window, and does not learn from experience.To tune GPT-4 for our use case, we have fed it countless examples of correct TQL queries for various datasets (of different modalities and schemas). For our use case, we noticed that GPT-4 currently walks a fine line between being to 'verbose' and generous. For instance, for a query 'find all things related to trees' it may produce a result that contains lemons (as they grow on trees) or even cliffs (…? Our guess is as good as yours here). On the other hand, it sometimes can err in over-relying on user-specified query. For example, Imagenet does have entire schools of fish (enough to build an entire education system, might I add). Asking TQL + GPT-4 to find all the fish in dataset might result in the following:1 select * where contains(labels, 'fish') 2CopyWhen the actual expected query could have been:1 select * where any(contains(labels, 'shark'), contains(labels, 'ray'), contains(labels, 'pike'), contains(labels, 'trout'), contains(labels, 'bass'), contains(labels, 'salmon'), contains(labels, 'eel'), contains(labels, 'sturgeon'), contains(labels, 'coho'), contains(labels, 'barracouta')) 2Copyand so on…One thus should be cautious when using the outputs of GPT-4, particularly in contexts where reliability is important. However, we are working at making this as reliable as possible. Whatever the case, we are confident this is the right step forward: out of hundreds of tests we've conducted, the new feature outperformed first-time query users whopping 99% of the time.


Concluding Remarks

In sum, querying databases has come a long way since the dawn of magnetic tapes in the 1940s. From punch cards to natural language processing, the evolution of querying has been driven by advancements in computer technology and the need for more accessible and efficient methods for extracting data from databases. The recent emergence of Text to SQL technology, including our own Text to TQL, has revolutionized the querying process by allowing users to retrieve data from databases using plain English queries. This technology saves time, increases efficiency, and makes querying more accessible even to non-technical users. Try out Text to TQL yourself and experience the benefits of simplified, efficient querying for complex datasets such as Imagenet dataset to explore the data, as well as to train and edit it with data lineage or evaluate ML model performance.


How long does it take to learn SQL?

According to Berkley, the basics of SQL can be learned in 2 to 3 weeks due to its relative simplicity, but higher fluency requires additional time. Learning independently can take up to six months or longer. With Text to SQL, non-technical users need not learn how to code to easily access their data, which can result in faster and more accurate decision-making.


GPT-4

Since it's release a mere week ago, GPT-4 (Chat-GPT's younger but stronger cousin) has been making waves across the Twitterverse and the industry. Companies (including us!) are already integrating it in their day-to-day operations to improve the user experience. We, too, wanted to make our users productive. :)


What is a Database Query?

A database query is a request for information from a database. It usually involves some form of filtering or selecting data based on specific criteria or conditions. Queries are used to retrieve data, perform calculations or transformations on that data, and to combine data from multiple tables or sources. They are an essential part of database management and support a wide range of applications and business processes. Queries can be written in different programming languages or using graphical user interfaces provided by database management systems. The most common query language used today is Structured Query Language (SQL), used for relational databases.


When to use NoSQL vs. SQL?

When choosing between SQL and NoSQL databases, it's crucial to evaluate your application's unique needs. SQL databases are an ideal choice for applications that demand complicated queries and data consistency. They're also useful for applications that must abide by strict data integrity rules since they can manage complicated transactions and contain well-defined schemas. SQL databases perform best for applications that handle heavy write loads and data relationships, such as e-commerce websites and financial systems. Naturally, the expected datatype is structured (tabular).When to use NoSQL? In contrast to SQL databases, NoSQL databases are a better fit for applications that require high scalability and flexibility, especially when managing unstructured or semi-structured data. They can effectively manage large volumes of data, including non-tabular and non-relational datasets, and are tailored to high-performance, distributed environments. Big data and real-time analytics applications, content management systems, and mobile applications prioritizing speed & scalability are great use cases for when to use a NoSQL database.Ultimately, selecting between SQL and NoSQL hinges on your specific demands, including the size and intricacy of your data, the type of queries you need to execute, and the scalability and flexibility you demand. Here's a brief table summarizing the differences between SQL vs NoSQL databases. We've also added TQL comparison, with TQL being powered by Deep Lake. CriteriaSQLNoSQLTQL Data StructureStructured dataUnstructured dataBoth unstructured and structured data, higher performance on unstructured Data VolumeRelatively smaller data setsVery large and rapidly growing data setsDataset size irrelevant, handles rapid scalability for more than 1B rows Data RelationshipComplex relationships and joins between tablesSimple or no relationships between dataRelationship-agnostic ScalabilityVertical scaling (adding more resources to a single server)Horizontal scaling (distributing data across multiple servers)Both horizontal and vertical scaling ACID ComplianceACID compliant (Atomicity, Consistency, Isolation, Durability)Mostly eventual consistencyACID compliant TransactionsSupports complex transactions with rollbacks and commitsDoes not support complex transactionsSupports complex transactions with rollbacks, commits and time travel PerformanceOptimized for complex queries on a few large data setsOptimized for many simple queries on large and rapidly changing data setsOptimized for complex queries on any amount of large and rapidly evolving datasets FlexibilityRigid schema structureDynamic schema structureDynamic schema structure CostHigh cost for large data setsLower cost for large data setsUltra low cost for large data sets and related machine learning training processes on them Use CasesSuitable for financial, e-commerce, and other transactional applicationsSuitable for real-time data analytics, social media, and IoT applicationsSuitable for generative AI, machine learning and computer vision use cases on multi-modal data


References

Database History Timeline until 1990s Timeline of Database History The Evolution Of Data Models And Approaches To Persistence In Database Systems


What is Retrieval Augmented Generation (RAG) in AI?

Retrieval Augmented Generation, or RAG, is an advanced technique in AI that bridges information retrieval and text generation. It is designed to handle intricate and knowledge-intensive tasks by pulling relevant information from external sources and feeding it into a Large Language Model for text generation. When RAG receives an input, it searches for pertinent documents from specified sources (e.g., Wikipedia, company knowledge base, etc.), combines this retrieved data with the input, and then provides a comprehensive output with references. This innovative structure allows RAG to seamlessly integrate new and evolving information without retraining the entire model from scratch. It also enables you to fine-tune the model, enhancing its knowledge domain beyond what it was trained on.


Introduction to Retrieval Augmented Generation (RAG) in AI

Retrieval Augmented Generation (RAG), a new frontier in AI technology, is transforming the digital landscape. With platforms like Cohere & Activeloop, this advanced technology is now easily accessible and customizable, catalyzing a wave of AI-first businesses.RAG's impact is considerable. MIT research shows businesses incorporating RAG report up to 50% productivity gains on knowledge-based tasks. By automating mundane tasks, businesses improve resource allocation and employee satisfaction. Notably, Goldman Sachs estimates that such advancements could boost global GDP by 7%.RAG's versatility is seen across industries. In customer support, it leads to a 14% productivity increase per hour. In sales, AI-assisted representatives send five times more emails per hour. With the maturation of this technology, these figures will rise even further.The future of RAG points towards the development of Knowledge Assistants. Acting as intelligent tools for workers, they will retrieve and process corporate data, interact with enterprise systems, and take action on a worker's behalf. This heralds a new age of AI-driven productivity.As the third significant revolution in human-computer interfaces, RAG, and LLMs could unlock an estimated $1 trillion in economic value in the U.S. alone. Therefore, businesses and developers must adopt these technologies to remain competitive in the rapidly evolving AI-centric future.At the end of this article, we cover the Retrieval Augmented Generation History and other fun facts.


Build LLM-powered Chatbot with RAG

To demonstrate the power of Retrieval Augmented Generation for building AI Chatbots with LangChain & Vector Databases, we will build a course companion chatbot for our LangChain & Vector Databases in Production course.Educational Chatbot harnesses the power of AI to answer queries and provide relevant information to users by retrieving data from an extensive and detailed knowledge base. It returns a natural response to the user's question and the truth source.Application Demo: Databutton | LangChain Chat with Deep Lake Vector DatabaseApplication Building Steps: Data Loading Retrieving Data Building Conversation Chain with Memory and Retrieval Building the Chat UI


Learning Material and Resources

In order to build this sophisticated Retrieval Augmented Chatbot, I undertook a comprehensive educational journey that includes courses from the ‘LLM University by Cohere' and ‘LangChain & Vector Databases in Production'.


Setting up LangChain & Databutton

LangChain is a standard interface through which you can interact with a variety of large language models (LLMs). It provides modules you can use to build language model applications. It also provides chains and agents with memory capabilities.The flowchart below demonstrates the pipeline initiated through LangChain to complete the Conversation Process. The tutorial goes into each of the steps in the pipeline, this visual helps to give you an overview of how the components are working together and in what order.The design pattern started by thinking about the following: What problem am I trying to solve? Who is going to benefit from this solution? How am I going to get and pre-process my data sources? How am I going to store and retrieve my data sources? How is the user going to interact with my data sources? Taking a step back before building a solution can really help to save time and importantly considers your end user.


Build your Databutton Application

Create a free account with Databutton Create a new app Once you have signed up for your free Databutton account, you can create a new app in seconds by clicking on ‘New app' Add secrets and packages To use the API Key in your app, copy the code snippet from the secret, this will look something like this: ‘COHERE_API_KEY = db.secrets.get(name='COHERE_API_KEY')' Add the packages below and click install. 1langchain 2deeplake 3openai 4cohere 5apify-client 6tiktoken 7Copy Add entire code from the tutorial to either the Jobs section or the Home Page as specified in the steps below.


Application Platform and Required API Keys

Databutton: All-in-one app workspace where we will build and deploy our application. $25 free monthly quota (covers one app a month), community and student plans available. Cohere API key: Generative AI endpoint for embeddings, rerank and chatbot. Get free, rate-limited usage for learning and prototyping. Usage is free until you go into production Apify API Key: Web scraping data for the chatbot to retrieve. $5 free usage (more than enough for website contents) Activeloop token: We will use Deep Lake to store the text scraped from a website. Deep Lake Community version is free to use.


Step 1: Loading the Data with RecursiveCharacterTextSplitter

In this stage, we are gathering the data needed to provide context to the chatbot. We use ApifyLoader to scrape the content from a specific website. The RecursiveCharacterTextSplitter is then used to split the data into smaller, manageable chunks. Next, we embed the data using CohereEmbeddings which translates the text data into numerical data (vectors) that the chatbot can learn from. Lastly, we load the transformed data into Deep Lake.The code for this step is located in the ‘Jobs' section within Databutton because this is a task that only needs to be run once. Once the data is collected and loaded into DeepLake, it can be retrieved by the chatbot.


Helper Functions

ApifyWrapper(): Scrapes the content from websites. 1from langchain.document_loaders import ApifyDatasetLoader 2from langchain.utilities import ApifyWrapper 3from langchain.document_loaders.base import Document 4import os 5 6os.environ['APIFY_API_TOKEN'] = db.secrets.get('APIFY_API_TOKEN') 7 8apify = ApifyWrapper() 9loader = apify.call_actor( 10 actor_id='apify/website-content-crawler', 11 run_input={'startUrls': [{'url': 'ENTER\YOUR\URL\HERE'}]}, 12 dataset_mapping_function=lambda dataset_item: Document( 13 page_content=dataset_item['text'] if dataset_item['text'] else 'No content available', 14 metadata={ 15 'source': dataset_item['url'], 16 'title': dataset_item['metadata']['title'] 17 } 18 ), 19) 20 21docs = loader.load() 22Copy ApifyWrapperRecursiveCharacterTextSplitter(): Splits the scraped content into manageable chunks. 1from langchain.text_splitter import RecursiveCharacterTextSplitter 2 3# we split the documents into smaller chunks 4text_splitter = RecursiveCharacterTextSplitter( 5 chunk_size=1000, chunk_overlap=20, length_function=len 6) 7docs_split = text_splitter.split_documents(docs) 8Copy CohereEmbeddings(): Translates text data into numerical data. DeepLake(): Stores and retrieves the transformed data. 1from langchain.embeddings.cohere import CohereEmbeddings 2from langchain.vectorstores import DeepLake 3import os 4 5os.environ['COHERE_API_KEY'] = db.secrets.get('COHERE_API_KEY') 6os.environ['ACTIVELOOP_TOKEN'] = db.secrets.get('APIFY_API_TOKEN') 7 8embeddings = CohereEmbeddings(model = 'embed-english-v2.0') 9 10username = 'elleneal' # replace with your username from app.activeloop.ai 11db_id = 'kb-material'# replace with your database name 12DeepLake.force_delete_by_path(f'hub://{username}/{db_id}') 13 14dbs = DeepLake(dataset_path=f'hub://{username}/{db_id}', embedding_function=embeddings) 15dbs.add_documents(docs_split) 16Copy


A Brief Intro to Cohere's Rerank Endpoint for Enhanced Search Results

Within a search process, Cohere's Rerank endpoint serves as a final step to refine and rank documents in alignment with a user's search criteria. Businesses can seamlessly integrate it with their existing keyword-based (also called 'lexical') or semantic search mechanisms for initial retrieval. The Rerank endpoint will take over the second phase of refining results.Cohere's Rerank & Deep Lake: The Solution to Imprecise Search Outcomes: This tool is powered by Cohere's large language model, which determines a relevance score between the user's query and each of the preliminary search findings. This approach surpasses traditional embedding-based semantic searches, delivering superior outcomes, especially when dealing with intricate or domain-specific search queries.This tool is powered by Cohere's large language model, which determines a relevance score between the user's query and each of the preliminary search findings. This approach surpasses traditional embedding-based semantic searches, delivering superior outcomes, especially when dealing with intricate or domain-specific search queries.The DeepLake instance is then turned into a retriever with specific parameters for distance metric, number of items to fetch (fetch_k), use of maximal marginal relevance and the number of results to return (k).1from langchain.vectorstores import DeepLake 2from langchain.embeddings.cohere import CohereEmbeddings 3from langchain.retrievers import ContextualCompressionRetriever 4from langchain.retrievers.document_compressors import CohereRerank 5import os 6 7os.environ['COHERE_API_KEY'] = db.secrets.get('COHERE_API_KEY') 8os.environ['ACTIVELOOP_TOKEN'] = db.secrets.get('ACTIVELOOP_TOKEN') 9 10@st.cache_resource() 11def data_lake(): 12 embeddings = CohereEmbeddings(model = 'embed-english-v2.0') 13 14 dbs = DeepLake( 15 dataset_path='hub://elleneal/activeloop-material', 16 read_only=True, 17 embedding_function=embeddings 18 ) 19 retriever = dbs.as_retriever() 20 retriever.search_kwargs['distance_metric'] = 'cos' 21 retriever.search_kwargs['fetch_k'] = 20 22 retriever.search_kwargs['maximal_marginal_relevance'] = True 23 retriever.search_kwargs['k'] = 20 24 25 compressor = CohereRerank( 26 model = 'rerank-english-v2.0', 27 top_n=5 28 ) 29 compression_retriever = ContextualCompressionRetriever( 30 base_compressor=compressor, base_retriever=retriever 31 ) 32 return dbs, compression_retriever, retriever 33 34dbs, compression_retriever, retriever = data_lake() 35Copy


Step 4: Building the Chat UI

In this final step, we set up the chat user interface (UI).We start by creating a button that, when clicked, triggers the clearing of cache and session states, effectively starting a new chat session.Then, we initialize the chat history if it does not exist and display previous chat messages from the session state.1# Create a button to trigger the clearing of cache and session states 2if st.sidebar.button('Start a New Chat Interaction'): 3 clear_cache_and_session() 4 5# Initialize chat history 6if 'messages' not in st.session_state: 7 st.session_state.messages = [] 8 9# Display chat messages from history on app rerun 10for message in st.session_state.messages: 11 with st.chat_message(message['role']): 12 st.markdown(message['content']) 13CopyThe chat_ui function is used to handle the chat interactions. Inside this function, we accept user input, add the user's message to the chat history and display it, load the memory variables which include the chat history, and predict and display the chatbot's response.The function also displays the top 2 retrieved sources relevant to the response and appends the chatbot's response to the session state. The chat_ui function is then called, passing in the ConversationalRetrievalChain object.1def chat_ui(qa): 2 # Accept user input 3 if prompt := st.chat_input( 4 'Ask me questions: How can I retrieve data from Deep Lake in Langchain?' 5 ): 6 7 # Add user message to chat history 8 st.session_state.messages.append({'role': 'user', 'content': prompt}) 9 10 # Display user message in chat message container 11 with st.chat_message('user'): 12 st.markdown(prompt) 13 14 # Display assistant response in chat message container 15 with st.chat_message('assistant'): 16 message_placeholder = st.empty() 17 full_response = '' 18 19 # Load the memory variables, which include the chat history 20 memory_variables = memory.load_memory_variables({}) 21 22 # Predict the AI's response in the conversation 23 with st.spinner('Searching course material'): 24 response = capture_and_display_output( 25 qa, ({'question': prompt, 'chat_history': memory_variables}) 26 ) 27 28 # Display chat response 29 full_response += response['answer'] 30 message_placeholder.markdown(full_response + '▌') 31 message_placeholder.markdown(full_response) 32 33 #Display top 2 retrieved sources 34 source = response['source_documents'][0].metadata 35 source2 = response['source_documents'][1].metadata 36 with st.expander('See Resources'): 37 st.write(f'Title: {source['title'].split('·')[0].strip()}') 38 st.write(f'Source: {source['source']}') 39 st.write(f'Relevance to Query: {source['relevance_score'] * 100}%') 40 st.write(f'Title: {source2['title'].split('·')[0].strip()}') 41 st.write(f'Source: {source2['source']}') 42 st.write(f'Relevance to Query: {source2['relevance_score'] * 100}%') 43 44 # Append message to session state 45 st.session_state.messages.append( 46 {'role': 'assistant', 'content': full_response} 47 ) 48 49# Run function passing the ConversationalRetrievalChain 50chat_ui(qa) 51Copy